Monitor Aura system

• 1: Aura Analytics 1.1.
• 1.1: Architecture
• 1.2: OB analytics
• 1.3: Data model
• 1.4: Annex: Dataset fields
• 2: Aura Analytics 2.0.0
• 2.1: Architecture
• 2.2: Operation
• 2.3: Guidelines for OBs
• 2.4: Analytics Dashboard
• 2.5: Annex: Dataset fields
• 3: Aura Billing Module
• 3.1: Aura Billing Module operation
• 4: Manage Aura logs
• 5: Manage metrics
• 5.1: Aura Bot metrics
• 5.2: Aura Groot metrics
• 5.3: Atria Model Gateway metrics
• 5.4: Atria RAG server metrics
• 5.5: Aura Authentication API metrics
• 5.6: Aura Configuration API metrics
• 5.7: Aura Gateway API metrics
• 5.8: Aura Bridge metrics
• 5.9: Aura KPIs uploader metrics
• 5.10: Aura NLP metrics
• 5.11: T&C API metrics
• 5.12: NLP provisioning metrics
• 5.13: Aura Complex Logic metrics
• 5.14: Aura Context metrics
• 5.15: Aura File Manager metrics
• 5.16: Aura Redis MongoDB sync metrics
• 6: Aura dashboards
• 6.1: Aura system dashboards
• 6.1.1: Alertmanager dashboard
• 6.1.2: Elasticsearch dashboard
• 6.1.3: Fluent bit dashboard
• 6.1.4: Kubernetes cluster monitoring dashboard
• 6.1.5: Kubernetes cron and batch job monitoring dashboard
• 6.1.6: Kubernetes nodes dashboard
• 6.1.7: Kubernetes services dashboard
• 6.1.8: Kubernetes storage monitoring dashboard
• 6.1.9: NLP provisioning dashboard
• 6.1.10: Prometheus stats dashboard
• 6.1.11: Redis dashboard
• 6.2: Aura components dashboards
• 6.2.1: Aura bot latencies dashboard
• 6.2.2: Aura bridge dashboard
• 6.2.3: Authentication API dashboard
• 6.2.4: Aura HTTP Inbound dashboard
• 6.2.5: Aura HTTP Outbound dashboard
• 6.2.6: Pod resources dashboard
• 7: Aura Alerts
• 8: Queries
• 8.1: Basic monitoring queries
• 8.2: Basic database queries

1 - Aura Analytics 1.1.

Aura Analytics 1.1.

Introduction

This document contains a description of a joint dataflow between LCDO OB teams and Aura Global Team for processing Aura log files created in production environment (i.e., coming from actual Aura users) in order to create PPDs (Privacy-Preserving Datasets). All this process is known as Active Listening.

The dataflow produces as a result, among other elements, an analytics component, named as Aura Analytics Dashboard, that can be used to gather statistics on the production system and to analyze user’s behavior. The latest version 1.1 of this dashboard is described in the current document.

The main objectives of the unified dataflow are:

• Consolidate the processing of Aura logs into a framework.
• Provide LCDOs and Aura Global Team with a unified common source for analytics, in a privacy-preserving way.
• Enable extensibility of the dataflow.

In this framework, the current documents provide:  

• The available data analytics versions
• Prerequisites and recommended tools for its use
• Overall description of Aura Analytics architecture
• Description of the OB Analytics subsystem that enables the management by OBs
• Current Aura Analytics data model

The target audience of this document includes the following roles both in LCDO Teams and Aura Global Team:

• Data Scientists and Product teams, that wish to access Aura logs information and perform analytics on them.
• Operation teams, for the architectural description and the requirements on OB environments.

Aura Analytics versions

Release 1.0.

The first release 1.0. sets up the basic paths, deploys the PPD infrastructure and produce:

• Version 1.0. of the OB Analytics system, which includes the OB Dashboard.
• The first version of pre-processed datasets (clean PPDs) for training and analytics at Aura Global.

As mentioned, this version enables OBs to go further by:

• Enhancing the OB Dashboard with new visualizations, as they seem fit (given that panels and dashboards can be exported and imported, it is possible to share new ones across all OBs, as they are developed).

• Processing the PPD files as desired (they are standard CSV files, which can be ingested in alternative platforms if desired). Restrictions on them are softer than on the original logs due to the anonymization process they have been subjected to, although they are still subjected to management precautions (a code of conduct is being prepared for that).

Release 1.1.

Version 1.1. introduces the following changes:

• The table of data has been enlarged with these new fields: AURA_ID, STATUS_CD, sesId, sesSize, sesDuration.
• An expanded list of test users is used, so that the userType column contains more identifications.
• The code for data ingestion into a local Kibana, which previously consisted on a single Python script, has been turned into a full Python package to be installed, due to its increasing complexity.

Prerequisites and recommended tools

The prerequisites for the use of version 1.1. of Aura Analytics Dashboard are set below:

• Aura Platform version:

• Recommended operating system:Ubuntu 18.04 system

• Recommended tool for data visualization: ELK stack

1.1 - Architecture

Aura Analytics 1.1. architecture

Architecture description

The following figure shows a full overview of Aura Analytics Dashboard architecture and operation, which is also described below:

1.  Aura logs generated in local instance are converted to datasets and transferred to local Kernel via the standard procedure and with the established frequency (typically, daily).

2.  Once there, the “Active listening” process flow fires up daily. Through a specialized process that runs on an Aura local instance and with access to the stored datasets in the Kernel local storage space:

   • PII (Personally Identifiable Information) is removed or encrypted.
   • The result is transferred to a bucket/blob set up for this task and managed by Global Aura team.
   • Here, the PPDs (Privacy-Preserving Datasets) are created. Currently, only MESSAGE, RECOGNIZER and API datasets are involved in this process.

   In order to convert PII data to PPD, every field in these datasets can be:

   • a. Not transferred.
   • b. Pseudo-anonymized. In this situation, the field is transformed through a cryptographic hashing process using a secret set up by the OB.
   • c. Anonymized fragments of the field (e.g., credit card number, email, etc.). The field is processed to detect specific patterns and replaces them with a specific tag (idemail, idpassport, etc.). The list of anonymization strings is agreed with each OB.
   • d. Transferred as is.

3.  After that, the Raw PPD Datasets stored in bucket/blog managed by the Global Team are processed generating clean PPD Datasets in order to adapt them to the analytics tools.

4.  From that space, the clean PPD Datasets can be:

• Accessed by the Aura Global Team that use them for several tasks, with the purpose of evaluating Aura quality and taking the best decisions regarding to product evolution:

  • Perform analytics on Aura behavior and prototype Analytics Dashboard features
  • Improve Aura Platform capabilities (e.g., adapting machine learning models)

• Accessed by a Local Aura Team, ingesting the data to a dedicated server managed by the OB with analytics and data visualization capabilities. In order to do that, the Aura Global Team provides a component with the ELK (elasticsearch, logstash & kibana) preconfigured with a set of dashboards that can be deployed and adapted by the OB team.

All the code involved in this process can be found in Github. Particularly:

• PPD RAW creation package

• Conversion from PPD RAW to PPD Clean

• Pseudo-anonymization function for identifiers

• Utterance anonymization (agreed individually for ES and UK)

1.2 - OB analytics

OB analytics

Introduction

The OB Analytics subsystem is an optional component in the dataflow, which enables the management of clean PPDs (Privacy-Preserving Datasets) by LCDOs for the analysis of Aura behavior.

In order to work with OB Analytics subsystem, the following items must be fulfilled:

1.  The legal agreement for log management and creation of PPDs must be signed between the OB and Aura Global Team.

2.  The mechanism for PPD creation and transfer must be installed. This requires the deployment of a piece of software (provided by Aura Global Team) inside the OB cloud, with access to the repository (AWS bucket or Azure Blob Storage) holding Aura logs.

3.  A virtual machine must be deployed on the OB cloud to hold the OB Dashboard. This virtual server must be provisioned by the OB on the same cloud environment (provider and region, e.g., AWS West Europe) than the Kernel cloud, but separated from it in terms of access rights (placing it in the same cloud enables saving transfer costs from the cloud provider for PPD access).

Architecture and installation

The basic infrastructure of the OB Analytics subsystem consists on a Virtual Machine that is fed with the extracted and cleaned PPDs. This virtual machine is set up with a proposed stack of tools based on the open-source ELK framework (See figure in Architecture document).

• Elastic Search: indexing database.

• Logstash: ingester for PPD data, configured to upload the anonymized clean PPD tables into Elastic Search.

• Kibana: visualization tool offering dashboards and panels created over Elastic Search data.

The OB is required to set up the base VM, for which an Ubuntu 18.04 system is advised.

On top of this base system, Aura Global Team provides an installation kit that includes:

• The pre-processing and ingesting configuration for feeding clean PPD data into logstash.
• The indexing configuration for Elastic Search.
• Certain prototype dashboards and panels for Kibana.
• Basic security provisions (providing web-based secure access to the dashboard).

Once installed, the system automatically ingests any new clean PPD being produced, so that the index and dashboards remain up to date.

In principle, the PPD creation process specifies daily production, since Aura logs are sent to Kernel once a day. This means that information about Aura behavior and user actions on one given day will be available in the dashboards on the following day.

The provided system and installed dashboards are only visualization examples for clean PPDs. The system allows the creation of additional panels that may provide complementary insights on clean PPD elements and OBs are encouraged to explore data as they see fit.

Dashboards can be exported and reimported in a different system, so in addition to the LCDO team adding new analysis features, it is possible to provide later updates to the OB Analytics system. These updates can be provided by the Aura Global Team or shared between OBs.

Outside the dashboard stack, it is also possible to process clean PPD with alternative tools (PPDs are essentially CSV files with a defined structure, so they can be processed with a variety of tools).

Kibana dataflow

The Aura Analytics dashboard follows a standard ELK deployment:

1. An Elastic Search index has been created. It is called aura-message-COUNTRY, and its index schema contains a cleaned version of the AURA MESSAGE table (which registers input and output messages). For details on the fields that this index contains, go to the document Data model.

2. A Logstash configuration ingests into this index the cleaned sets of datapoints that are produced daily as a result of the transfer and processing of Aura logs. This is usually done in the early morning (which will then upload data for the previous day).

3. A Kibana index pattern has been created, matching the uploaded Elastic Search index. An Elastic Search index is how the data is stored inside the DB; a Kibana index pattern is how it is visualized in the interface. Typically, Kibana index patterns match Elastic Search indices, but it is, for example, possible to create a Kibana index pattern that matches more than one Elastic Search index and hence combines different data sources.

4. A small set of visualizations have been pre-installed in Kibana over that index pattern, as a means to get a default peek on the index data. See the section preinstalled visual elements to check them.

This configuration is deployed on the Kibana default space (the only one available on a freshly created Aura Analytics dashboard). If there is the need to create additional spaces, to better organize visualizations, then the Elastic Search index pattern needs to be installed into those additional spaces.

Preinstalled visual elements

Kibana offers many possibilities to visualize the ingested data and there are many resources and tutorials around explaining its mechanics. We therefore refer to the official Kibana documentation, or tutorials available on the web, for generic information.

In the particular case of the Aura Analytics deployment, there is an Elastic Search index that gets automatically ingested daily. It is called Aura-message-COUNTRY and contains a cleaned version of the AURA MESSAGE table (which registers input and output messages).

Over this index, three types of panels/visualizations have been preinstalled, to provide a starting point:

• Discover panel
• Visualizations
• Dashboards

These preinstalled elements are described in the following subsections. To access them, select the appropriate icon in the left navigation panel.

Discover panel

The Discover panel in Kibana is an essential tool where one can perform queries to an Elastic Search index (and save those searches if desired), and explore users’ interactions with Aura in detail log by log, these being filtered by:

1.  Search terms or conditions
2.  A time interval
3.  Additional filters applied to the query results
4.  A set of index fields to show in the result table

These 4 steps are represented in the following figure:

As shown in the previous figure, the starting point is the Elastic Search index holding all the data. The three first steps in the chain reduces the amount of data handled, by pruning out elements that do not satisfy the defined condition. The fourth step is just a display adjustment: on the final dataset, define which of the available fields will be shown on the output table that appears in the panel. However, the retrieved data contains all fields (clicking on any of the rows will show them).

In the Aura Dashboard default set, there is one Discover panel preinstalled. It is called question-answer pairs and has the following characteristics:

• A blank query (i.e., provide all the results)
• A time interval for the last 7 days
• A “only user” filter: it filters out all intents that correspond to non-user queries (suggestions, help commands from the client application, etc.)
• A visualization that includes: the timestamp, the (cleaned) user message, the detected aura intent, associated entities (if applicable), the dialog that was invoked and Aura’s response

This figure shows a snapshot of this panel. To load it, select the Discover tool in the left navigation bar and then click on the “Open” menu option in the top menu bar. A list of saved panels will be shown, together with the already mentioned “question-answer pairs”.

Once the panel is loaded, each one of the aforementioned four elements can be freely modified. For example, the interface allows:

• Adding new filters with the “+Add Filters” button
• Deactivating the current filters by pressing over the predefined filter and clicking over the “Temporarily Disable” option
• Modifying the query interval with the “calendar” button or “Dates Box”
• Adding a specific query on a given index field(s) by using the “Search Box”, instead of the (default) blank query.

Discover panels can be saved as named objects, to be later loaded at will. So, if needed, any panel (a modified panel or a newly created one) can be saved with a new name to have it available for later loading.

Visualizations

A total of 7 visualizations come preinstalled with the base Aura Dashboard. The list can be obtained from the “visualizations” item in the left menu bar, as shown in the figure, and they are:

• Three “Stats” type visualizations, which provide general statistics on platform usage.
• Four “User” type visualizations, which provide insights on user behavior.

Note that this distinction between “User” and “Stats” is purely conceptual and based on the fields that have been used to generate the visualizations that, from the point of view of Kibana, are all regular visualizations. Those visualizations can be instantly loaded by clicking on their names. But they can also be integrated into dashboards, as described in the next section.

Dashboards

A dashboard in Kibana is essentially a spatial arrangement of visualizations. For example, to construct a dashboard, just place visualizations into a page, resizing them as required, so they can be observed in a single place.

It is interesting to know that in a dashboard all visualizations are linked. So that if, for example, time interval is changed, or a filter is added using the interface, these modifications affect all visualizations in the dashboard and all of them get updated.

Elements in the dashboard visualizations can also generate instant filters by clicking on graphs or table elements. Those filters are then added to the top of the page as a filter and, therefore, can then be modified or removed.

The Aura Analytics default installation preloads two dashboards. Those are available for selection when we click on the “dashboard” icon in the left navigation bar:

There are different types of dashboards, described in the following sections.

System dashboard

This dashboard integrates the three predefined “Stats” visualizations (generic statistics):

• A timeline of interactions (user messages sent and answered), segmented by channel
• A heatmap of interactions by weekday and time of day (hour)
• A bar graph classifying the interactions produced in the period by detected intent

The following figure shows a screenshot of this dashboard:

User dashboard

The user dashboard contains the four visualizations:

• Most Frequent User Utterances: list of the most frequent user’s sentences (in the time interval and filter active at the moment). It uses the usrMsgSig field to group together very similar utterances.
• Aura Answer Groups: list of the most frequent answers that Aura generates, grouped by the semantic categories in AuraMsgGroup field.
• Words per query: distribution of sizes for the user messages, measured as number of words in the utterance and segmented by channel.
• Tag cloud: set of most frequent user utterances, as a tag cloud in which the font size represents the utterance frequency. The MESSAGE_USR_NORM field is used for its representation, so it contains normalized utterances.

The next screenshots show the dashboard with all these visualizations (it is a large dashboard, so typically it needs scrolling to visualize all its components).

Note that those four visualizations are linked as they correspond to the same subset of the data (as given by filters and time interval) but they are NOT linked at the individual item level (i.e., a given most frequent user utterance in the left table does not correspond to any specific Aura answer in the right bar graph).

Instead, the dashboard can be manipulated by selecting one specific item in any of the visualization and this will create a filter for the others. For instance, as the following image shows, if we select <CHURN> in the Aura answer group visualization, we can see in the others the user utterances that led Aura to generate that answer (i.e., an answer about contract cancelation).

1.3 - Data model

Aura Analytics data model

Introduction

New elements can be built (or the current elements modified) by making use of the available fields in Kibana through the ingested Elastic Search index.

In this document, we provide a reference of the schema that the index follows, so that it can be used to build such new visualizations, or to better understand the existing ones.

Elements in the Aura-message data model have 3 different types:

• Numeric: single numbers, integer or real. Suitable for numerical statistics, such as averages, or for plotting variation across time in graphs.

• Keyword: they are opaque strings, i.e., terms that cannot be searched within (it is not possible to look for words inside a keyword field). They can, however, be used to create some term-level queries, such as prefix queries (find all instances that begin with) and they usually work great for aggregations, since most of them are categorical variables (fields that only have a limited number of possible values) and can therefore be bucketed and counted.

• Text: these fields are divided into separate terms (words), and some pre-processing is done to them before indexing to improve access though an Elastic Search analyzer. Text fields cannot be used in aggregated visualizations, since they cannot be grouped. They are most useful for queries, because they allow searching for fragments (only a few words) and fuzzy searches.

Fields list

The following table lists all the fields available in the Aura-message-COUNTRY Elastic Search index, together with their type and a brief description.

The most relevant ones include a more detailed description in the section fields explanations.

Note that some fields of text type have a mirror field of type keyword, with the same content. Having the same data indexed in two different ways at the same time (as text and as keyword) enables to perform different types of analysis by choosing the right field.

The “Raw” column indicates if this field is already present in the Aura raw PPD files:

• Yes: field contained in raw PPDs.

• No: generated field, produced when creating clean PPDs. They can be recognized as lowercase fields.

• Partial: It exists in the raw PPDs, but in a somehow different shape.

Fields explanations

This subsection contains more detailed descriptions of some of the key fields in the schema.

AURA_ID_GLOBAL

This element (mostly) uniquely identifies the user generating the interaction.

Note the concrete value of this field is not the same as the actual identifier used within Aura and uploaded to Kernel: for privacy reasons, the identifier was hashed when generating the PPD and has no resemblance to the original one. The correspondence is however maintained across time, so it is possible to analyse user behavior.

The “mostly” qualifier reflects one quirk of the original Aura identifier: it is generated with a dependence to the authentication method used by the channel, so if two channels follow different authentication methods (e.g., MobileConnect vs. User/Password) then the AURA_ID_GLOBAL identifier for the same user will be different. In summary:

• The identifier stays the same for a given user across time.

• Different users will not have the same identifier.

• But the same user could produce two different identifiers if connected to two channels that use a different authentication method.

AURA_ID

This is a “local” identifier, i.e., one that is generated inside the channel according to specific channel characteristics and it is not tied as much as AURA_ID_GLOBAL to user authentication.

Its main disadvantage is its transient nature: the same user, on the same channel, could generate different AURA_ID strings when connecting different times on a different session. Therefore, for user accounting and tracing, AURA_ID_GLOBAL is usually preferred.

However, there are instances in which AURA_ID works better, namely for anonymous access (when the user is not authenticated). This depends on the channel:  

• In the WhatsApp channel, the initial use of the channel will be anonymous from the Aura side (i.e., no authentication is done), hence AURA_ID_GLOBAL will also be empty (at least until the user authenticates, which depends on the use case). But in this channel, AURA_ID has a permanent value, linked to the WhatsApp user, so here it is a good substitute for a persistent id, even for unauthenticated users.

MESSAGE_USR

This field includes the message sent by the user.

It has been partially processed to enhance anonymization by removing some standard identifiers contained in it with <idxxx> strings (e.g., phone numbers appear as <idphone>).

Removal is done mostly through regular expressions, so there might be occasional glitches (such as identifying as phone a number that does not really correspond to a phone, just because it follows the phone number pattern).

MESSAGE_USR is a field of text type. As such, it is searchable: it is possible to search for specific words the user might have said.

Furthermore, it has been processed through an ElasticSearch analyzer adapted to the specific language used. This means that searches are able to match related words (e.g., plural versions of a singular query word, or verb conjugations). Phrase searches are also possible (by using double quotes around the phrase). If a phrase (several words) is used as a query without the quotes, ElasticSearch interprets it as a query for any of the words, so it will return all data elements that contain any of the words in the query.

In Kibana, more sophisticated text searches can be made by switching Lucene query syntax: proximity queries (words close to each other), fuzzy searches (query words allowing typos), wildcards, etc.

MESSAGE_USR_NORM

This is a normalized version of MESSAGE_USR, in which the user text has been streamlined by:

• Converting all the sentence to lowercase
• Removing all punctuation
• Removing any extra spaces

Furthermore, this field is not processed through a language-dependent analyzer as MESSAGE_USR is, so queries on this field must match words exactly. It is still a text type field. However, the same query language can be used.

MESSAGE_AURA

This contains the text message generated by Aura and sent to the user as response to the user query. It is a text type field, so it is possible to search for specific words in it.

In the current version of Aura KPIs logs, this field only contains the text response. Some Aura use cases do not generate a purely textual message, but a more elaborated one (e.g., a card with text and graphics). These complex answers are inserted as attachments into Aura’s response to the channel and since attachments are not logged into the MESSAGE field, this field will appear empty in those cases. So, an empty MESSAGE_AURA field does not necessarily mean that Aura did not provide an answer. As an alternative for those situations, looking at the DIALOG_ID field (or INTENT) may give a hint of the type of answer that Aura delivered.

 MODALITY_CD_USR

This field contains the modality in which the user sent the message.

It is a slightly transformed field because there are some variations across Aura versions and, in order to unify it, the modalities are consolidated into only four different keywords: audio (spoken message), text (written free-text message) o form (commands sent via automatic processing or menus).

 DIALOG_ID

This field contains the identifier for the user case dialog module at the aura-bot Framework that was selected to construct the Aura response.

Dialog identifiers have two components (library  and dialog) separated by a colon e.g., services:service-usage

This field uses a custom analyser that splits the identifier at the colon, generating two terms. This makes possible to construct queries with one of the terms, e.g., “give me all the elements for the domain services”. But being a text field makes it impossible to do aggregations on it, so it cannot be used for statistics like bar charts (use DIALOG_ID.keyword for that).

DURATION_NU

This number reflects the time that took Aura to understand, process and respond to the user message. It is the difference (in milliseconds) between the timestamp of the moment the user message was received and the timestamp in which Aura’s response was finalized and sent to the channel.

Note that it is not a complete end-to-end delay time from the user’s point of view, since it does not include either the time it took the request to arrive to Aura through the channel or the time it took the response to travel back through the channel and get rendered at the client application (those times are outside Aura, and as such not registered by it).

Session Information

Session information includes the fields: sesId, sesSize, sesDuration.

These fields are generated by running a process over the time series formed by interactions from each user at each channel.

A session is automatically identified as a consecutive list of such user’s interactions, each separated from the next by a time interval shorter than 5 minutes. Once each session is identified, it is tabulated and labelled with three fields:

1.  sesId: string, forming a unique identifier for the session. It should be considered as an opaque identifier and the guarantee is that no other session in the data stream carries the same identifier.
   As an aside, interactions that do not correspond to actual user interactions (because no user could be identified or because the datapoint corresponds to an interaction not triggered by the user) are all labelled with a <void> sesId.

2.  sesSize: number of interactions this session contains. This is labelled only for the first interaction in the session, all other interactions carry a 0 in this field. Non-sessions such as the ones with <void> sesId will be left empty. This facilitates computing averages or other statistics on valid sessions, by just first filtering out all zero and empty values.

3.  sesDuration: time duration for each session, counted from the instant the first user message was received, to the instant the last Aura message was sent. For single-interaction sessions its value will be the same as DURATION_NU, for multiple interactions it will contain the time interval between all of them.

As with sesSize, only the first interaction in a session is annotated with sesDuration; the remaining interactions will be assigned a 0 value (and interactions that do not correspond to a session will be left empty). Therefore, to compute statistics on sesDuration, remove the 0 and empty values first.

userType

This field may be used, in certain cases, to help identify rows that do not correspond to real users but to test users (internal users that belong to test/QA teams and whose behaviour is, therefore, not representative of actual Aura users).

The field contains a single character, which is s for standard (real) users, and can be Q or T for QA/Test users respectively (there are also lowercased versions q and t, referring to unconfirmed test users).

Note that test user identification is not available on every country, since it depends on having a register of the AURA_GLOBAL_ID identifiers that QA/Test users authenticate and this is not always available.

usrMsgSig

This field is not useful by itself. Instead, it is intended to be used to help grouping together very similar user utterances. It does so by generating a signature of the utterance that is (hopefully) insensitive to small variations in the sentence.

This is an experimental field; it might change if we reach a variant that is better suited for its purpose.

The way to generate this signature is by following these steps with the utterance:

• Start with the normalized utterance (i.e., MESSAGE_USR_NORM).

• Perform stemming (removal of word suffixes) on all the words. This makes bills and bill the same word.

• Substitute words from a fixed list of very common, uninformative tokens (stopwords) by an asterisk. For example, this converts both “get my bill” and “get the bill” to the same phrase “get * bill”.

• Group words in sets of 3 elements (trigrams) and sort them alphabetically. This removes the global structure of the sentence, while retaining local structure.

The resulting string is a non-understandable version of the original utterance (hence, it cannot be used by itself), but the fact that several very similar utterances produce the same signature helps cluster those utterances. An example is one of the preinstalled visualizations “Most Frequent User Utterances” which uses this field to group very similar utterances.

Another example is provided in the following figure, which shows message utterances generating the same signature:

As it can be seen, the signature is the same for “how can I upgrade” and “when can I upgrade”, “when does my contract end” and “when is my contract ending”, and “live chat” & “live chats”. So, they would be counted together when aggregating by signature.

The procedure has its limitations and, as explained, it is experimental, so we are trying to improve it, but it can already alleviate a bit the inherent variability in user expressions.

AuraMsgGroup

Messages produced by Aura are as generated by its text resource database. In some cases, the same category of message produces different output texts, maybe because the message includes some user-dependent parameter or because the text database contains several variants of the same text (and Aura picks one at random).

The AuraMsgGroup field is a keyword field that helps categorize Aura answer by abstracting away some of this variation. It classifies the response given by Aura into two types of elements:

• Generic group: a name such as <NONE>, <GREETING> or <NOTFOUND>, which corresponds to a response category (see Table 3)

• Truncated answer: for answers that do not have a defined generic group, as a fallback the literal answer text is inserted, after substituting all numbers in it with a placeholder and truncating it (i.e., retain only the first characters).

The following table contains the generic groups defined so far. They correspond to the most frequent Aura messages. It is country-dependent, since it also depends on the use cases deployed in each country. As said above, responses not falling into these groups will be assigned a truncated version of the response text.

Note that th emost frequent Aura messages list can be enlarged with time. Also, the correspondence between Aura messages and groups is not static, if the text database is updated with new variants, it will be necessary to also update the translation table in the PPD cleaner process that generates this field.

: The list can be enlarged with time. Also, the correspondence between Aura messages and groups is not static, if the text database is updated with new variants, it will be necessary to also update the translation table in the PPD cleaner process that generates this field.

1.4 - Annex: Dataset fields

Annex: Dataset fields detail

Introduction

The objective of the following tables is to explain the process that each field is going through within this flow:

• Each cell of the table explains the process that the data field is undergoing in this specific moment before it gets to the concrete stage (table column).

• For example, the field GLOBAL_AURA_ID is undergoing a “hashing” before it gets stored in PPD_RAW. After this, the “hashed data” is progressed without any further processing to PPD_CLEAN.

Tables used in the Active Listening process are described in the following sections. They belong to the Aura Entities database.

• Columns “FIELD” and “DESCRIPTION”: instances managed by the OB

• Columns “PPD RAW” and “PPD CLEAN”: instances managed by Aura Global Team

MESSAGE dataset

Message dataset (stored in local Kernel)

• COLUMNS “field” and “description”: instances managed by the OB

• COLUMNS “PPD raw” and “PPD clean”: instances managed by Aura Global Team

RECOGNIZER dataset

Recognizer dataset stored in local Kernel

• Columns “FIELD” and “DESCRIPTION”: instances managed by the OB

• Columns “PPD RAW” and “PPD CLEAN”: instances managed by Aura Global Team

This Markdown table can be directly used in your GitHub Markdown files.  

API dataset

API request dataset (stored in local Kernel)

• Columns “FIELD” and “DESCRIPTION”: instances managed by the OB

• Columns “PPD RAW” and “PPD CLEAN”: instances managed by Aura Global Team

2 - Aura Analytics 2.0.0

Aura Analytics 2.0.0

What is Aura Analytics 2.0.0?

Active listening is defined as a key process that involves a continuous monitoring of Aura performance based on real logs from the users to analyze them and gather insights on the efficiency and effectiveness of Aura as a system and also to track the interaction of our users with Aura.

In this framework, Aura Analytics 2.0.0 is a tool used by Aura Global Team that uses active listening with the ultimate goal of improving Aura quality, as it generates accurate information to carry out both corrective and predictive actions and to decide how Aura should evolve in the future.

How does Aura Analytics 2.0.0 work?

• The process is built upon Aura users logs generated in production environment

• From these logs, Aura Analytics 2.0.0 create PPDs (Privacy-Preserving Datasets)

• Datasets are processed, enabling the visualization through dashboards and the extraction of statistical insights

• The Aura Global Team consumes this data to support decision-making processes

Target users

• The Aura Global Team is the target user of the Aura Analytics 2.0.0 tool, responsible for its design and management as well as for the interpretation of results for decision-making.

• OBs should allow the generation of datasets from their Aura users logs in their local environment just by installing and executing a single process, as shown in the document Guidelines for OBs.

Index of documents

Aura Analytics 2.0.0 includes the following documents:

• Version history
• Architecture overview and main processes and components
• Internal operation
• Guidelines for OBs willing to consume data provided by Aura Analytics 2.0.0
• Aura Analytics dashboard
• Annex: Dataset fields

Aura Analytics versions

Release 1.0.0

The first release 1.0.0. sets up the basic paths, deploys the PPD infrastructure and produce:

• Version 1.0.0. of the OB Analytics system, which includes the OB Dashboard.
• The first version of pre-processed datasets (clean PPDs) for training and analytics at Aura Global.

As mentioned, this version enables going further by:

• Enhancing the analytics dashboard with new visualizations.

• Processing the PPD files as desired (they are standard CSV files, which can be ingested in alternative platforms if desired). Restrictions on them are softer than on the original logs due to the anonymization process they have been subjected to, although they are still subjected to management precautions (a code of conduct is being prepared for that).

Release 1.1.0

Version 1.1.0. introduces the following changes:

• The table of data has been enlarged with these new fields: AURA_ID, STATUS_CD, sesId, sesSize, sesDuration.
• An expanded list of test users is used, so that the userType column contains more identifications.
• The code for data ingestion into a local Kibana, which previously consisted on a single Python script, has been turned into a full Python package to be installed, due to its increasing complexity.

Release 2.0.0

Version 2.0.0 introduces the following changes:

• In 2.0.0 version, Aura Analytics has undergone a refactor to improve its structure and make it easier to understand, maintain and extend in the future.
• Aura Analytics 2.0.0 simplifies the deployment and execution process.
• But one of the most significant enhancements in Aura Analytics 2.0.0 is its capability to manage both processed and to-process files centrally in one place (database). 

Prerequisites and recommended tools

The prerequisites for the use of Aura Analytics 2.0.0 are set below:

• Recommended tool for data visualization: ELK stack

2.1 - Architecture

Aura Analytics 2.0.0. architecture

Architecture overview

Aura Analytics 2.0.0 contains two different environments:

• OB local environment: Processes in this side are managed by the OB, that should install and execute certain processes related to the PPD-Creator for the creation of raw datasets.

• Global environment: Processes here are managed by Aura Global Team for data recovery, processing and generation of dashboards and statistics. The output includes data and metrics to be consumed by Aura Global Team for decision-making.

Aura Analytics 2.0.0 architecture flowchart

The following diagram shows an overview of Aura Analytics 2.0.0 architecture, including the environments involved and the main components and processes, which are fully described in succeeding sections.

Figure 1. Aura Analytics 2.0.0 Architecture flowchart

Aura Analytics 2.0.0 processes

PPD-Creator process

The PPD-Creator is a Python module for the creation of PPD-Raw datasets.

It is the only component that belongs to the OB environment. The OB should install it and is responsible for its execution. The PPD-Raw datasets will be stored in the destination blob PPD-RAW.

This process reads the files included in OB MANAGED INSTANCES columns of the tables in Annex: Dataset fields. The result of this process is the PPD RAW columns of the tables in the above-mentioned annex.

The main tasks executed by the PPD-Creator are summarized below:

• Reads the Aura log files in a Kernel Blob.
• Anonymize the sensible fields (AuraID, AuraGlobalID, and personal information of user sentence such as DNI, phone numbers, etc).
• Save the anonymized files to another directory of blob (PPD-Raw).

Figure 2. PPD-Creator process

The PPD-Creator anonymizes the following data, in the different OBs:

Manage PPD-Raw process

The Manage PPD-Raw process inserts the PPD-Raw path files (output from PPD-Creator) to PostgreSQL table for files management data centric:

• It reads the output of PPD-Creator JSON file
• Afterwards, it saves the paths to PostgreSQL server

Figure 3. Manage PPD-Raw process

PPD-Clean process

The PPD-Clean is a Python package used to clean PPD-Raw datasets.

Firstly, this process locates the directory where the PPD-Raw files are located, reads the corresponding files and processes them.

Once the process is finished, it writes to the files_processed table in the database and saves them in the PPD-Clean directory.

The main tasks executed by the PPD-Clean are summarized below:

• Apply transformations to columns
• Extract the explicit frustration
• Calculates the Nones n-grams
• Save the data in Directory or blob, PostgreSQL server and ElasticSearch for visualization

Figure 4. PPD-Clean process

User Dynamics process

User dynamics is a script used to measure the user’s behavior through metrics. It extracts statistics on the recurrence of users in Aura in a monthly basis.

The processes executed are summarized below:

• User dynamics reads the file_processed table of the database and the all PPD-Clean files stored for 1 month.
• It extracts metrics regarding new users, recurrent users, lost users and recovered users.
• Afterwards, it saves these metrics in the User_dynamics schema, in a PostgreSQL database, within the tables connections, daycount, user and channel.
• Data is also saved in ElasticSearch.

Figure 5. User Dynamics process

Components

Active Listening Database

The Active Listening Database is a PostgreSQL database that stores the processed and to-process files centrally in one place. It is used by the PPD-Clean and User Dynamics processes to store the processed data and metrics.

Schema files management

Currently, in the Active Listening project, we have input and output files for each of the processes and files that are processed. With the proposed database solution through the files management database, a more efficient management of raw files is achieved:

• The PPD-Creator process transfers files from the OB to a shared blob.
• The transferred files are written to a file in that blob called aura-sync-cache-dst.json.
• The manage_ppd_raw process will read the aura-sync-cache-dst.json file from the PPD-Raw folder and ingest the records into the FILE_PPD_RAW table of the database.
• It will also insert into the EMPTY_DATA_FILES table the days that are not found in the JSON file. This table is necessary for logging metrics in Prometheus. This process will run daily.

Figure 6. Files management database

Schema user dynamics

The User Dynamics process generates the statistics of Aura users, number of daily active users and types of users, with 4 categories: new, recurring, lost and recovered.

• The Channel table contains all the channels in Aura that have been processed by the User Dynamics process.
• The User table contains the unique Aura users in each environment and country.
• The Daycount table contains the number of total users for each day, indicating how many of them are new, recurring, recovered or lost users, the number of weekly unique recurring users and the number of monthly unique recurring users.
• The Connection table has the status of the user for each day (whether it is new, recurring, lost or recovered).

Figure 7. User dynamics database

Aura Analytics Dashboard

Aura Analytics 2.0.0 produces as a result, among other elements, an analytics component named Aura Analytics Dashboard that is the one used by Aura Global Team to gather statistics on the production system and to analyze user’s behavior.

This Analytics Dashboard is based on the ELK stack that contains:

• ElasticSearch: distributed search and analytics engine at the heart of the Elastic Stack. It allows the storage of data and its subsequent indexing, search and analysis.
• Kibana: provides a visualization tool that includes dashboards and panels created over the ElasticSearch data. Users interactively explore, visualize and share insights into data and manage and monitor the stack.

Once installed:

• An ElasticSearch index is created. It is called aura-ppd-ENTITY-COUNTRY-YEAR, and its index schema contains a cleaned version of the AURA MESSAGE, RECOGNIZER or API tables (which registers input and output messages).
• A Kibana index pattern is created, matching the uploaded ElasticSearch index.
• A pre-defined set of visualizations are installed in Kibana over that index pattern, as a means to get a default peek on the index data. See the section pre-installed analytics dashboard.
• The system automatically ingests any new clean PPD being produced in the ElasticSearch database, so that the index and dashboards remain up to date.

In principle, the PPD creation process specifies daily production, since Aura logs are sent to Kernel once a day. This means that information about Aura behavior and user actions on one given day will be available in the dashboards of the following day.

As mentioned above, the Aura Analytics Dashboard is conceived to be used by Aura Global Team. However, OBs can install locally the ELK stack or any other visualization tool for data consumption. Access to the document Local data visualization for further details.

2.2 - Operation

Aura Analytics 2.0.0 operation

Introduction

Based on Aura Analytics 2.0.0 architecture, the current documents provides an overview of its global operation.

Take the Aura Analytics 2.0.0 architecture flowchart as a reference to follow each step of the dataflow described below:

1. Aura logs generated in local instance are converted to datasets and transferred to local Kernel via the standard procedure and with the established frequency (typically, daily). Once there, the Active listening process flow fires up daily.

2. PPD-Creator: This is the first process that runs, and it is the only one executed in the OBs’ environment. It retrieves Kernel data, anonymizes all sensitive data that could identify users, and then transfers this data to an environment shared with the Aura Global team.

3. Manage-PPD-Raw: This is the first process executed from the global environment. It solely stores the paths of the data transferred by the PPD-Creator into a PostgreSQL database to keep a record of which data has been transferred.

4. PPD-Clean: This process runs from the global environment. Once the data is anonymized, it is processed to extract additional features (such as user frustration or the extraction of n-grams from user phrases about iterations that do not have an intent).

5. Once the data is processed, a path is saved in the environment and also in ElasticSearch to create dashboards that tracks Aura usage by its customers.

6. User-Dynamics: This is the last process, also executed in global environment. It is responsible for extracting statistics about users’ recurrence and the number of users per day. Among that, it identifies new users, recurring users (those making iterations every day), recovered users (those who have stopped using Aura at some point and have returned to the system) and lost users (those who have stopped using Aura in 3 months).

Examples of different dashboards are included below:

Figure 1. Users dashboard

Figure 2. Daily users dashboard

Figure 3. Weekly users dashboard

Figure 4. Trends dashboards

2.3 - Guidelines for OBs

Guidelines for OBs

Introduction

As seen in the Aura Analytics 2.0.0 architecture flowchart, Aura Analytics 2.0.0 contains two different environments: the OB local environment, managed by the OB and the Global one, managed by Aura Global Team.

Within this framework, the current guidelines are tailored towards OBs, indicating how to install and execute the PPD-Creator, for the creation and processing of PPD RAW datasets.

Once it is carried out, the subsequent processes of Aura Analytics 2.0.0 are executed in global environment by Aura Global Team.

Installation of PPD-Creator

The OB must install and store the PPD-Creator in a specific destination blob PPD-RAW and is responsible for its execution.

Guidelines are included in installer Aurak8s documentation: Active listening deployment.

Execution of PPD-Creator

The execution of the PPD-Creator must be done by the OBs, previous to its installation.

Parameters to launch the PPD-Creator

These are the parameters that the PPD-Creator takes from Kernel:

Mandatory parameters:

Optional parameters:

Launching PPD-Creator

Execute the following command:

docker run aura/ppd-creator --country <country-code> --anon-key <KEY> <source-params> <dest-params>

Example:

docker run aura/ppd-creator \
   --country ar \
   --anon-key as34-dre23-4127 \
   --src-name 4P-bucket-name-for-uk \
   --src-user EF45IHWD34DE4FGA \
   --src-pass k/Erf/6DSWWPjhdde1/abc123def-2331ldf \
   --dst-name aura-ppd-ar \
   --dst-user EF4341sdf3EFGUA1 \
   --dst-pass J/DQW/Sdde5k12ldsf/1abcde12dd1d-123c11 \
   --dst-encryption 1234ab56-12a3-45eb-8e06-8c522cdbb668-75f1b00f-6ca6-4a13-a741-64514cce728b \
   --table message \
   --environment prod \
   --transfer

Output from PPD-Creator

The output includes the following items:

• BOT_XXXXXX.txt.bz2: raw files (processed).

• log folder: if the logging options have been configured.

• aura-sync-cache-dst.json: table/month: processed files (automatically generated in destination). For example:

  {
    "AURA_DATA/ES/API/202212/": [
      "BOT_04095750-724e-11ed-9565-53054255c842_ES_API_20221202T150000Z.txt.bz2",
      "BOT_d2e93fc0-7656-11ed-a8eb-49a811568ab3_ES_API_20221207T170000Z.txt.bz2",
      "BOT_987780e0-7660-11ed-ba4a-2dac114c5321_ES_API_20221207T180000Z.txt.bz2"
    ],
    ...
  }
  

• aura-sync-cache-src.json: table/month: raw files_to_process (source). For example:

  {
  "AURA_DATA/ES/API/202212/": [
    "BOT_04095750-724e-11ed-9565-53054255c842_ES_API_20221202T150000Z.txt",
    "BOT_05a5b860-7663-11ed-bbf7-cb8fd9eb3c25_ES_API_20221207T190000Z.txt",
    "BOT_05ae43e0-7663-11ed-a0aa-8b7e0e134809_ES_API_20221207T190000Z.txt",
    "BOT_0d69fb10-7492-11ed-a1fc-95dce7e56901_ES_API_20221205T110000Z.txt"
  ],
    ... 
  }
  

• aura-sync-key-dst.json: key used to encrypt sensitive fields. For example:

  {
    "sample": "abcd1234-ab12-12ab-ab12-1abc234e56fg"
  }
  

Local data visualization (optional)

As explained before, Aura Global Team will be in charge of the analysis of the generated data through the global tool Aura Analytics Dashboard.

Nevertheless, just in case the OB wants to visualize certain data locally:

• This will be done following a prior agreement with the OB on privacy-related matters.
• Aura Global Team will provide access to the clean data stored in the corresponding PPD-clean blob container.
• The OB can install locally the ELK stack or other alternative tool for data visualization.
• No support will be offered by Aura Global team for this task.

2.4 - Analytics Dashboard

Aura Analytics 2.0.0. Dashboard

Aura Analytics 2.0.0 produces as a result, among other elements, an analytics component named Aura Analytics Dashboard that is the one used by Aura Global Team to gather statistics on the production system and to analyze user’s behavior. This Analytics Dashboard is based on the ELK stack.

The dashboards provides a pre-defined set of visualizations, described throughout this document. Nevertheless, it is possible to build additional dashboards using the ELK stack.

Pre-installed analytics dashboard

Kibana offers many possibilities to visualize the ingested data, and there are many resources and tutorials around explaining its mechanics. We therefore refer to the official Kibana documentation, or the many tutorials available on the Web, for generic information.

In the particular case of Aura Analytics 2.0.0, there is an ElasticSearch index that gets automatically ingested daily. It is called Aura-message-COUNTRY, and contains a cleaned version of the AURA MESSAGE table (which registers input and output messages).

Over this index, three types of panels/visualizations have been pre-installed:

• Discover panel
• Visualizations
• Dashboards

Discover

The Discover panel in Kibana is an essential tool for performing queries to an ElasticSearch index (save those searches, if desired), and explore users’ interactions with Aura in detail log by log, these being filtered by:

Search terms or conditions » A time interval » Additional filters applied to the query results » A set of index fields to show in the result table.

These 4 steps are represented in Figure 1:

Figure 1. Discover panel

As shown in this figure, the starting point is the ElasticSearch index holding all the data.

Each of the three first steps in the chain reduces the amount of data handled, by pruning out elements that do not satisfy the defined condition. The fourth step is just a display adjustment: on the final dataset, define which of the available fields will be shown on the output table that appears in the panel.

In the Aura Dashboard default set, there is one such Discover panel pre-installed. It is called question-answer pairs and has the following characteristics:

• A blank query (i.e., provide all the results)
• A time interval for the last 7 days
• A “only user” filter: filters out all intents that correspond to non-user queries (suggestions, help commands from the client application, etc)
• A visualization that includes: timestamp, (cleaned) user message, detected Aura intent, associated entities (if applicable), dialog that was invoked and Aura’s response.

Figure 2 shows a snapshot of this panel. To load it, select the Discover tool in the left navigation bar, and then click on the “Open” menu option in the top menu bar. A list of saved panels will be shown, with this one in it named “question-answer pairs”.

Figure 2. Question-answer pairs panel

Once the panel is loaded, each one of the aforementioned four elements can be freely modified, for example, the interface allows:

• Adding new filters with the “+Add Filters” button
• Deactivating the current filters by pressing over the predefined ones and clicking over the “Temporarily Disable” option
• Modifying the query interval with the “calendar” button or “Dates Box”
• Adding a specific query on a given index field(s) by using the “Search Box”, instead of the (default) blank query

Discover panels can be saved as named objects, to be later loaded at will. So, if needed, any panel (a modified panel or a newly created one) can be saved with a new name, to have it available for later loading.

Visualizations

A total of 7 visualizations come pre-installed with the base Aura Dashboard. The list can be obtained from the visualizations item in the left menu bar, shown in Figure 3:

• Three “Stats” type visualizations, which provide general statistics on platform usage.
• Four “User” type visualizations, which provide insights on user behavior.

Figure 3. Preinstalled visualizations dashboard

Note that this distinction between “User” and “Stats” is purely conceptual and based on the fields that have been used to generate the visualizations as from the point of view of Kibana, they are all regular visualizations.

Those visualizations can be instantly loaded by clicking on their names. But they can also be integrated into dashboards, which is described in the next section.

Dashboards

A dashboard in Kibana is essentially a spatial arrangement of visualizations. For example, to construct a dashboard, we just place visualizations into a page, resizing them as we wish, so we can observe all of them in a single place afterwards.

Within a dashboard all visualizations are linked. For example, if we change the time interval or add a filter using the interface, this modification affects all visualizations in the dashboard, and all of them get updated.

Elements in the dashboard visualizations can also generate “instant filters” by clicking on graphs or table elements. Those filters are added to the top of the page as a filter afterwards and can be modified or removed.

The Aura Analytics default installation preloads two dashboards. Those are available for selection when we click on the Dashboard icon in the left navigation bar:

Figure 4. Aura analytics default dashboards

Nones dashboard

This dashboard integrates the n-grams extracted from PPD-Clean process.

Figure 5. Nones dashboards

System dashboard

This dashboard integrates the three predefined “Stats” visualizations (generic statistics):

• A timeline of interactions (user messages sent and answered), segmented by channel
• A heatmap of interactions by weekday and time of day (hour)
• A bar graph classifying the interactions produced in the period by detected intent

Figure 6. System dashboard

User Dashboard

The User dashboard contains 4 user visualizations:

• Most Frequent User Utterances: list of the most frequent user sentences (in the time interval and filter active at the moment). It uses the msgUsrSig field to group together very similar utterances.
• AURA Answer Groups: list of the most frequent answers that Aura generates, grouped by the semantic categories in AuraMsgGroup field.
• Words per query: distribution of sizes for the user messages, measured as number of words in the utterance, and segmented by channel.
• Tag cloud: set of plain most frequent user utterances, as a tag cloud in which the font size represents the utterance frequency. The MESSAGE_USR_NORM field is used for the representation, so it contains normalized utterances.

The next screenshots show the dashboard with all these visualizations (it is a large dashboard, so typically it needs scrolling to visualize all its components).

Figure 7. User dashboard

Note that those four visualizations are linked in the sense of corresponding to the same subset of the data (as given by filters and time interval) but they are NOT linked at the individual item level (i.e., a given most frequent user utterance in the left table does not correspond to any specific Aura answer in the right bar graph).

Instead, the dashboard can be manipulated by selecting one specific item in any of the visualizations, and this will create a filter for the others.

For instance, as the following image shows, if we select “CHURN” in the Aura answer group visualization, we can observe in the others the user utterances that led Aura to generate that answer (i.e., an answer about contract cancelation).

Figure 8. Example of Aura answer groups in the user dashboard

Building new visualizations and dashboards

If the OB has installed locally the ELK stack, new elements can be built (or the current ones modified) by making use of the available fields in Kibana through the ingested ElasticSearch index.

In this section, we provide a reference of the schema that the index follows, so it can be used to build such new visualizations or to better understand the existing ones.

Data model

Field types

Elements in the Aura-message data model have 3 different types:

• Numeric: single numbers, integer or real. Suitable for numerical statistics, such as averages, or for plotting variation across time in graphs.
• Keyword: they are opaque strings, i.e., terms that cannot be searched within (it is not possible to look for words inside a keyword field). They can however be used to create some term-level queries, such as e.g., prefix queries (find all instances that begin with) and they usually work great for aggregations, since most of them are categorical variables (fields that only have a limited number of possible values) and can therefore be bucketed and counted.
• Text: these fields are divided into separate terms (words), and some pre-processing is done to them before indexing, to improve access, though an ElasticSearch analyzer. Text fields cannot be used in aggregated visualizations, since they cannot be grouped. They are most useful for queries, because they allow searching for fragments (only a few words) and fuzzy searches.

Field list

The following table lists all the available fields in the Aura-message-COUNTRY ElasticSearch index, with their type and a brief description. Some of them have more detailed explanations in Section Field explanations.

Note that some fields of text type have a mirror field of type keyword, with the same content. Having the same data indexed in two different ways at the same time (as text and as keyword) enables to perform different types of analysis by choosing the right field.

The Raw column indicates if this field is already present in the AURA raw PPD files:

• Yes: it is a field contained in raw PPDs.
• No: it is a generated field, produced when creating clean PPDs. They can be recognized as lowercase fields.
• Partial: It exists in the raw PPDs, but in a somehow different shape.

Field explanations

This subsection contains more detailed descriptions of some of the fields in the schema.

AURA_ID_GLOBAL

This element (mostly) uniquely identifies the user generating the interaction.

Note the concrete value of this field is not the same as the current identifier used in Aura and uploaded to Kernel: for privacy reasons, the identifier was hashed when generating the PPD and has no resemblance to the original one. The correspondence is however maintained across time, so it is possible to analyze user behavior.

The “mostly” qualifier reflects one quirk of the original Aura identifier: it is generated with a dependence to the authentication method used by the channel, so if two channels follow different authentication methods (e.g., MobileConnect vs. User/Password) then the AURA_ID_GLOBAL identifier for the same user will be different.

In summary:

• The identifier stays the same for a given user across time.
• No two users will have the same identifier.
• But the same user could produce two different identifiers if it connects to two channels that use a different authentication method.

AURA_ID

This is a “local” identifier, i.e., it is generated inside the channel according to the specific channel characteristics, and it is not tied as much as AURA_ID_GLOBAL to user authentication.

Its main disadvantage is its transient nature: the same user, on the same channel, could generate different AURA_ID strings when connecting different times, on different session. Therefore, for user accounting and tracing, AURA_ID_GLOBAL is usually preferred.

However, there are instances in which AURA_ID works better, namely for anonymous access (when the user is not authenticated). This depends on the channel:

• In the WhatsApp channel, the initial use of the channel will be anonymous from Aura side (i.e., no authentication is done), hence AURA_ID_GLOBAL will also be empty (at least until the user authenticates, which depends on the use case). But in this channel, AURA_ID has a permanent value, linked to the WhatsApp user, so here it is a good substitute for a persistent id even for unauthenticated users.

MESSAGE_USR

This field includes the message sent by user1. It has been partially processed to enhance anonymization by removing some standard identifiers contained in it with <idxxx> strings (e.g., phone numbers appear as <idphone>).

Removal is done mostly through regular expressions, so there might be occasional glitches (such as identifying as a phone number that does not really correspond to a phone, just because it follows the phone number pattern).

MESSAGE_USR is a field of text type. As such, it is searchable: it is possible to search for specific words the user might have said. Furthermore, it has been processed through an ElasticSearch analyzer adapted to the specific language used. This means that searches will be able to match related words (e.g., plural versions of a singular query word, or verb conjugations). Phrase searches are also possible (by using double quotes around the phrase).

In Kibana, more sophisticated text searches can be made by switching Lucene query syntax: proximity queries (words close to each other), fuzzy searches (query words allowing typos), wildcards, etc

MESSAGE_USR_NORM

This is a normalized version of MESSAGE_USR, in which the user text has been streamlined by:

• Converting all the sentence to lowercase
• Removing all punctuation
• Removing any extra spaces

Furthermore, this field is not processed through a language-dependent analyzer, as MESSAGE_USR is, so queries on this field must match words exactly. It is still a text field, however, so the same query language can be used.

MESSAGE_AURA

This contains the text message generated by Aura and sent to the user as response to the user query. It is a text field, so it is possible to search for specific words in it.

IMPORTANT In the current version of Aura KPI logs, this field contains only the text response.
Some Aura use cases do not generate a purely textual message, but a more elaborated one (e.g., a card with text and graphics). These complex answers are inserted as attachments into Aura’s response to the channel, and since attachments are not logged into the MESSAGE field, this field will appear empty in those cases.
So, an empty MESSAGE_AURA field does not necessarily mean that AURA did not provide an answer. As an alternative for those situations, looking at the DIALOG_ID field (or INTENT) may give a hint of the type of answer that Aura delivered.

MODALITY_CD_USR

This field contains the modality in which the user sent the message.

It is a slightly transformed field because there is some variation across Aura versions, and to unify the modalities are consolidated into only four different keywords: audio (spoken message), text (written free-text message) o form (commands sent via automatic processing or menus).

DIALOG_ID

This field contains the identifier for the user case dialog module at the Aura Bot Framework that was selected to construct the Aura response.

Dialog identifiers have two components (library and dialog) separated by a colon e.g., services:service-usage.

This field uses a custom analyzer that splits the identifier at the colon, generating two terms. This makes possible to construct queries with one of the terms, e.g., “give me all the elements for the domain services”). But being a text field makes it impossible to do aggregations on it, so it cannot be used for statistics like bar charts (use DIALOG_ID.keyword for that).

DURATION_NU

This number reflects the time that took Aura to understand, process and respond to the user message. It is the difference (in milliseconds) between the timestamp of the moment the user message was received and the timestamp in which Aura’s response was finalized and sent to the channel.

Note that it is not a complete end-to-end delay time from the user’s point of view, since it does not include either the time it took the request to arrive to Aura through the channel or the time it took the response to travel back through the channel and get rendered at the client application (those times are outside Aura, and as such not registered by it).

Session Information (sesId, sesSize, sesDuration)

These fields are generated by running a process over the time series formed by interactions from each user at each channel. A session is automatically identified as a consecutive list of such user’s interactions, each separated from the next by a time interval shorter than 5 minutes. Once each session is identified, it is tabulated and labelled with three fields:

1. sesId: a string, forming a unique identifier for the session. It should be considered an opaque identifier and the guarantee is that no other session in the data stream carries the same identifier.

As an aside, interactions that do not correspond to actual user interactions (because no user could be identified, or because the datapoint corresponds to an interaction not triggered by the user) are all labelled with a <void> sesId

2. sesSize: the number of interactions this session contains. This is labelled only for the first interaction in the session, all other interactions carry a 0 in this field. Non-sessions such as the ones with sesId will be left empty. This facilitates computing averages or other statistics on valid sessions, by just first filtering out all zero and empty values

3. sesDuration: the time duration for each session, counted from the instant the first user message was received, to the instant the last Aura message was sent. For single-interaction sessions its value will be the same as DURATION_NU, for multiple interactions it will contain the time interval between all of them.

As with sesSize, only the first interaction in a session is annotated with sesDuration; the remaining interactions will be assigned a 0 value (and interactions that do not correspond to a session will be left empty). Therefore, to compute statistics on sesDuration, remove the 0 and empty values first.

userType

This field may be used, in certain cases, to help identify rows that do not correspond to real users but to test users (internal users that belong to test/QA teams, and whose behavior is therefore not representative of actual Aura users). The field contains a single character, which is s for standard (real) users, and can be Q or T for QA/Test users respectively (there are also lowercased versions q and t, which means unconfirmed test users).

Note that test user identification is not available on every country, since it depends on having a register of the AURA_GLOBAL_ID identifiers that QA/Test users authenticate to, and this is not always available.

usrMsgSig

This field is not useful by itself. Instead, it is intended to be used to help grouping together very similar user utterances. It does so by generating a signature of the utterance that is (hopefully) insensitive to small variations in the sentence. The way to generate this signature is by following these steps with the utterance:

• Start with the normalized utterance (i.e., MESSAGE_USR_NORM)
• Perform stemming (removal of word suffixes) on all the words. This makes bills and bill the same word
• Substitute words from a fixed list of very common, uninformative tokens (stopwords) by an asterisk. For example, this converts both “get my bill” and “get the bill” to the same phrase “get * bill”
• Group words in sets of 3 elements (trigrams), and sort them alphabetically. This removes the global structure of the sentence, while retaining local structure.

The resulting string is a non-understandable version of the original utterance (hence it cannot be used by itself), but the fact that several very similar utterances produce the same signature helps to cluster those utterances. An example is one of the preinstalled visualizations, Most Frequent User Utterances, which uses this field to group very similar utterances.

Another example is provided in the following figure, which shows message utterances generating the same signature:

Figure 9. Message utterances generating the same signature

As it can be seen, the signature is the same for "how can I upgrade" and "when can I upgrade", "when does my contract end" and "when is my contract ending", and "live chat" & "live chats". So, they would be counted together when aggregating by signature.

The procedure has its limitations, and as explained is experimental, so we are trying to improve it, but it can already alleviate a bit the inherent variability in user expressions.

AuraMsgGroup

Messages produced by Aura are as generated by its text resource database. In some cases, the same category of message produces different output texts, maybe because the message includes some user-dependent parameter or because the text database contains several variants of the same text (and Aura picks one at random). The AuraMsgGroup field is a keyword field that helps categorizing Aura answer by abstracting away some of this variation. It classifies the response given by Aura into two types of elements:

• Generic group: a name such as <NONE>, <GREETING> or <NOTFOUND>, which corresponds to a response category (see Table 3)
• Truncated answer: for answers that do not have a defined generic group, as a fallback the literal answer text is inserted, after substituting all numbers in it with a placeholder and truncating it (i.e., retain only the first characters)

Table 4 contains the generic groups defined so far. They correspond to the most frequent Aura messages. It is country-dependent, since it also depends on the use cases deployed in each country. As said above, responses not falling into these groups will be assigned a truncated version of the response text.

EXPLICIT_FRUSTRATION

The sentiment model generates explicit frustration regarding the user’s message. In this field, the probability indicates that a user’s sentence is an explicit expression of frustration.

AllNGrams

For intents none and tv.none, an extraction of the most common n-grams generated by these none responses is applied. In these fields n-grams for 1 word, 2 words and 3 words are represented.

AllNGramsFilter

This field represents the AllNGrams field but filtered by stopwords.

NGrams1

This field represents the n-grams for 1 word.

NGrams1Filter

This field represents the n-grams for 1 word filtered by stopwords.

NGrams2

This field represents the n-grams for 2 words.

NGrams3

This field represents the n-grams for 3 words.

2.5 - Annex: Dataset fields

Annex: Dataset fields detail

Introduction

The objective of the following tables is to explain the process that each field is going through within this flow:

• Each cell of the table explains the process that the data field is undergoing in this specific moment before it gets to the concrete stage (table column).

• For example, the field GLOBAL_AURA_ID is undergoing a “hashing” before it gets stored in PPD_RAW. After this, the “hashed data” is progressed without any further processing to PPD_CLEAN.

Tables used in the Active Listening process are described in the following sections. They belong to the Aura Entities database.

• Columns FIELD and DESCRIPTION: instances managed by the OB

• Columns PPD RAW and PPD CLEAN: instances managed by Aura Global Team

MESSAGE dataset

Message dataset (stored in local Kernel).

GROOTMESSAGE dataset

Groot Message dataset (stored in local Kernel).

RECOGNIZER dataset

Recognizer dataset stored in local Kernel.

This Markdown table can be directly used in your GitHub Markdown files.

API dataset

API request dataset (stored in local Kernel).

3 - Aura Billing Module

Aura Billing Module

Introduction

The Aura Billing Module is a tool for the generation of Liceo invoices, that allow charging each customer for the services that she has used. This is a mandatory process for OBs.

It is based on the storage and processing of specific logs in the OB’s Aura systems to track the type and number of interactions of a user or service with Aura.

This information is used to assign costs based on different billing models and criteria chosen by the OB, which ultimately determines the total amount of the invoice.

The invoices will be generated in XLSX (Excel) format and stored in an Azure Storage Explorer blob container, along with the historical invoice records.

These invoices will be available for download by the Aura Global Team, to be sent to the OBs.

Interested in how the Aura Billing Module works and which are the tasks required to bring it into use? Access the document Aura Billing Module operation.

Generated Liceo invoices

The Liceo invoices generated by Aura Billing Module will contain the following information:

• Invoicing model (based on the payment model of the OB)
• Aura components used to provide the service
• Service/app that used this component
• Number of queries per component
• Cost of each query in each specific component
• Total amount generated by each component
• Total number of requests made during the billing period
• Total amount of the invoice

3.1 - Aura Billing Module operation

Aura Billing Module operation

Aura Billing Module operational flowchart

Figure 1 schematically shows how Aura Billing Module operates, where three different instances come into play:

• Aura: OB managed environment
• Aura: Global Team managed environment
• Kernel

Figure 1. Aura Billing Module operation

The operational processes executed by the Aura Billing Module are outlined below. In each step, the tasks that must be carried out by the OBs in order to bring it into use, are described.

1. Data generation

This task takes place in Aura’s OB managed environment.

• Aura components automatically generate logs every time a user/service interacts with Aura in local environment.

• These logs are pre-processed, cleaned and converted into datasets, in Avro format.

• These are the required Avro-formatted datasets for the Aura Billing Module:

  • Aura_Audit, that stores the minimum information needed for generating the Liceo invoices.​

  • Aura dimensional entities:

    • D_Aura_App schema definition: List of possible Apps defined in Aura.
    • D_Aura_Channel: List of possible channels defined in Aura.
    • D_Aura_Component: List of possible components defined in Aura.
    • D_Aura_Preset: List of possible presets defined in Aura.
    • D_Aura_Recognizer: List of possible recognizers defined by Aura.
    • D_Aura_Skill: List of possible skills defined in Aura.
      
      

  • Aura Gateway Message: summary of Aura interactions handled by aura-gateway-api.

2. Data publication in Kernel

• The latest versions of the previous Avro-formatted datasets must be published into Kernel productive environment by the Kernel team.

3. Kernel apps configuration to write/read datasets

• Two Kernel applications (clients) must be created/configured by the Kernel team to allow the use of Kernel resources:

  • aura-bot-[environment]: already existing app in Kernel
  • aura-billing-[environment]: new application

• Specifically, the applications must be configured with concrete scopes that provide permissions to write/read the datasets.

• The obligation to indicate the exact version in the configuration is removed. Therefore, in the following deployments, the version number indicated in the scope will be eliminated. For example, the configuration of the Brazil OB will have to be updated when a new scope change is made. For example: data:Aura_Audit:6:read —> data:Aura_Audit:read.

4. Data processing

Data processing is executed with Azure Databricks.

In this process, the information from the Kernel datasets is recovered and read by the Aura Billing Module, that uses algorithms to assign a unitary cost to each concept that composes the invoice to calculate the total amount of this invoice.

5. Data consumption

This step refers to the generation of the Liceo invoices and its storage in Azure Storage Explorer in xlsx format.

These invoices will be available for download by the Aura Global Team, to be sent to the OBs.

4 - Manage Aura logs

Manage Aura logs

Introduction

Logs are files that record specific single events, warnings and errors as they occur within a software environment. They can include contextual information, such as the time an event occurred and which user or endpoint was associated with it.

In Aura, logs are generated by specific components when an event happens and stored in order to monitor or debug the system.

Logs are stored in an ElasticSearch cluster.

Once stored, Aura integrates a logging system based on Kibana, which is the official tool to manage logs in Aura. Moreover, logs can be managed with Grafana and fluentd for specific features.

⚠️ You should not integrate third-party applications or scripts with ElasticSearch. These kinds of integrations are weak because the ElasticSearch API is not part of the public interface with the OB. This means that it could change without notice for several reasons such as updating the version of ElasticSearch or changing Aura internal architecture.

Manage logs in Kibana

The official Kibana User Guide is the reference guide to use Kibana.

Moreover, the current section includes certain useful points for managing Aura logs through this tool.

Policies in Kibana

Kibana includes index lifecycle policies.

By default, we add one policy for each index created (service and system index), to delete the logs older than seven days.

Snapshot in Kibana

Index snapshot is configured by default as long-term storage for the logs. These snapshots are taken daily and end in the cluster Azure Storage blob container (aura-backups/elk).

Manage logs in Grafana

Discover section

The “Discover” section in Grafana is very useful to look for logs and troubleshoot issues.

You can full-text search logs using Lucene query syntax.

Moreover, logs are tagged with many fields that can be useful to narrow down a search, such as:

• kubernetes.labels.app: name of the Kubernetes application that generated the log.
• kubernetes.pod_name: name of the Kubernetes pod that generated the log.
• corr: correlator that tracks E2E requests.
• lvl: log level (TRACE, DEBUG, INFO, WARN, ERROR or FATAL).

Queries that rely on a specific text are weak. Aura cannot guarantee that log messages do not change between versions. In fact, they do change. This is why metrics based on logs will not be reliable and it is not recommended to use Kibana to get metrics.

Manage logs in fluentd

Logs external forwarding feature

It is possible to send logs to an external system (a fluentd endpoint).

To enable this feature, add the following configuration to your config file:

external_forwarding:
  secret_shared_key: "mysecretkey"
  tls_config:
    tls_enabled: True
  remote_servers:
  - hostname: xxx
    port: yyy

• Set hostname and port fields with the remote endpoint. If you configure more than one remote server, fluentd load balances the traffic to them in a round-robin order.
• The hostname value can be an IP address, but it is not recommended if TLS is enabled. Turning off TLS is possible but discouraged for security reasons.
• secret_shared_key is used to verify client’s identity and must be configured properly in all the remote servers.

You can find additional information regarding receivers’ configuration (including TLS configuration and password authentication procedure) here.

5 - Manage metrics

Manage Aura metrics

Introduction

Metrics provide a measurement of certain data that represent a specific aspect of the monitored system at a point in time and offer an aggregated view over the system. They are useful to visualize long-term trends and alerts on log data.

Each Aura component is in charge of publishing its own metrics, which are typically generated at fixed-time intervals from aggregated logs.

Once generated, Aura metrics are pooled by Prometheus, which is in charge of gathering and exposing them.

Grafana is the most suitable tool to represent metrics through different dashboards. Each component counts on a Grafana dashboard to show its current behavior and there is a single dashboard for an Aura overview.

If you think a new metric could useful, please contact the Aura Platform Team, so it can be officially included as part of the platform.

The aim of this section is to explain both how Aura metrics work and all the metrics stored by each component.

⚠️ Saved dashboards, visualizations and queries are not guaranteed to be kept between upgrades because all the stack, including ElasticSearch and Grafana can be upgraded to newer versions.

Prometheus

Aura metrics system is based on Prometheus, a Cloud Native Computing Foundation project that works as systems and services monitoring system. Prometheus collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.

prom-client is being used to implement prometheus functionality in Node.js.

Prometheus service pools every component to get the metrics generated during the last time period. Every component counts on a private endpoint (not accessible from Internet) called /metrics where Prometheus requests the metrics.

Currently, the metric types used in this component are:

• Summary: similar to histogram metrics, it includes samples observations (such as request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

• Counter: cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart. For example, you can use a counter to represent the number of requests served, tasks completed, or errors.

• Gauge: similar to Counter, but it represents a single numerical value that can arbitrarily go up and down.

Prometheus-es-exporter

Working with Prometheus, we can create metrics using queries to ElasticSearch indexes (as well as create alarms, dashboard, etc) using prometheus-es-exporter.

This component is not deployed by default, but it can be enabled changing the variable prometheus_es_exporter_enabled to true in you config.yml file. (In Brazil, it is set to true by default). Access here the guidelines to enable prometheus-es-exporter component.

To config your own metrics from queries, write the new section, as in the following example, in your config.yml.

prometheus_es_exporter:
  query_blocks:
    ob:
      - name: "query_ob_br"
        QueryIntervalSecs: "60"
        QueryJson: '{"size":0,"query":{"bool":{"must":[],"filter":[{"bool":{"filter":[{"bool":{"should":[{"match_phrase":{"msg":"[AzureEventHub] emit"}}],"minimum_should_match":1}},{"bool":{"should":[{"match_phrase":{"kubernetes.labels.app":"aura-bot"}}],"minimum_should_match":1}}]}},{"range":{"@timestamp":{"gte":"now-1m","lte":"now"}}}]}}}'
        QueryIndices: "aurak8s-service-*"

Where:

• name: Mandatory. Name of the query. It must start with query_*
• QueryIntervalSecs: Optional. It indicates how often to run queries in seconds. By default, 60.
• QueryJson: Mandatory. The search query to run.
• QueryIndices: Optional. Indices to run the query on. Any way of specifying indices supported by your ElasticSearch version can be used. By default, _all. Although this field is optional, it is highly recommended to delimit the search query.

Aura components metrics

The main Aura components can generate their own metrics.

Select your intended component in the left menu and access to its details.

5.1 - Aura Bot metrics

Aura Bot metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-bot.

It is stored as a Summary in Prometheus. So every sample, besides the defined labels, also includes its duration.

It measures the duration since the request lands in aura-bot until its HTTP response is returned, indicating to the client that Aura is processing the request to obtain a proper answer for the user.

The metric allows measuring the behavior of the requests from any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response

This metric was stored since Iron Maiden (7.2.0) release.

outgoing_request_duration_seconds

This metric is intended to store the information related to all the outgoing HTTP requests made by aura-bot.

It is stored as a Summary in Prometheus so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

This metric was stored since Camela (5.0.0) release.

outgoing_message_duration_seconds

This metric is intended to store the number of Direct Line requests arriving to aura-bot.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-bot is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the user is sent back to the client callback. This metric measures the duration since the request lands in aura-bot until the last message of its answer is sent to the client callback.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status code returned in the response.
• originStatus: status sent by Direct Line in the body of the response in the happening of an error.
• origin: specific host of the request.
• channel: channel of the request.

This metric was stored since Iron Maiden (7.2.0) release.

aura_component_version

This metric is intended to store the number of aura-bot instances (pods) running each version of the code. It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

This metric was stored since Camela (5.0.0) release with the name of bot_version and updated to aura_component_version in Iron Maiden (7.2.0).

bot_request_version

This metric is intended to store the number of incoming requests to aura-bot depending on their channelData.version. It is stored as a Counter in Prometheus.

Labels:

• version: channelData.version in the incoming request. If the incoming request has no version field, 1 will be set.

This metric was stored since Iron Maiden (7.2.0) release.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-bridge.

It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

This metric was stored since Iron Maiden (7.2.0) release.

aura_token_generate

This metric is intended to store the information related to Kernel accessToken refreshments in aura-bridge. It is intended to make it possible to set an alarm in the happening of any error during refresh of the 2-legged accessToken needed to access Kernel WhatsApp APIs.

It is stored as a Summary.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status returned by Kernel in the response.
• originStatus: status sent by Kernel in the body of the response in the happening of an error.
• origin: channelId of the channel that needs the accessToken in Aura.
• channel: channel of the request.

This metric was stored since Iron Maiden (7.2.0) release.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

5.2 - Aura Groot metrics

Aura Groot metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-groot.

It is stored as a Summary in Prometheus. So every sample, besides the defined labels, also includes its duration.

It measures the duration since the request lands in aura-groot until its HTTP response is returned, indicating to the client that Aura is processing the request to obtain a proper answer for the Direct Line or aura-bridge.

The metric allows measuring the behavior of the requests from any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response

outgoing_request_duration_seconds

This metric is intended to store the processing time related to all the outgoing HTTP requests made by aura-groot.

It is stored as a Summary in Prometheus so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

outgoing_message_duration_seconds

This metric is intended to store the processing time of Direct Line or aura-bridge requests arriving to aura-groot.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-goot is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the user is sent back to the client callback. This metric measures the duration since the request lands in aura-groot until the last message of its answer is sent to the client callback.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status code returned in the response.
• originStatus: status sent by Direct Line in the body of the response in the happening of an error.
• origin: specific host of the request (Direct Line or aura-bridge).
• channel: channel of the request.

incoming_message_duration_seconds

This metric is intended to store the processing time of Direct Line, aura-bridge or skills requests arriving to aura-groot.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-goot is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the channel or skill is sent back to the client callback. This metric measures the duration from when the request arrives at aura-groot until it is processed to send to the channel/bridge or skill.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status code returned in the response.
• originStatus: status sent by Direct Line in the body of the response in the happening of an error.
• origin: specific host of the request (Direct Line, aura-bridge or skill name). If origin is missing, the content of path label will be added.
• channel: channel of the request.

aura_component_version

This metric is intended to store the number of aura-groot instances (pods) running each version of the code. It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-groot.

It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

skill_access_error

This metric is intended to store the number of times a skill has been misconfigured in aura-groot.

It is stored as a Counter in Prometheus.

Labels:

• skill: skill name.
• code: noRespond or noFound
• channel: channel of the request.

skill_request_status

This metric is intended to store the number of times we have obtained a response status per skill in aura-groot.

It is stored as a Counter in Prometheus.

Labels:

• skill: skill name.
• code: status code of the request.
• channel: channel of the request.

skill_response_error

This metric is intended to store the number of times a skill has been blocked in aura-groot.

It is stored as a Counter in Prometheus.

Labels:

• skill: skill name
• code: blocked
• channel: channel of the request.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

5.3 - Atria Model Gateway metrics

Atria Model Gateway metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by atria-model-gateway.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests from any given endpoint. Specifically, the duration since the request lands in atria-model-gateway until its HTTP response is returned:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response
• application: application name that is using the model

outgoing_request_duration_seconds

This metric is intended to store the information related to all the outgoing HTTP requests made by atria-model-gateway. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

The metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

generative_tokens

This metric is intended to store the information related to tokens used by OpenAI in atria-rag-server. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its tokens usages.

The metric allows measuring the behavior of the tokens using any given OpenAI model:

• The number of tokens during a time
• The average/min/max tokens of these requests

Labels:

• application: application name that is using the model
• deployment_model_name: name of the deployment model
• model_type: identifier of the model

5.4 - Atria RAG server metrics

Atria RAG server metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by atria-rag-server.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests from any given endpoint. Specifically, the duration since the request lands in atria-rag-server until its HTTP response is returned:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response
• application: application name that is using the model

outgoing_request_duration_seconds

This metric is intended to store the information related to all the outgoing HTTP requests made by atria-rag-server. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

The metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

5.5 - Aura Authentication API metrics

Authentication API metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-authentication-api. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests from any given endpoint. Specifically, the duration since the request lands in aura-authentication-api until its HTTP response is returned:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.).
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response

This metric was stored since Greenday (6.0.0) release.

outgoing_request_duration_seconds

This metric is intended to store the information related to all the outgoing HTTP requests made by aura-authentication-api. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

The metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

This metric was stored since Camela (5.0.0) release.

aura_token_generate

This metric is intended to store the information related to Kernel accessToken generation, used during the integrated authorization process of the Aura users in aura-authentication-api.

It is intended to make it possible to set an alarm in the happening of any error during token validation. It is stored as a Summary in Prometheus.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status returned by Kernel in the response.
• originStatus: status sent by Kernel in the body of the response in the happening of an error.
• origin: channelId of the channel that needs the accessToken in Aura.

This metric was stored since Iron Maiden (7.2.0) release.

aura_component_version

This metric is intended to store the number of aura-authentication-api instances (pods) running each version of the code.

It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

This metric was stored since Barricada (5.3.0) release with the name of authentication_api_version and updated to aura_component_version in Iron Maiden (7.2.0).

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-bridge. It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

This metric was stored since Iron Maiden (7.2.0) release.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

5.6 - Aura Configuration API metrics

Aura Configuration metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-configuration-api.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests from any given endpoint. Specifically, the duration since the request lands in aura-configuration-api until its HTTP response is returned:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response

This metric was stored since Greenday (6.0.0) release.

outgoing_request_duration_seconds

This metric is intended to store the information related to all the outgoing HTTP requests made by aura-configuration-api. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

The metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

aura_component_version

This metric is intended to store the number of aura-configuration-api instances (pods) running each version of the code.

It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-configuration-api. It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

5.7 - Aura Gateway API metrics

Gateway API metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-gateway-api.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests from any given endpoint. Specifically, the duration since the request lands in aura-gateway-api until its HTTP response is returned:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response
• application: Application name of the request.
• channel: Channel name of the request. Only for NLPaaS endpoint.
• preset: Preset name of the request. Only for Generative endpoint.

outgoing_request_duration_seconds

This metric is intended to store the information related to all the outgoing HTTP requests made by aura-gateway-api. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

The metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

aura_component_version

This metric is intended to store the number of aura-gateway-api instances (pods) running each version of the code.

It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

This metric was stored since Beatles (8.9.0) release.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-gateway. It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

5.8 - Aura Bridge metrics

Aura Bridge metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-bridge. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

It measures the duration since the request lands in aura-bridge until its HTTP response is returned, indicating to the client that Aura is processing the request to obtain a proper answer for the user.

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response

This metric allows measuring the behavior of the requests from any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

This metric was stored since Greenday (6.0.0) release.

outgoing_message_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-bridge.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-bridge is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the user is sent back to the client callback.

This metric measures the duration since the request lands in aura-bridge until the last message of its answer is sent to the client callback.

Labels:

• host: host and domain where the request is being sent.
• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• path: specific endpoint of the request.
• originStatus: third party status sent in the body of the response. Usually, this status is sent by whatsapp.
• status: HTTP status code returned in the response.
• origin: specific source of the request. The value could be: ‘4p’, ‘whatsapp’, ‘aura-bot’ or ‘genesys’.
• channel: channel of the request.

This metric allows measuring the behavior of the requests from any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

This metric was stored since Greenday (6.0.0) release.

incoming_message_duration_seconds

This metric is intended to store the number requests arriving to aura-bridge from a channel or Direct Line.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-bridge is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the channel or Direct Line is sent back to the client callback. This metric measures the duration from when the request arrives at aura-bridge until it is processed to send to the channel or Direct Line.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status code returned in the response.
• originStatus: status sent by Direct Line or channel in the body of the response in the happening of an error.
• origin: specific host of the request. If origin is missing, the content of path label will be added.
• channel: channel of the request. In Auraline requests used to get conversationId with path: /aura-services/v1/auraline/conversations, channel will be missing.

aura_response_ack_duration_seconds

This metric is intended to store the information related to all the ACK requests sent by the clients to aura-bridge. The ACK requests are used by the clients (WhatsApp) to notify if in the end Aura’s answer was delivered to the user or not.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration. The duration measures since the ACK request lands in aura-bridge until its asynchronous answer is sent to the user.

Labels:

• host: host and domain where the request is being sent.
• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• path: specific endpoint of the request.
• originStatus: third party status sent in the body of the response. Usually, this status is sent by whatsapp.
• status: HTTP status code returned in the response.
• origin: specific source of the request. The value could be: ‘4p’, ‘whatsapp’, ‘aura-bot’ or ‘genesys’.
• channel: channel of the request.

This metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

This metric was stored since Heroes (7.0.0) release.

outgoing_request_duration_seconds

This metric is intended to store the information related to all the outgoing HTTP requests made by aura-bridge. It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, …)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

This metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

This metric was stored since Greenday (6.0.0) release.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-bridge. It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

This metric was stored since Iron Maiden (7.2.0) release.

aura_token_generate

This metric is intended to store the information related to Kernel accessToken refreshments in aura-bridge. It is intended to make it possible to set an alarm in the happening of any error during refresh of the 2-legged accessToken needed to access Kernel WhatsApp APIs.

It is stored as a Summary in Prometheus.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status returned by Kernel in the response.
• originStatus: status sent by Kernel in the body of the response in the happening of an error.
• origin: channelId of the channel that needs the accessToken in Aura.

This metric was stored since Iron Maiden (7.2.0) release.

aura_component_version

This metric is intended to store the number of aura-bridge instances (pods) running each version of the code.

It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

This metric was stored since Greenday (6.0.0) release with the name of aura_bridge_version and updated to aura_component_version in Iron Maiden (7.2.0).

aura_bridge_wa_incoming_message

This metric is intended to store the number of unhandled errors happening in aura-bridge. It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

This metric was stored since Iron Maiden (7.2.0) release.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

5.9 - Aura KPIs uploader metrics

Aura KPIs Uploader

aura_kpis_uploader_metrics_duration

This KPI measures the time required by aura-kpis-uploader to process each type of KPI. KPI management has several steps (load, process, upload), and this KPI represents the time it takes to perform all those steps for each of the KPIs defined in AURA_SOURCE_PATH_AVRO_ADAPTERS.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

Labels:

• format: File format in which the KPI will be stored.
  • csv: File format will be CSV (deprecated).
  • avro: File format will be AVRO.
• kpiType: Type of KPI:
  • entity: KPI is of type Entity.
  • dimensional: KPI is of type Dimensional.
• kpiName: Name of the KPI.
• duration: Time in seconds with the time used to process the KPI.
• numberFilesProcessed: Number of KPIs processed. If the format is AVRO, it represents the number of records processed. If the format is CSV, it only represents the number of processed files.

aura_kpis_uploader_metrics

This metric is intended to store the information related to all processes executed by aura-kpis-uploader. It is stored as a Counter in Prometheus, so every sample, besides the defined labels.

This KPI measures the amount of KPI registers processed, if the format is AVRO it represents the number of records processed. If the format is CSV, it only represents the number of processed files.

Labels:

• format: File format in which the KPI will be stored.
  • csv: File format will be CSV (deprecated).
  • avro: File format will be AVRO.
• kpiType: Type of KPI:
  • entity: KPI is of type Entity.
  • dimensional: KPI is of type Dimensional.
• kpiName: Name of the KPI.
• duration: Time in seconds with the time used to process the KPI.
• numberFilesProcessed: Number of KPIs processed. If the format is AVRO, it represents the number of records processed. If the format is CSV, it only represents the number of processed files.

aura_kpis_uploader_errors

This metric is intended to store the information related to all errors generated by execution of aura-kpis-uploader. It is stored as a Counter in Prometheus, so every sample, besides the defined labels.

This KPI measures the amount of KPI errors produced when generating KPIs.

Labels:

• type: Name of the method or function where the error occurred.
• format: File format in which the KPI will be stored.
  • csv: File format will be CSV (deprecated).
  • avro: File format will be AVRO.
• kpiType: Type of KPI:
  • entity: KPI is of type Entity.
  • dimensional: KPI is of type Dimensional.
• kpiName: Name of the KPI.
• url: If the error contains a file with more information stored in Azure Storage, this field contains the URL to download the file.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-kpis-uploader. It is stored as a Counter in Prometheus.

Labels:

• error: Exception message that forced the unhandled error.

aura_server_unhandled_error is stored from Loquillo (7.5.0) release onwards.

5.10 - Aura NLP metrics

Aura NLP metrics

http_request_duration_seconds

This Prometheus metric is modelled as a summary where its value is the spent time until the remote host responds to an HTTP request.

Note that the value is a float number rounded to its third decimal. It is stored as a Summary in Prometheus.

This metric is intended to store the duration of outgoing requests in seconds.

Labels:

All label values are strings.

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.).
• path: HTTP path of the incoming request.
• status_code: the responded HTTP status code (as a string).

Value:

• Request duration in seconds.

outgoing_request_duration_seconds

This Prometheus metric is a modelled as a summary where the value is the spent time until the remote host responds to an HTTP request.

Note the value is a float number rounded to its third decimal. It is stored as a Summary in Prometheus.

This metric is intended to store the duration of outgoing requests in seconds.

Labels:

All label values are strings.

• method: HTTP method (GET, POST; etc.), a string in uppercase.
• host: remote host that will receive the outgoing request.
• path: HTTP path of the outgoing request.
• status: the responded HTTP status code (as a string).

5.11 - T&C API metrics

Terms & Conditions API metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests handled by tac-api. It is stored as a Histogram in Prometheus, so every sample, besides the defined labels, also includes its duration.

It measures the duration since the request lands in tac-api until its HTTP response is returned.

This metric allows measuring the behavior of the requests from any given endpoint:

• The number of requests during a period of time
• The average/min/max duration of these requests
• Quantiles of the duration and the number of requests in a period

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response

This metric was stored since Barricada (5.0.0) release.

http_requests_total

This metric is intended to store information about all the request handled by tac-api. It is stored as a Counter in Prometheus.

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• status_code: HTTP status code returned in the response.

This metric allows measuring the behavior of the requests from any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests
• Quantiles

This metric was stored since Barricada (4.0.0) release.

http_in_flight_requests_total

This metric is intended to store the information related to all the concurrent HTTP requests being handled by tac-api in a period.

It is stored as a Gauge in Prometheus because it is a value that can go up and down at every moment.

This metric allows to measure the behavior of the requests from any given endpoint:

• The number of requests during a period of time
• The average/min/max duration of these requests
• Quantiles of the duration and the number of requests in a period.

This metric was stored since Barricada (4.0.0) release.

tac_internal_errors

This metric is intended to store the number of internal errors happening in tac-api. It is stored as a Counter in Prometheus because its value can only go up.

Labels:

• name: it will contain the exception message that forced the unhandled error.

This metric was stored since Barricada (4.0.0) release.

tac_service_acceptances_total

This metric is intended to store the number of acceptances of Terms and Conditions per service handled by tac-api. It is stored as a Counter in Prometheus because its value can only go up.

Labels:

• name: it will contain the name of the accepted service. Currently, it could contain one of: aura, whatsapp-anonymous, whatsapp-authenticated
• version: T&C version accepted by the user

This metric was stored since Barricada (4.0.0) release.

tac_service_updates_total

This metric is intended to store the number of updates of terms and conditions per service handled by tac-api. It is stored as a Counter in Prometheus because its value can only go up.

Labels:

• name: name of the updated service. Currently (Iron Maiden) it could contain one of: aura, whatsapp-anonymous, whatsapp-authenticated
• version: T&C version updated by the user

This metric was stored since Barricada (4.0.0) release.

tac_user_deletions_total

This metric is intended to store the number of deletions of terms and conditions per service handled by tac-api. It is stored as a Counter in Prometheus because its value can only go up.

This metric was stored since Barricada (4.0.0) release.

aura_component_version

This metric is intended to store the number tac-api instances (pods) running each version of the code. It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

This metric was stored since Iron Maiden (7.2.0).

5.12 - NLP provisioning metrics

NLP Provisioning metrics

Introduction

In the Aura NLP provisioning component, it is important to know in each moment the quantity of processes restarted in relation with the total processes that, at this moment, work to process the different container. In that way, it could be alerted to an abnormal performance and take measures in this regard.

http_request_duration_seconds

This Prometheus metric is modelled as a summary where its value is the spent time until the remote host responds to an HTTP request.

Note that the value is a float number rounded to its third decimal. It is stored as a Summary in Prometheus.

This metric is intended to store the duration of outgoing requests in seconds.

Labels:

All label values are strings.

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.).
• path: HTTP path of the incoming request.
• status_code: the responded HTTP status code (as a string).

Value:

• Request duration in seconds.

nlp_provisioning_killed_processes

This metric is intended to store the number of processes killed in each iteration of the Aura NLP provisioning execution. It is stored as a Gauge in Prometheus.

Value:

• Number worker processes killed in each iteration

nlp_provisioning_alive_processes

This metric is intended to store the number worker processes alive in each iteration of NLP Provisioning. It is stored as a Gauge.

Value:

• Total alive processes.

nlp_provisioning_expected_alive_processes

This metric is intended to store the number of expected alive processes in the NLP Provisioning. It is stored as a Gauge.

Value:

• Set gauge with total alive processes.
• Decrease gauge with finished processes.

nlp_provisioning_container_killed_count

This metric is intended to store the counter of all the processes killed in Aura NLP provisioning. It is stored as a Counter in Prometheus.

Labels:

• container: container URL.

Value:

• Dead process ids (pids).

5.13 - Aura Complex Logic metrics

Aura Complex Logic metrics

http_request_duration_seconds

This Prometheus metric is modelled as a summary, where its value is the spent time until the remote host responds to an HTTP request.

Note that the value is a float number rounded to its third decimal. It is stored as a Summary in Prometheus.

This metric is intended to store the duration of outgoing requests in seconds.

Labels:

All label values are strings.

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.).
• path: HTTP path of the incoming request.
• status_code: the responded HTTP status code (as a string).

Value:

• Request duration in seconds

supervised_complex_logic_app_restarted_counter

This metric is intended to store a count of the restarted plugins.

It is stored as a Counter in Prometheus.

Labels:

All label values are strings.

• app: clf
• supervised_plugin: Supervised plugin class path.
• plugin_status: Plugin response code status.
• plugin_handler_name: Handler name.

complex_logic_app_http_requests

This metric is intended to store the HTTP requests of Aura Complex Logic plugins.

It is stored as a Counter in Prometheus.

Labels:

All label values are strings.

• app: clf
• plugin: plugin class path.
• status_code: plugin response code status.
• handler_name: handler name.

5.14 - Aura Context metrics

Aura Context metrics

http_request_duration_seconds

This Prometheus metric is modelled as a summary where its value is the spent time until the remote host responds to an HTTP request.

Note that the value is a float number rounded to its third decimal. It is stored as a Summary in Prometheus.

This metric is intended to store the duration of outgoing requests in seconds.

Labels:

All label values are strings.

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.).
• path: HTTP path of the incoming request.
• status_code: the responded HTTP status code (as a string).

Value:

• Request duration in seconds.

database_request_duration_seconds

This metric is intended to store the duration of database requests in seconds.

It is stored as a Summary in Prometheus.

Labels:

All label values are strings.

• database: database name (Redis or Mongo).
• operation: database operation (i.e., update, create, get_by_date, get_last_n, get_by_corr).

Value:

• Request duration in seconds.

5.15 - Aura File Manager metrics

Aura File Manager metrics

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-file-manager.

It is stored as a Summary in Prometheus. So every sample, besides the defined labels, also includes its duration.

It measures the duration since the request lands in aura-file-manager until its HTTP response is returned, indicating to the client that Aura is processing the request to obtain a proper answer.

The metric allows measuring the behavior of the requests from any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status_code: HTTP status code returned in the response

outgoing_request_duration_seconds

This metric is intended to store the processing time related to all the outgoing HTTP requests made by aura-file-manager.

It is stored as a Summary in Prometheus so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests to any given endpoint:

• The number of requests during a time
• The average/min/max duration of these requests

Labels:

• method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
• host: host and domain where the request is being sent
• path: specific endpoint of the request
• status: HTTP status code returned in the response

outgoing_message_duration_seconds

This metric is intended to store the processing time of aura-bot requests arriving to aura-file-manager.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-file-manager is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the user is sent back to the client callback. This metric measures the duration since the request lands in aura-file-manager until the last message of its answer is sent to the client callback.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status code returned in the response.
• origin: aura-bot

incoming_message_duration_seconds

This metric is intended to store the processing time of aura-bot requests arriving to aura-file-manager.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-file-manage is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the channel or skill is sent back to the client callback. This metric measures the duration from when the request arrives at aura-file-manager until it is processed to send the response.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status code returned in the response.
• originStatus: status sent in the body of the response in the happening of an error.
• origin: aura-bot

aura_component_version

This metric is intended to store the number of aura-file-manager instances (pods) running each version of the code. It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-file-manager.

It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

aura_token_generate

This metric is intended to store the processing time of aura-file-manger to get/refresh kernel token.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

Labels:

• path: specific endpoint of the request.
• httpStatus: HTTP status code returned in the response.
• originStatus: status sent by Direct Line in the body of the response in the happening of an error.
• origin: kernel client identifier

file_validation_duration_seconds

This metric is intended to store the validation time of a file.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

Labels:

• path: specific endpoint of the request.
• code: OK when file is valid.
• origin: specific endpoint of the request.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

5.16 - Aura Redis MongoDB sync metrics

Aura Redis MongoDB Synchronizer metrics

aura_component_version

This metric is intended to store the number of aura-bot instances (pods) running each version of the code. It is stored as a Gauge in Prometheus.

Labels:

• version: version field in the package.json file included in the running docker container.
• component: name of the component that is writing the metric.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-redis-mongo-sync.

It is stored as a Counter in Prometheus.

Labels:

• error: exception message that forced the unhandled error.

redis_mongo_sync_duration_milliseconds

This metric measures the data upload time from the service to the Mongo database.

It is stored as a Histogram in Prometheus. So every sample, besides the defined labels, also includes its duration.

The aura-redis-mongo-sync service contains a data collector that helps the event service move stale data from Redis to MongoDB. This collector sends the data in packets to optimize performance. This metric measures the time MongoDB takes to process the packet.

Labels:

• status: HTTP status returned in the response. Values: success.
  • success: if the status is success, the time is stored.

redis_mongo_synced_items_total

This metric is intended to store the registers synchronized between Redis and MongoDB by events.

It is stored as a Counter in Prometheus.

Labels:

• type: register type. Values: event, active_context
  • event: Items synchronized by event.
  • active_context: Items synchronized by active context process.

redis_mongo_synced_errors

This metric is intended to store the errors that have occurred in the synchronization.

It is stored as a Counter in Prometheus.

Labels:

• error: Values : create, syncData, executeBulk.
  • create: If the error occurred when creating the service.
  • syncData: If the error occurred when synchronizing the data.
  • executeBulk: If the error occurred when uploading the data to MongoDB in bulk mode.

redis_mongo_sync_configuration_settings

This metric contains the service configuration data.

It is stored as a Gauge in Prometheus.

Labels:

• setting_name: Values: shard_count, pod_count, active_context_ttl_seconds, redis_cache_ttl_seconds.
  • shard_count: Current shard used to distribute the data to synchronize between pods.
  • pod_count: Current number of services of aura-redis-mongo-sync.
  • active_context_ttl_seconds: Time interval to run the data collector.
  • redis_cache_ttl_seconds: Time in seconds that will be set to the context elements in the Redis cache.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

• moduleId: Id of the module.
• status: OK or ERROR

6 - Aura dashboards

Aura dashboards

Introduction

Dashboards are reporting tools that aggregate and display metrics and key indicators, so they can be examined at a glance by all possible audiences.

These dashboards allow data interpretation and provide an overall view for the evaluation of Aura’s performance, thus improving decision-making. Each component counts on a dashboard to show its current behavior and there is a single dashboard for an Aura overview.

There are two types of dashboards for Aura metrics (Prometheus) that are generated in Grafana:

• Aura components dashboards

• Aura system dashboards

6.1 - Aura system dashboards

Aura system dashboards

Introduction

Currently, these are the available Aura system dashboards in Grafana based on metrics stored in Prometheus:

• Alertmanager dashboard
• Elasticsearch dashboard
• Fluent bit dashboard
• Kubernetes cluster monitoring dashboard
• Kubernetes cron and batch monitoring dashboard
• Kubernetes nodes dashboard
• Kubernetes services dashboard
• NLP provisioning dashboard
• Prometheus stats dashboard
• Redis dashboard

6.1.1 - Alertmanager dashboard

Alertmanager dashboard

Panels

Received alerts rate

It shows a time series with the received alerts rate aggregated by one minute.

The x-axis shows the time series and the y-axis shows received alerts rate.

The queries used to get the panel information are:

sum(rate(prometheus_notifications_alertmanagers_discovered[1m])) by(status)

An example of this panel is shown below:

The available metrics are defined in the following sections.

Successful notification rate

It shows a time series with the successful notifications rate aggregated by one minute.

The x-axis shows the time series and the y-axis shows the successful notifications rate.

The queries used to get the panel information are:

sum(rate(prometheus_notifications_sent_total[1m])) by(integration)

An example of this panel is shown below:

Failed notifications rate

It shows a time series with the failed notifications rate aggregated by one minute.

The x-axis shows the time series and the y-axis shows the failed notifications rate.

The queries used to get panel information are:

sum(rate(prometheus_notifications_errors_total[1m])) by(integration)

An example of this panel is shown below:

CPU usage rate

It shows a time series with the CPU usage rate aggregated by one minute. It also shows the current minimum, maximum and average cpu consumption of alertmanager.

The x-axis shows the time series and the y-axis shows the CPU usage rate.

The queries used to get panel information are:

sum(rate(container_cpu_usage_seconds_total{container="alertmanager"}[1m])) by (pod_name)

An example of this panel is shown below:

Memory usage

It shows a time series with the memory usage. It also shows the current minimum, maximum and average memory consumption of alertmanager.

The x-axis shows the time series and the y-axis shows the memory usage.

The queries used to get panel information are:

sum (container_memory_working_set_bytes{container="alertmanager"}) by (pod_name)

An example of this panel is shown below:

Pods network I/O

It shows a time series with the network I/O average aggregated by one minute. It also shows the current minimum, maximum and average network I/O.

The x-axis shows the time series and the y-axis shows the network usage.

The queries used to get panel information are:

sum (rate (container_network_receive_bytes_total{pod!="",pod=~"alertmanager.*"}[1m])) by (pod)
- sum (rate (container_network_transmit_bytes_total{pod!="",pod=~"alertmanager.*"}[1m])) by (pod)

An example of this panel is shown below:

6.1.2 - Elasticsearch dashboard

Elasticsearch dashboard

Introduction

Elastic dashboard monitors multiple data, service and system related metrics.

The different graphs are shown in the following sections:

• Cluster graphs
• Shard graphs
• system graphs
• Documents graphs
• Total operations stats graphs
• Elastic search times graphs
• Caches graphs
• Thread pool graphs
• JVM garbage collection graphs

Cluster graphs

The current section includes cluster related graphs.

Health status

Code coloured indicator of cluster health.

Metrics:

((sum(elasticsearch_cluster_health_status{color="green"})*2)+sum(elasticsearch_cluster_health_status{color="yellow"}))/count(elasticsearch_index_stats_up)

Nodes

Number of nodes.

Metrics:

count(elasticsearch_index_stats_up)

Data nodes

Number of data nodes per node.

Metrics:

sum(elasticsearch_cluster_health_number_of_data_nodes{cluster="elasticsearch"})/count(elasticsearch_index_stats_up)

Pending tasks

Pending tasks per node.

Metrics:

sum(elasticsearch_cluster_health_number_of_pending_tasks{cluster="elasticsearch"})/count(elasticsearch_index_stats_up)

Graph visual

Shards graphs

Shards related graphs.

Active primary shards

Number of active primary shards per node.

Metrics:

sum(elasticsearch_cluster_health_active_primary_shards{cluster="elasticsearch"})/count(elasticsearch_index_stats_up)

Active shards

Number of active shards per node.

Metrics:

sum(elasticsearch_cluster_health_active_shards{cluster="elasticsearch"})/count(elasticsearch_index_stats_up)

Initializing shards

Number of shards initializing per node.

Metrics:

sum(elasticsearch_cluster_health_initializing_shards{cluster="elasticsearch"})/count(elasticsearch_index_stats_up)

Relocating shards

Number of relocating shards per node.

Metrics:

sum(elasticsearch_cluster_health_relocating_shards{cluster="elasticsearch"})/count(elasticsearch_index_stats_up)

Unassigned shards

Number of unassigned shards per node.

Metrics:

sum(elasticsearch_cluster_health_delayed_unassigned_shards{cluster="elasticsearch"})/count(elasticsearch_index_stats_up)

Graph visual

System graphs

System related graphs.

CPU usage

Percentage of used CPU on master and data nodes.

Metrics: It includes two metrics:

• Master node CPU usage

elasticsearch_process_cpu_percent{cluster="elasticsearch",es_master_node="true",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

• Data nodes CPU usage:

elasticsearch_process_cpu_percent{cluster="elasticsearch",es_data_node="true",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

JVM memory usage

Memory used by JVM graph in bytes.

Metrics:

It includes three metrics:

• Used memory

elasticsearch_jvm_memory_used_bytes{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

• Committed memory

elasticsearch_jvm_memory_committed_bytes{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

• Max memory

elasticsearch_jvm_memory_max_bytes{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

Disk usage

Disk usage in bytes.

Metrics:

1-(elasticsearch_filesystem_data_available_bytes{cluster="elasticsearch"}/elasticsearch_filesystem_data_size_bytes{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"})

Network usage

Bytes rate sent and received, aggregated by one minute.

Metrics: It includes two metrics:

• Sent bytes

irate(elasticsearch_transport_tx_size_bytes_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Received bytes

irate(elasticsearch_transport_rx_size_bytes_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Graph visual

Documents graphs

Documents state related graphs.

Documents count

Number of documents in cluster.

Metrics:

elasticsearch_indices_docs{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

Documents indexed rate

Rate of indexed documents, aggregated by one minute.

Metrics:

irate(elasticsearch_indices_indexing_index_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Documents deleted rate

Rate of deleted documents, aggregated by one minute.

Metrics:

rate(elasticsearch_indices_docs_deleted{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Documents merged rate

Rate of merged documents, aggregated by one minute.

Metrics:

rate(elasticsearch_indices_merges_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Graph visual

Total operations stats graphs

Data related to total operations.

Total operations rate

Total operations number rate, aggregated by one minute.

Metrics: It includes six metrics:

• Indexing index

irate(elasticsearch_indices_indexing_index_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Search queries

irate(elasticsearch_indices_search_query_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Search fetch

irate(elasticsearch_indices_search_fetch_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Merges

irate(elasticsearch_indices_merges_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Refresh

irate(elasticsearch_indices_refresh_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Flush

irate(elasticsearch_indices_flush_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Total operations time

Time rate for the different operations in milliseconds, aggregated by one minute.

Metrics: It includes six metrics:

• Indexing index

irate(elasticsearch_indices_indexing_index_time_seconds_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Search queries

irate(elasticsearch_indices_search_query_time_ms_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Search fetch

irate(elasticsearch_indices_search_fetch_time_ms_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Merges

irate(elasticsearch_indices_merges_total_time_ms_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Refresh

irate(elasticsearch_indices_refresh_total_time_ms_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

• Flush

irate(elasticsearch_indices_flush_time_ms_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Graph visual

Elasticsearch times graphs

Graphs related to elapsed times of different actions.

Query time

Time rate for search query operations in seconds, aggregated by one minute.

Metrics:

rate(elasticsearch_indices_search_query_time_seconds{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m]) 

Indexing time

Time rate for indexing index operations in seconds, aggregated by one minute.

Metrics:

rate(elasticsearch_indices_indexing_index_time_seconds_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Merging time

Time rate for merge operations in seconds, aggregated by one minute.

Metrics:

rate(elasticsearch_indices_merges_total_time_seconds_total{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Caches graphs

Graphs related to caches metrics.

Field data memory size

Field data memory size in bytes.

Metrics:

elasticsearch_indices_fielddata_memory_size_bytes{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

Field data evictions

Rate of field data evicted, aggregated by one minute.

Metrics:

rate(elasticsearch_indices_fielddata_evictions{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Query cache size

Bytes of memory occupied by cached queries.

Metrics:

elasticsearch_indices_query_cache_memory_size_bytes{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

Query cache evictions

Rate of queries evicted, aggregated by one minute.

Metrics:

rate(elasticsearch_indices_query_cache_evictions{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Graph visual

Thread pool graphs

Graphs related to the thread pool.

Operations rejected

Rate of rejected operations, aggregated by one minute.

Metrics:

irate(elasticsearch_thread_pool_rejected_count{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Operations queued

Rate of queued operations, aggregated by one minute.

Metrics:

elasticsearch_thread_pool_active_count{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

Threads active

Number of active threads.

Metrics:

elasticsearch_thread_pool_active_count{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}

Operations completed

Shows rate of completed operations, aggregated by one minute

Metrics:

irate(elasticsearch_thread_pool_completed_count{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Graph visual

JVM Garbage collection graphs

Graphs related to JVM garbage collector activity.

GC count

Rate of GC count, aggregated by one minute.

Metrics:

rate(elasticsearch_jvm_gc_collection_seconds_count{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

GC time

Rate of GC execution time, aggregated by one minute.

Metrics:

rate(elasticsearch_jvm_gc_collection_seconds_count{cluster="elasticsearch",name=~"(elasticsearch-es-aura-0|elasticsearch-es-aura-1|elasticsearch-es-aura-2)"}[1m])

Graph visual

6.1.3 - Fluent bit dashboard

Fluent bit dashboard

Introduction

Fluent bit dashboard monitors system metrics related to fluent bit.

The available metrics are defined in the following sections.

Input bytes

Input bytes rate, aggregated by one minute.

Metrics:

rate(fluentbit_input_bytes_total[1m])

Graph visual

Output bytes

Output bytes rate, aggregated by one minute.

Metrics:

rate(fluentbit_output_proc_bytes_total[1m])

Graph visual

Retries/fails

Rate of retries and fails, aggregated by one minute

Metrics:
It includes two metrics:

• Retries rate

rate(fluentbit_output_retries_total[1m])

• Fails rate

rate(fluentbit_output_retries_failed_total[1m])

Graph visual

Errors

Rate of output errors, aggregated by one minute.

Metrics:

rate(fluentbit_output_errors_total[1m])

Graph visual

6.1.4 - Kubernetes cluster monitoring dashboard

Kubernetes cluster monitoring dashboard

Introduction

Kubernetes cluster monitoring dashboard monitors multiple systems and networks related data from Kubernetes clusters.

The available metrics are defined in the following sections.

Network I/O pressure graph

Rate of total received/sent data on all cluster containers, in bytes and aggregated by one minute.

Metrics:
It includes two metrics:

• Received bytes

sum (rate (container_network_receive_bytes_total{kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m]))

• Sent bytes (negative value)

- sum (rate (container_network_transmit_bytes_total{kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m]))

Graph visual

Total usage

Graphs with different system parameters usage.

Cluster memory usage

It is composed by three graphs:

• Memory usage, showing percentage of used memory
• Used, showing used memory
• Total, showing total memory

Metrics:
It includes three metrics:

• Memory usage percentage

sum (container_memory_working_set_bytes{id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) / 
sum (machine_memory_bytes{kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) * 100

• Used memory

sum (container_memory_working_set_bytes{id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"})

• Total cluster memory

sum (machine_memory_bytes{kubernetes_io_hostname=~"^.*$",agentpool=~".*"})

Cluster CPU usage

It is composed by three graphs:

• CPU usage, showing percentage of used CPU cores, aggregated by one minute
• Used, showing used CPU cores, aggregated by one minute
• Total, showing total CPU cores

Metrics:
It includes three metrics:

• CPU usage percentage

sum (rate (container_cpu_usage_seconds_total{id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m])) / 
sum (machine_cpu_cores{kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) * 100

• Used CPUs

sum (rate (container_cpu_usage_seconds_total{id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m]))

• Total cluster CPUs

sum (machine_cpu_cores{kubernetes_io_hostname=~"^.*$",agentpool=~".*"})

Cluster filesystem usage

It is composed by three graphs:

• Filesystem usage, showing percentage of used filesystem space
• Used, showing used filesystem space
• Total, showing total filesystem space

Metrics:
It includes three metrics:

• Filesystem usage

sum (container_fs_usage_bytes{device=~"^/dev/.*$",id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) / 
sum (container_fs_limit_bytes{device=~"^/dev/.*$",id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) * 100

• Used

sum (container_fs_usage_bytes{device=~"^/dev/.*$",id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"})

• Total

sum (container_fs_limit_bytes{device=~"^/dev/.*$",id="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"})

Graph visual

Pods CPU usage

CPU usage rate, classified by pod and aggregated by one minute.

Metrics:

sum (rate (container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m])) by (pod_name)

Graph visual

Containers CPU usage

CPU usage rate, classified by container and aggregated by one minute.

Metrics:
It includes two metrics:

• Containers with “k8s_”

sum (rate (container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m])) by (pod_name)

• Containers without “k8s_”

sum (rate (container_cpu_usage_seconds_total{image!="",name!~"^k8s_.*",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m])) by (kubernetes_io_hostname, name, image)

Graph visual

All processes CPU usage

Total CPU usage rate, aggregated by one minute.

Metrics:

sum (rate (container_cpu_usage_seconds_total{id!="/",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m])) by (id)

Graph visual

Pods memory usage

Memory usage, classified by pod.

Metrics:

sum (container_memory_working_set_bytes{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) by (pod_name)

Graph visual

Containers memory usage

Memory usage, classified by container.

Metrics:
It includes two metrics:

• Containers with “k8s_”

sum (container_memory_working_set_bytes{image!="",name=~"^k8s_.*",container_name!="POD",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) by (container_name, pod_name)

• Containers without “k8s_”

sum (container_memory_working_set_bytes{image!="",name!~"^k8s_.*",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) by (kubernetes_io_hostname, name, image)

Graph visual

All processes memory usage

Total memory usage rate.

Metrics:

sum (container_memory_working_set_bytes{pod_name!="",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}) by (pod_name)

Graph visual

Pods network I/O

Total network received/sent usage rate, classified by pod and aggregated by one minute.

Metrics:

• Received bytes

sum (rate (container_network_receive_bytes_total{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^.*$",agentpool=~".*"}[1m])) by (pod_name)