Aura –

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura entities definition

Aura entities definition versions:

Version 5.0.0 for those entities using CSV format
Version 6.0.0 for those entities using Avro format

Introduction

Aura entities definition includes the different entities currently written in Aura and required to calculate Aura KPIs and generate Aura’s invoices.

Entities are organized by types, as shown in the table below, together with the definition, how they are generated and its format, currently, CSV or Avro.

Entities types	Description	Generation	Format
Dimensions	They are Aura’s configuration settings that are needed to understand Aura’s entities	They are generated directly by aura-kpis-uploader from the content of the aura-configuration-api.	Avro
Message	Message entity is used to store information about the messages handled by aura-bot and the actions performed on them.	They are created in aura-bot.	CSV
Recognizer	Recognizer entity is used to store the request to any of the recognizers during the utterance recognition phase of the messages.	They are created by aura-groot, aura-bot and aura-nlp.	CSV
Extended Message	Extended Message entity is used to store extra information of a Message.	They are created in aura-bot.	CSV
Groot Message	Groot Message entity is used to store information about the messages handled by aura-groot.	They are created in aura-groot.	CSV
Gateway Message	Gateway Message entity is used to store information about the messages handled by aura-gateway-api.	They are created by aura-gateway-api.	Avro
Audit	Audit entity is used to store information about the interactions handled by Aura components. They will be used to generate Aura’s invoices.	They are created by aura-kpis-uploader based on the information received in Groot Message and Gateway Message.	Avro

Entities generation

Until Aura’s release 9.3.0 (Gwen Stefani), all entities in Aura were generated in CSV format.
The full procedure is explained in Aura Entities in CSV format.

In release 9.4.0 (Hannah Montana), the process to migrate Aura entities format from CSV to full-URM-compliant Avro format started. It is going to be a phased process, with entities being migrated in small groups to minimize the impact on services already consuming Aura entities in CSV.
The full procedure is explained in Aura Entities in Avro format.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura KPIs uploader

Aura KPIs uploader is the component in charge of the management of Aura entities and KPIs dimensions

Introduction

aura-kpis-uploader is the component responsible for handling entities and KPIs dimensions in Aura.

Currently, it manages both CSV entities and AVRO ones, that coexist in Aura.

aura-kpis-uploader-cli is an executable script that uploads KPIs in the storage location indicated in the destination config. It is a cron-job deployed in Aura’s cluster that is executed every hour (although it is configurable in the range from 1h to 24h). It is developed with Node 14.

Detailed information regarding aura-kpis-uploader-cli is found in the following documents:
. Architecture and main components
. How does Aura KPIs uploader work?
. How to use Aura KPIs uploader?
. Environment variables
. KPIs dimensions and Aura entities processing
. Troubleshooting

Aura KPIs uploader architecture

In the following diagram the architecture of aura-kpis-uploader-cli is represented, including its main components, which are described in the following sections.

Aura KPIs uploader components

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a channel configuration or environment variables.

Main process

Main process is referred to the process that a cron-job executes in Aura’s cluster.

KPI Upload

It contains the logic to perform the upload of KPIs dimensions and Aura entities in parallel mode to Kernel Azure Storage Account.

This process is in charge of converting Aura entity files to Avro format, so that they can be imported in Kernel DataLake and be consumed by third-party teams, such as Network Tokenization or BI of each OB. Aura DataSet Importer can only handle blobs of type Block Blob.

KPIs Processes

The processes are defined in a configuration file configured in AURA_SOURCE_PATH_AVRO_ADAPTERS and they are classified into:

KPIs Dimensions AVRO
Currently, KPIs dimensions are generated in Avro format
KPIs Entities
- KPIs Entities CSV
- KPIs Entities AVRO

Storage File Manager

This module is used to download and upload files from and to Azure Storage.

Aura KPIs uploader operation

The execution flowchart of aura-kpis-uploader-cli is shown in the following image:

aura-kpis-uploader-cli is responsible for copying the CSV files in Aura KPIs container (environment variables: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT) to the correspondent container in Kernel Azure Storage (environment variable: AURA_KPI_UPLOADER_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION).

Once the files are copied, the local copy is moved to a folder inside the container (environment variables: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT/AURA_KPI_UPLOADER_PROCESSED_FOLDER) and kept there during a fixed time, for recovering purposes.

The configured time by default is 7 days and is set in installer aurak8s in the template lifecycle.json.j2:

    {
      "name": "retention-kpis-processed-policy",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": [ "blockBlob" ],
          "prefixMatch": [ "{{ kpi_blob_container_name_processed }}" ]
        },
        "actions": {
          "baseBlob": {
            "delete": { "daysAfterModificationGreaterThan": {{ backup_retention_time | default(7) }} }
          }
        }
      }
    }

Independently of when it runs, aura-kpis-uploader-cli always performs the same process:

It gets all the files in KPIs container (environment variable: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT).
Firstly, it verifies the number of files to process in order not to overload the memory. This is given by the environment variable AURA_KPI_NUM_MAX_FILES_FOR_EXECUTION. If the number of files exceeds this number, it will adapt the dates until it gets a range that fulfills the condition. The initial date is determined by AURA_DAYS_INTERVAL and the final date is calculated by subtracting the hours indicated in AURA_KPI_HOURS_TO_SUBTRACT_TO_GET_NUM_MAX_FILES.
When the number of files condition is fulfilled, all the aura-kpis-uploader processes will use that date range, except for the generation of the AUDIT file, since this file is generated at run time and must be moved to the entities folder if everything has proceeded successfully, with the current date.

It can also be launched manually to process old files that have not been processed in the regular execution, due to lack of communication with Kernel or any other error.

Furthermore, aura-kpis-uploader-cli generates for dimensions of entity type files: Channels, Skills, Presets, Applications, Components and Recognizers. You can see more information in KPIs dimensions.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Databricks Jobs

aura-databricks-jobs is a component based on Databricks. Discover in the current section its technical description and main components.

Introduction

aura-databricks-jobs is a component based on Databricks for the optimization of data processing and the training of ML-based models.

Currently, its primary function is to import Avro-formatted files into Kernel datasets. For this purpose, we will see later that it is necessary to configure a run job in the Databricks environment. Find the method of the job in avro_to_dataset_job.py.

aura-to-dataset-job-cli is an executable script that imports Avro KPIs into the storage location indicated in the Kernel dataset destination config. It is configured in a Databricks cluster that is executed every day (although it is configurable in the job schedule). It is developed with Python and uses the Kernel Spark SDK to read the Avro files and write in Kernel datasets.

Detailed information regarding aura-databricks-jobs is found in the following documents:
. Architecture and main components
. How does aura-databricks-jobs work?
. aura-databricks-jobs configuration
. How to use aura-databricks-jobs?
. Environment variables
. Troubleshooting

Aura Databricks Jobs architecture

In the following diagram, the architecture of aura-to-dataset-job-cli is represented, including its main components, which are described in the following sections.

Avro to Dataset Job components

ConfigManager

ConfigManager is a handler for configuration that is gathered from input config_dict to fulfill the variables needed in the import process. It also validates the configuration. In any error case, the process is not executed.

AuraLogging

AuraLogging is a wrapper of LoggerWrapper class imported from aura-pytraces library. It used to register logs adding the required items such as version, app, stck etc.

The behavior of logs in the file logging.cfg is internally configurable, following the format established by the aura-pytraces library. This configuration may be overwritten:

level of handler config by environment variable AURA_LOGGING_LEVEL. By default, INFO value.
formatter of handler config by environment variable AURA_LOGGING_FORMAT. By default, simple value.
version by environment variable AURA_VERSION. By default, not-reachable value.

Avro to Dataset Job

It is referred to the process that a cron-job executes in the Databricks.

It contains the logic to configure coroutines to import Avro files by type of dataset with asyncio library.

The result of each coroutine is a report. When all the coroutines are finished, the reports are processed, generating a single one with the information of all the import process and including Spark processing info.

Avro KPI importer

It contains the logic to import Avro-formatted files by type of dataset. If there are not Avro-formatted files of this type of dataset, this coroutine finishes.

The result of each routine is the report of the importation process of the specific type of dataset.

Azure Storage Manager

This module is used to download and upload files from and to Azure Storage.

Spark SDK Manager

This module is used to load data as a Dataframe from Azure Storage and write in dataset of Kernel Datalake.

Aura Databricks Job operation

The execution flowchart of avro-to-dataset-job-cli is shown in the following image:

avro-to-dataset-job-cli

It is responsible for importing the Avro-formatted files in Aura KPIs container (job’s variable: AURA_MICROSOFT_AZURE_STORAGE_KPIS_CONTAINER_NAME) to the correspondent dataset in Kernel.

The information necessary to import the Avro-formatted files with the same Avro schema to their corresponding dataset is obtained from the configuration file stored in the Azure KPIs container, specifically the file path configured in the job’s variable: AURA_KPI_AVRO_ADAPTER_CONFIG_PATH.

In addition, there is a file that will provide us with the average size of the files by type of dataset, specifically the file path configured in the job’s variable: AURA_KPI_AVRO_SOURCE_SIZE_REPORT_PATH. This information will be useful when writing in Kernel datasets with the Spark tool to correctly indicate how the data should be partitioned to improve performance.

From this file, we will obtain all the schemas that are imported. For this purpose, it is required that targetType is set with avro value in each item.

Below, it is defined the information that the job gathers for each Avro schema:

name: dataset_id used to import into Kernel. For example, D_Aura_Channel.
schema: type of schema. For example, dimensional or entity.
versionSchema: Version of avroSchema. For example, 6.0.0. The major version will be used in the Spark stage to write in Kernel dataset.
avroSchema: name of the schema stored in the container within the folder configured in the AURA_KPI_AVRO_SCHEMAS_PATH variable. The Avro schema necessary when reading the files in spark is obtained from the path configured in the job variable: AURA_KPI_AVRO_SCHEMAS_PATH and extra parameters: $AURA_KPI_AVRO_SCHEMAS_PATH/$schema/$versionSchema/$avroSchema. Example: schemas/dimensional/6.0.0/aura-channel-asvc.json.

Sample of Aura Avro adapter file:

[
    {
        "name": "D_Aura_Channel",
        "schema": "dimensional",
        "avroSchema": "aura-channel-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "CHANNEL"
        },
        "targetType": "avro",
        "fields": {
            "AURA_CHANNEL_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_CHANNEL_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "AURA_CHANNEL_SHORT_NAME": {
                "sourceName": "prefix",
                "targetType": "string"
            }
        }
    },
    {
        "name": "D_Aura_Recognizer",
        "schema": "dimensional",
        "avroSchema": "aura-recognizer-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "RECOGNIZER"
        },
        "targetType": "avro",
        "fields": {
            "AURA_RECOGNIZER_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_RECOGNIZER_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }
]

The job will run the import process for each schema type, running in coroutines and using the asyncio library.

The following process is carried out for each type of schema:

Check if there are schemas configured not to be loaded. The job variable where this configuration is configured is: AURA_KPI_AVRO_SCHEMAS_NOT_TO_UPLOAD. The format is a list formatted as schema_1:dataset_id_1,schema_1:dataset_id_2,schema_2;dataset_id_3. Example: dimensional:D_Aura_Channel,entity:E_Aura_GROOT. The number of files that have been skipped for that type are recorded in a report.
Check if there are files of that type to import in its corresponding folder. The path where the Avro-formatted files are stored is: AURA_KPI_AVRO_SOURCE_PATH. Within this path, the files are stored by their corresponding $schema/$dataset/$version. Example dimensional/6.0.0/D_Aura_Channel. If there are no files, the coroutine ends up generating a report without uploaded files.
If there are files, the reading will be carried out with Spark, indicating the Azure Blob where the files with the same Avro schema are located. Additionally, they will be written to its corresponding dataset of Kernel Datalake. This step is configured with locking using asyncio to prevent asyncio.Lock() from protecting read and write operations on a DataFrame.
Once the files are imported, the local copy is moved to a folder inside the container (job’s variables: AURA_MICROSOFT_AZURE_STORAGE_KPIS_CONTAINER_NAME/AURA_KPI_AVRO_PROCESSED_FOLDER_PATH) and kept there during a fixed time, for recovering purposes.
All the details of the process are recorded in a report that is stored in the job variable: AURA_KPI_AVRO_REPORTS_DESTINATION_PATH/aura-avro-kpis-report-{iso-date}.json.
Depending on the configured report mode, AURA_KPI_AVRO_REPORTS_MODE will be generated only when errors occur, always or never.

Independently of when it runs, avro-to-dataset-job-cli always performs the same process: it gets all the Avro-formatted files in KPIs container (job variable: AURA_MICROSOFT_AZURE_STORAGE_KPIS_CONTAINER_NAME) from the last upload executed by the aura-kpis-uploader component.

When running independently on the Databricks cluster, Prometheus alerts cannot be configured. Therefore, the process information will be obtained from the report generated along with the following generated files:

If the process has ended with errors:
- A file with the name set in the variable AURA_KPI_AVRO_PROCESS_ERROR_FILENAME will be generated containing the execution date.
- Additionally, if the report has been generated in Azure Storage, the link to it will be included, valid for the time configured in the variable AURA_KPI_AVRO_REPORTS_SAS_EXPIRATION.
- If the report cannot be recorded, the error will appear in the file.
If the process terminates abruptly due to a timeout and the databricks manager kills the process:
- A report will be generated, showing each dataset in its corresponding stage.
- The stages of each dataset can be completed, as when it is run again, it will obtain the last report generated. From this one, it will identify the stage to continue.
- If the process remained in the stage WRITING_DATASET_OK, the files from the last execution will be moved to the processed folder and deleted from the avro folder.
- If the process remained in the stage READING_BLOBS or WRITING_DATASET, the files will be loaded together with the rest of the files that have been generated without making distinctions.
- If the process remained in the stage MOVING_BLOBS_TO_PROCESSED, the files will be moved to the processed folder. If this second attempt fails again, the stage will be set to the value NOT_PROCESSED_PREVIOUS_ERRORS to indicate that it is not recoverable and that a manual review must be carried out in case there is a corrupt Avro file.
- If the process remained in the stage REMOVING_BLOBS, the files will be deleting from the avro folder. If this second deletion attempt fails again, the stage will be set to the value NOT_PROCESSED_PREVIOUS_ERRORS to indicate that it is not recoverable and that a manual review must be carried out in case there is a corrupt Avro file.
- If the process remained in the stage WRITING_DATASET_ERROR_NOT_RECOVERABLE, the files of the last execution and the possible ones that have been added since the last run will not be loaded, since there are unrecoverable errors that must be verified manually to be resolved. This involves writing datasets with malformed records or discarded records. So, for the dataset, the stage is recorded as NOT_PROCESSED_PREVIOUS_ERRORS to avoid loading this dataset.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

KPI entity handler

Description of the KPI entity handler, a module in charge of calling aura-kpi-handler utility.

It is used by aura-authentication-api, aura-bot and aura-groot.

Introduction

One of the elements ready to be used across aura-bot is the kpi-handler, that is the aura-bot module in charge of calling aura-kpi-handler utility.

This kpi-handler is a singleton module that, during its initialization, starts a KpiHandler used to write the rows in the corresponding blob.

Methods

The kpi-handler provides a method to write each entity row that is used at the different stages of the activity processing: aura-bridge-outbound .

incomingMessage: generates a MessageEntity with the information available when the message enters aura-bot, if the user is properly authenticated.
extendedIncomingMessage: generates a ExtendedMessageEntity with the information available when the message enters aura-bot, if the user is properly authenticated.
unauthenticatedIncomingMessage: generates a ExtendedMessageEntity with the information available when the message enters aura-bot, if there is any error during the user’s authentication.
outgoingMessage: it is called once per activity returned by aura-bot as response of an incoming activity. It fills all the data generated during the activity processing.
extendedOutgoingMessage: it is called once per activity returned by aura-bot as response of an incoming activity. It fills all the data generated during the activity processing.
unauthenticatedOutgoingMessage: generates a MessageEntity with the information available when *aura-bot sends the response, if there is any error during the user’s authentication.
recognize: generates a RecognizerEntity row with the information of the execution of every recognizer called during the activity processing, this kpi is written by both the aura-bot and the aura-groot.
incomingGrootMessage: generates a GrootMessageEntity with the information available when the message enters aura-groot.
outgoingGrootMessage: it is called once per activity returned by aura-groot as response of an incoming activity. It fills all the data generated during the activity processing.

All the above-mentioned methods work following the same process:

The information is gathered from their incoming parameters, from the TurnContext and from ConversationState and UserStage .
Data is converted into the values and formats needed by aura-kpi-handler.
The corresponding event handled by aura-kpi-handler is emitted, that is in charge of writing the row in the corresponding blob.
aura-kpi-handler receives the event, processes the data in the corresponding entity and pushes it to the entity buffer stream. This stream is appended to the corresponding BlobFile every 10 seconds (configurable).
If the POD is stopped, all the content in the buffer stream is appended automatically to the log.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Status codes stored in KPIs entities

Description of the aura-bot status codes which are stored in the Aura entities

KPI response codes

List of response codes stored in KPIs by aura-bot classified by KPI type:

Type	StatusCodes	Reason
MESSAGE / GROOT MESSAGE	[ 200, any (only in events) ]	[ 200 ] : https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L88 https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L120 https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L167 https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L200 [any] : https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L625
EXTENDED_MESSAGE	[ 200, any (only in events) ]	[ 200 ] : https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L471 https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L120 https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L563 https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L200 [any] : https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L625
RECOGNIZER	[ 200, 400, 404, 424, 500, 502, 504 ]	[ 200, 400, 404, 500, 502, 504 ] : https://github.com/Telefonica/aura-clients/blob/master/packages/aura-nlp-client/swagger/aura-nlp-client.json https://github.com/Telefonica/aura-bot-platform/blob/master/src/utils/kpi-handler.ts#L411 [ 200, 424 ]: https://github.com/Telefonica/aura-bot-platform/blob/master/src/middlewares/recognizers/base-recognizer.ts#L55