Aura entities

• 1: Aura entities definition
• 1.1: Aura entities definition in Avro
• 1.1.1: Aura_Atria_Message
• 1.1.2: Aura_Audit
• 1.1.3: Aura_Gateway_Message
• 1.1.4: Aura_Message
• 1.1.5: D_Aura_Dimensional
• 1.2: Aura entities definition in CSV
• 1.2.1: Aura_Gateway_Message
• 1.2.2: Message
• 1.2.3: Recognizer
• 1.2.4: Extended Message
• 1.2.5: Groot message
• 2: Aura KPIs uploader
• 2.1: User guide
• 2.2: Adapter Manager
• 2.3: Environment variables
• 2.4: Aura KPIs dimensions
• 2.5: Aura entities processing
• 2.6: Troubleshooting
• 3: Aura Databricks Jobs
• 3.1: Configuration
• 3.2: Environment variables
• 3.3: User guide
• 3.4: Troubleshooting
• 4: KPI entity handler
• 5: Status codes

1 - Aura entities definition

Aura entities definition

Introduction

Aura entities definition includes the different entities currently written in Aura and required to calculate Aura KPIs and generate Aura’s invoices.

Entities are organized by types, as shown in the table below, together with the definition, how they are generated and its format, currently, CSV or Avro.

Entities generation

Until Aura’s release 9.3.0 (Gwen Stefani), all entities in Aura were generated in CSV format.
The full procedure is explained in Aura Entities in CSV format.

In release 9.4.0 (Hannah Montana), the process to migrate Aura entities format from CSV to full-URM-compliant Avro format started. It is going to be a phased process, with entities being migrated in small groups to minimize the impact on services already consuming Aura entities in CSV.
The full procedure is explained in Aura Entities in Avro format.

1.1 - Aura entities definition in Avro

Aura entities definition in Avro format

Introduction

The Aura entities definition includes the different entities currently written in Aura in Avro format, URM-compliant.

For further information regarding Kernel Avro datasets access the following links:

• How to handle datasets in Kernel
• Kernel datasets

Entities generated in Avro URM-compliant format

Entities generation

Avro Entities tables nomenclature

• Position: Field order in the Avro schema definition.
• Field: Name of the field being defined. Usually, capitalized, although more aliases will be available in the schema definition.
• PK: Parameter that indicates if the field is a Primary Key or not, that is, a value that uniquely identifies this field.
• Referenced Entity: When the field references a value from another dataset, indicating that a join must be defined in the schema, this parameter contains the name of the dataset.
• Referenced Entity Field: When the field references a value from another dataset, indicating that a join must be defined in the schema, this parameter contains the name of the field of the other dataset.
• Type: Type of the field. It can be one of Avro logical types defined in Kernel.
• Doc: Brief description of the field.
• Nullable doc: If the field can be null, it should contain the cases when a null value is valid.

1.1.1 - Aura_Atria_Message

Aura_Atria_Message v6.0.0

This entity contains the summary of Aura’s interactions handled by aura-rag-server, aura-groot and aura-bot

Avro schema definition published in Kernel - TO BE COMPLETED

1.1.2 - Aura_Audit

Aura_Audit v6.0.0

This entity contains the summary of Aura’s interactions intended for accountability purposes.

Avro schema definition published in Kernel

1.1.3 - Aura_Gateway_Message

Aura_Gateway_Message v6.0.0

This entity contains the summary of Aura’s interactions handled by aura-gateway-api.

Avro schema definition published in Kernel

1.1.4 - Aura_Message

Aura_Message v6.0.0

This entity contains the summary of Aura’s interactions handled by aura-bridge, aura-groot and aura-bot

Avro schema definition published in Kernel - TO BE COMPLETED

1.1.5 - D_Aura_Dimensional

Aura Dimensional entities

D_Aura_App schema definition v6.0.0

This entity contains the list of possible Apps defined in Aura.

Avro schema definition published in Kernel

D_Aura_Channel v6.0.0

This entity contains the list of possible channels defined in Aura

Avro schema definition published in Kernel

D_Aura_Component v6.0.0

This entity contains the list of possible components defined by Aura

Avro schema definition published in Kernel

D_Aura_Preset v6.0.0

This entity contains the list of possible presets defined by Aura

Avro schema definition published in Kernel

D_Aura_Recognizer v6.0.0

This entity contains the list of possible recognizers defined by Aura

Avro schema definition published in Kernel

D_Aura_Skill v6.0.0

This entity contains the list of possible skills defined by Aura

Avro schema definition published in Kernel

1.2 - Aura entities definition in CSV

Aura entities definition in CSV format

Introduction

The Aura entities definition includes the different entities currently written in Aura in CSV and required to calculate Aura KPIs or by any other team in their data processes.

Entities generation

Until release 9.3.0 (Gwen Stefani), all entities in Aura were generated in CSV format. They were also uploaded into Kernel storage in CSV format. Although using a script provided by and running in Kernel, some of them are converted to Avro entities (in a deprecated Avro format, not URM compliant).

In particular, the following conversions are provided:

• Recognizer:
  • Aura Authenticated Users’ Recognizer
  • Aura Anonymous Users’ Recognizer

As can be seen, the entities in Avro are duplicated to hold authenticated and anonymous users interactions separately, because the field USER_4P_ID cannot be null if it exists in a dataset definition.

CSV files format

All entities’ files must be created following these rules:

• File format: UNIX. UTF8 without BOM
• Date format: ISO8601
  • Date: 2018-05-02
  • Datetime: 2018-05-02T15:18:11Z => Always UTC

Usually, these kinds of files are stored in .txt and zipped, before uploading.

• Entities used for the calculation of KPIs are stored in a Kernel bucket. Each entity must be included in a different folder, with files distributed by months, in the path:
  [OB]/[ENTITY]/YYYYMM

• Files generated in aura-bot can be stored as needed.
  Filename: BOT_[HOST_ID]_[OB]_[ENTITY]_YYYYMMDDTHH0000Z.txt

• Files generated in aura-services (authentication) can be stored as needed.
  Filename: SERVICES_[HOST_ID]_[OB]_[ENTITY]_YYYYMMDDTHH0000Z.txt

• Files generated in Aura NLP components can be stored as needed:
  Filename: NLP_<HOST_ID>_<OB>_<ENTITY>_YYYYMMDDTHH0000Z.txt

• Dimensions entities:
  Path: [OB]/DIMENSIONS/YYYYMM
  Filename: [OB]_DIM_[DIM_NAME]_YYYYMMDDTHH0000Z.txt
  For example: ES_DIM_CHANNEL_20180612T160000Z.txt

All the files are refreshed every day.

CSV Entities tables nomenclature

• #: Field ID
• FIELD: Specific field of the entity type
• PK: Parameter that indicates if the field is a Primary Key or not, that is, a value that uniquely identifies this field.
• NULLABLE: Parameter that indicates if a field is allowed to have a null value or not.
• TYPE: Type of the field. It can be one of: text, date, number, boolean
• DESCRIPTION: Brief description of the field
• FORMAT: Field mandatory format, if applicable
• ALLOWED VALUES: Prefixed values permitted for this field
• EXAMPLE: Example of application

The following considerations must be taken into account:

• Numeric values are rounded to two decimal positions
• The amount of money must be included in local currency

1.2.1 - Aura_Gateway_Message

Aura_Gateway_Message v6.0.0

This entity contains the summary of Aura’s interactions handled by aura-gateway-api.

1.2.2 - Message

Message

1.2.3 - Recognizer

Recognizer

1.2.4 - Extended Message

Extended Message

1.2.5 - Groot message

Groot message

2 - Aura KPIs uploader

Aura KPIs uploader

Introduction

aura-kpis-uploader is the component responsible for handling entities and KPIs dimensions in Aura.

Currently, it manages both CSV entities and AVRO ones, that coexist in Aura.

aura-kpis-uploader-cli is an executable script that uploads KPIs in the storage location indicated in the destination config. It is a cron-job deployed in Aura’s cluster that is executed every hour (although it is configurable in the range from 1h to 24h). It is developed with Node 14.

Detailed information regarding aura-kpis-uploader-cli is found in the following documents:
. Architecture and main components
. How does Aura KPIs uploader work?
. How to use Aura KPIs uploader?
. Environment variables
. KPIs dimensions and Aura entities processing
. Troubleshooting

Aura KPIs uploader architecture

In the following diagram the architecture of aura-kpis-uploader-cli is represented, including its main components, which are described in the following sections.

Aura KPIs uploader components

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a channel configuration or environment variables.

Main process

Main process is referred to the process that a cron-job executes in Aura’s cluster.

KPI Upload

It contains the logic to perform the upload of KPIs dimensions and Aura entities in parallel mode to Kernel Azure Storage Account.

This process is in charge of converting Aura entity files to Avro format, so that they can be imported in Kernel DataLake and be consumed by third-party teams, such as Network Tokenization or BI of each OB. Aura DataSet Importer can only handle blobs of type Block Blob.

KPIs Processes

The processes are defined in a configuration file configured in AURA_SOURCE_PATH_AVRO_ADAPTERS and they are classified into:

• KPIs Dimensions AVRO
  Currently, KPIs dimensions are generated in Avro format

• KPIs Entities

  • KPIs Entities CSV
  • KPIs Entities AVRO

Storage File Manager

This module is used to download and upload files from and to Azure Storage.

Aura KPIs uploader operation

The execution flowchart of aura-kpis-uploader-cli is shown in the following image:

aura-kpis-uploader-cli is responsible for copying the CSV files in Aura KPIs container (environment variables: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT) to the correspondent container in Kernel Azure Storage (environment variable: AURA_KPI_UPLOADER_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION).

Once the files are copied, the local copy is moved to a folder inside the container (environment variables: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT/AURA_KPI_UPLOADER_PROCESSED_FOLDER) and kept there during a fixed time, for recovering purposes.

The configured time by default is 7 days and is set in installer aurak8s in the template lifecycle.json.j2:

    {
      "name": "retention-kpis-processed-policy",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": [ "blockBlob" ],
          "prefixMatch": [ "{{ kpi_blob_container_name_processed }}" ]
        },
        "actions": {
          "baseBlob": {
            "delete": { "daysAfterModificationGreaterThan": {{ backup_retention_time | default(7) }} }
          }
        }
      }
    }

Independently of when it runs, aura-kpis-uploader-cli always performs the same process:

• It gets all the files in KPIs container (environment variable: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT).
• Firstly, it verifies the number of files to process in order not to overload the memory. This is given by the environment variable AURA_KPI_NUM_MAX_FILES_FOR_EXECUTION. If the number of files exceeds this number, it will adapt the dates until it gets a range that fulfills the condition. The initial date is determined by AURA_DAYS_INTERVAL and the final date is calculated by subtracting the hours indicated in AURA_KPI_HOURS_TO_SUBTRACT_TO_GET_NUM_MAX_FILES.
• When the number of files condition is fulfilled, all the aura-kpis-uploader processes will use that date range, except for the generation of the AUDIT file, since this file is generated at run time and must be moved to the entities folder if everything has proceeded successfully, with the current date.

It can also be launched manually to process old files that have not been processed in the regular execution, due to lack of communication with Kernel or any other error.

Furthermore, aura-kpis-uploader-cli generates for dimensions of entity type files: Channels, Skills, Presets, Applications, Components and Recognizers. You can see more information in KPIs dimensions.

2.1 - User guide

Aura KPIs uploader user guide

1. Prepare your CONFIG_FILE

First, review and prepare your environment variables in order to create your CONFIG_FILE.

An example is shown below:

AURA_AUTHORIZATION_HEADER=${AURA_AUTHORIZATION_HEADER}
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT=${AURA_CHANNELS_CONFIGURATION_API_ENDPOINT}
AURA_DEFAULT_LOCALE=${AURA_DEFAULT_LOCALE}
AURA_ENVIRONMENT_NAME=${AURA_ENVIRONMENT_NAME}
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY}
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT}
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION}
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION}
AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION}
AURA_VERSION=${AURA_VERSION}

2. Launch Aura KPIs uploader

There are two methods for launching the aura-kpis-uploader

• Directly with the source code: to launch it in your local environment
• As a docker container: to use it with the already generated Docker image.

2.1. Launch the tool directly with the source code

$ cd aura-kpis-uploader
$ npm install
$ export CONFIG_FILE=[PATH_TO_MY_CONFIG_FILE]
$ npm run start

2.2. Launch the tool as a docker container

2.2.1. Pull image from repository

This method needs login first.

docker pull auraregistry.azurecr.io/aura/aura-kpis-uploader

2.2.2. Run Docker container

There are two options for running the Docker container:

• With .env file:

  Change $VERSION accordingly, exporting it or changing the string in the commands.

  If $VERSION is not present, it will use the latest default tag.

  docker run --rm -e CONFIG_FILE=/opt/aura-kpis-uploader/env/config_file.env \
          -v $(pwd):/opt/aura-kpis-uploader/env \
          auraregistry.azurecr.io/aura/aura-kpis-uploader:$VERSION
  

• With environmental variables in command:

  Note that host.docker.internal only works for MACOS/Windows systems:

  docker run -e AURA_AUTHORIZATION_HEADER=${AURA_AUTHORIZATION_HEADER} \
          -e AURA_CHANNELS_CONFIGURATION_API_ENDPOINT=${AURA_CHANNELS_CONFIGURATION_API_ENDPOINT} \
          -e AURA_DEFAULT_LOCALE=${AURA_DEFAULT_LOCALE} \
          -e AURA_ENVIRONMENT_NAME=${AURA_ENVIRONMENT_NAME} \
          -e AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY} \
          -e AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT} \
          -e AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION} \
          -e AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION} \
          -e AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION} \
          -e AURA_VERSION=${AURA_VERSION} \
          -v ${HOST_DIRECTORY}:/opt/aura-kpis-uploader/logs \
          auraregistry.azurecr.io/aura/aura-kpis-uploader:$VERSION
  

3. Execution Dependencies

With the addition of the new component aura-databricks-jobs, now for Avro entities, the execution of aura-kpis-uploader depends on aura-databricks-jobs being executed correctly.

This is done to avoid duplicating KPIs when they are processed by aura-databricks-jobs, and an error has occurred.

3.1 Evaluation of conditions for execution

If avro-to-dataset-job-cli has been executed correctly, it will write a file to the default avro folder containing a date. If this file exists and the date it contains does not exceed the time set for the scheduled run in Databricks, aura-kpis-uploader-cli will run its processes.

If avro-to-dataset-job-cli has generated any error in its execution, it will generate a file in the same folder containing internally the error(s) that have occurred.

Environment variables involved:

• AURA_KPIS_AVRO_DESTINATION_PATH: Default value is avro.
• AURA_DATABRICKS_ERROR_FILENAME: Default value is databricks.ERROR.
• AURA_DATABRICKS_EXECUTION_PERIOD: Default value is 24 hours.

The flow that Aura KPIs Uploader follows to validate if it is going to be executed is as follows:

4 Generate Reports

By default, aura-kpis-uploader generates reports if any error has occurred in the upload process.

These reports are available in the Azure Storage defined in AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT, path aura-kpis/reports/.

If you want to change the behavior and generate reports of all uploaded files or disable their generation, you can do it by changing the environment variable AURA_KPIS_REPORTS_MODE. If the value is set to all, it will generate a report for each of the processed files and if it is set to none, it will not generate any report. The default value is error.

4.1 Report upload Files

When an error occurs, the system will send an event to Prometheus to alert about the error.

 this.prometheusHandler.addSummary(
                        { 
                        metricId: PrometheusMetrics.auraKpisUploaderReportErrorSummary, 
                        start: this.startTimer 
                        },
                        { 
                        reportError: reportLink },
                        this.corr
                        );

The event will contain the metricId auraKpisUploaderReportErrorSummary, when an error occurs and a link to the report containing the error.

4.1.1 Report Model

A report will contain the following template in JSON format:

• numberOfFiles: Number of files processed.
• numberOfFilesSkipped: Number of files that have been skipped because they have not yet been processed.
• NumberOfFilesMovedToProcessed: Number of files that have been moved to the processed folder.
• numberOfFilesUploaded : Number of files that have been successfully uploaded.
• NumberOfFilesDeleted: Number of files that have been deleted from the main folder.
• numberOfFilesUploadValidated: Number of files that have been verified as successfully uploaded.

If errors have occurred, it will contain an errors property with the files that have failed, arranged as properties:

• key: Name of the file that has failed.
• step: It indicates in which phase it has failed. The possible ones are “UPLOADING”, “CHECK_IF_UPLOADED”, “MOVING_TO_PROCESSED” AND “REMOVING”.
• error: Error message obtained.
• corr: Associated correlator.

Example:

{
 "numberOfFiles": 43,
 "numberOfFilesSkipped": 0,
 "numberOfFilesMovedToProcessed": 41,
 "numberOfFilesUploaded": 41,
 "numberOfFilesDeleted": 41,
 "numberOfFilesUploadValidated": 41,
 "errors": {
   "BOT_da5df187-8aa9-55b7-94d6-50586bd6b0b8_CR_API_20240216T080000Z.txt": {
     "step": "UPLOADING",
     "error": "getaddrinfo ENOTFOUND aurautils-test.blob.core.windows.net",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   },
   "BOT_b8efccfd-f2bd-5ef0-a19d-b2414ab94dc8_CR_API_20240215T160000Z.txt": {
     "step": "UPLOADING",
     "error": "getaddrinfo ENOTFOUND aurautils-test.blob.core.windows.net",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   }
 }
}

4.2 Report Generating Avro KPIs

When an error occurs, the system will send an event to Prometheus to alert about the error.

 this.prometheusHandler.addSummary(
                        { 
                        metricId: PrometheusMetrics.auraKpisGeneratedReportErrorSummary, 
                        start: this.startTimer 
                        },
                        { 
                        reportError: reportLink },
                        this.corr
                        );

The event will contain the metricId auraKpisGeneratedReportErrorSummary, when an error occurs and a link to the report containing the error.

4.2.1 Report Model

A report will contain the following template in JSON format.

• numberBlobsLoaded: Number of blobs loaded.
• numberDataRowsLoaded: Number of total lines loaded from blobs.
• numberDataRowsGenerated: Number of Avro registers generated.
• numberEntitiesGenerated: Number of entities generated.
• numberDimensionalGenerated: Number of dimensional files generated.

If errors have occurred, it will contain an errors property with the files that have failed, arranged as properties:

• key: The id of the source. Example: ‘AUDIT’.
• error: Error message obtained.
• corr: Associated correlator.

Example:

{
 "numberBlobsLoaded": 13,
 "numberDataRowsLoaded": 0,
 "numberDataRowsGenerated": 141,
 "numberEntitiesGenerated": 2,
 "numberDimensionalGenerated": 7,
 "errors": {
   "GATEWAY": {
     "error": "Error parsing data. Field name does not exist in source",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   },
   "AUDIT": {
     "error": "Error parsing data. Field id cannot be empty",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   }
 }
}

2.2 - Adapter Manager

Aura KPIs Adapter Manager

Definition

Set of classes that will be in charge of processing the necessary operations to copy the KPIs in CSV format to Kernel and to process or generate the KPIs of the entities and dimensions in Avro format.

This is done from a definition file that, by default, is called aura-avro-adapter.json and is defined in the AURA_SOURCE_PATH_AVRO_ADAPTERS environment variable. This file is remotely loaded from the Storage AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT and the path AURA_KPIS_ENTITIES_CONTAINER/schemas/AURA_SOURCE_PATH_AVRO_ADAPTERS. If the file does not exist remotely, a local one is loaded by default.

Adapter Definition File Model

For each KPI file to be managed, a model is defined containing an array with all the adaptation operations to be performed. The content format of this file is JSON.

AvroAdapterSchema

AvroAdapterSource

AvroAdapterSourceDataType

AvroAdapterField

AvroAdapterFieldModel

AvroPreCalculatedType

Avrofieldbasetype

AvroAdapterTargetType

Types of adapters

The Adapter manager can perform several types of processes, that will be explained in the following sections.

Copy CSV files

This adapter will copy the CSV files generated from the Aura servers (aura-bot, aura-groot, aura-nlp, etc.) to the Kernel storage.

{
        "version": "1.0.0",
        "name": "E_Aura_BOT",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "BOT"
        },
        "targetType": "csv",
        "fields": {}
    }

Generate Dimensional KPIs in Avro

This adapter will generate an Avro file with Aura available RECOGNIZER data, as they are defined only in aura-kpis-uploader and not in aura-configuration-api.

  {
        "version": "1.0.0",
        "name": "D_Aura_Recognizer",
        "schema": "dimensional",
        "avroSchema": "aura-recognizer-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "RECOGNIZER"
        },
        "targetType": "avro",
        "fields": {
            "AURA_RECOGNIZER_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_RECOGNIZER_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Copy Avro files

This adapter copies the KPIs generated in Avro by aura-gateway-api to be processed by aura-databricks-jobs.

    {
        "version": "1.0.0",
        "name": "Aura_Gateway_Message",
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "GATEWAY"
        },
        "targetType": "avro",
        "fields": {}
    }

Generate Avro from another Avro

This is an adapter that generates an AURA_AUDIT file based on the data from the AVRO file of the aura-gateway-api KPIs.

{       "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "avro",
            "id": "AUDIT",
            "avroSchema": "aura-gateway-message-asvc.json",
            "avroSchemaVersion": "6.0.0"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "AURA_CHANNEL_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "defaultValue": "nlpaas",
                "valueIfExist" : "ai"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "AURA_VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "DAY_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

Generate Avro from data in CSV file

This is an adapter that generates an AURA_AUDIT file based on the data from the CSV file of the aura-groot Message KPIs.

{
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "order": 2,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "AUDIT",
            "csvFolder": "groot",
            "entityName":"GROOTMESSAGE"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "CHANNEL_ID",
                "targetType": "string",
                  "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_SERVICE_NAME_CD",
                "targetType": "string",
                "defaultValue": "message"

            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

Another example, deleting CSV source file data when the adapter finalize its execution.

{
        "version":"1.0.1",
        "name": "Aura_Gateway_Message_converter",
        "stopWithErrors": true,
        "order": 1,
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "GATEWAY",
            "csvFolder": "gwapi",
            "entityName": "GATEWAYMESSAGE",
            "removeAfterProcessing": true
        },
        "targetType": "avro",
        "fields": {
                "MESSAGE_ID": {
                "sourceName": "MESSAGE_ID",
                "targetType": "string",
                "defaultValue": "NO_MESSAGE_ID"
            },
            "MESSAGE_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "DAY_DT": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            },
            "AURA_NLP_IND": {
                "sourceName": "AURA_NLP_IND",
                "targetType": "boolean"
            },
            "CORR_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string",
                "defaultValue": "NO_CORR_ID"
            }
        }
    }

Use filters in Avro files

With this model, it is possible to filter records from the Avro files that should not be loaded into these files:

export interface AvroAdapterSource {
    data: AvroAdapterSourceDataType;
    id: any;
    avroSchema?: string;
    avroSchemaVersion?: string;
    csvFolder: string;
    entityName: string;
    filter?: (SourceFilter | LogicalFilter);

}
export interface SourceFilter {
    field: string;
    value: any;
    operator: FilterOperator;
}
export interface LogicalFilter {
    logic: LogicalOperator.and | LogicalOperator.or;
    filters: (SourceFilter | LogicalFilter)[];
}
export enum FilterOperator {
    equal = 'equal',
    notEqual = 'notEqual',
    greaterThan = 'greaterThan',
    lessThan = 'lessThan'
}
export enum LogicalOperator {
    and = 'and',
    or = 'or'
}

These filters could be configured in aura-avro-adapter.json in two ways:

• With logical filter, with several conditions. For example:

      {
          "version": "1.0.0",
          "name": "Aura_Audit",
          "stopWithErrors": true,
          "schema": "entity",
          "avroSchema": "aura-audit-asvc.json",
          "versionSchema": "6.0.0",
          "source": {
              "data": "partialCsv",
              "id": "AUDIT",
              "csvFolder": "groot",
              "entityName": "GROOTMESSAGE",
              "filter": {
                  "logic": "and",
                  "filters": [
                      {
                          "field": "CHANNEL_ID",
                          "value": "",
                          "operator": "notEqual"
                      },
                      {
                          "field": "ACTION_CD",
                          "value": "receive",
                          "operator": "equal"
                      }
                  ]
              }
          }
  

• Or it could be configured in aura-avro-adapter.json with simple filter. For example:

  {
          "version": "1.0.0",
          "name": "Aura_Audit",
          "stopWithErrors": true,
          "schema": "entity",
          "avroSchema": "aura-audit-asvc.json",
          "versionSchema": "6.0.0",
          "source": {
              "data": "avro",
              "id": "AUDIT",
              "avroSchema": "aura-gateway-message-asvc.json",
              "avroSchemaVersion": "6.0.0",
              "filter": {
                  "field": "AURA_APP_ID",
                  "value": null,
                  "operator": "notEqual"
              }
          },
          "targetType": "avro",
          "fields": {
              "INTERACTION_ID": {
                  "sourceName": "CORR_ID",
                  "targetType": "string"
              },
              "AURA_TM": {
                  "sourceName": "MESSAGE_TM",
                  "targetType": "string",
                  "preCalculated": "DATE_ISO_8691"
              },
              "AURA_CHANNEL_ID": {
                  "sourceName": "AURA_CHANNEL_ID",
                  "targetType": "string",
                  "nullable": "true"
              },
              "AURA_APP_ID": {
                  "sourceName": "AURA_APP_ID",
                  "targetType": "string",
                  "nullable": "true"
              },
              "COUNTRY_3_ALPHA_CD": {
                  "sourceName": "COUNTRY_3_ALPHA_CD",
                  "targetType": "string"
              },
              "AURA_SERVICE_NAME_CD": {
                  "sourceName": "AURA_PRESET_NAME",
                  "targetType": "string",
                  "defaultValue": "nlpaas",
                  "valueIfExist": "ai"
              },
              "AURA_PRESET_NAME": {
                  "sourceName": "AURA_PRESET_NAME",
                  "targetType": "string",
                  "nullable": "true"
              },
              "AURA_COMPONENT_ID": {
                  "sourceName": "AURA_COMPONENT_ID",
                  "targetType": "string"
              },
              "AURA_COMPONENT_HOST_ID": {
                  "sourceName": "AURA_COMPONENT_HOST_ID",
                  "targetType": "string"
              },
              "AURA_VERSION_ID": {
                  "sourceName": "AURA_VERSION_ID",
                  "targetType": "string"
              },
              "DAY_DT": {
                  "sourceName": "DAY_DT",
                  "targetType": "string",
                  "preCalculated": "CUSTOM_DATE_FORMAT",
                  "format": "YYYY-MM-DD"
              },
              "BRAND_ID": {
                  "sourceName": "BRAND_ID",
                  "targetType": "string"
              }
          }
      },
  

2.3 - Environment variables

Aura KPIs Uploader environment variables

2.4 - Aura KPIs dimensions

Aura KPIs dimensions

Introduction

aura-kpis-uploader-cli component generates the following types of KPIs dimensions:
. Channel type dimensions
. Recognizers type dimensions
. Skill type dimensions
. Components type dimensions
. Presets type dimensions
. Applications type dimensions

All of them are stored into the correspondent container in Aura Common Azure Storage (environment variable: AURA_KPIS_STORE_COMMON_CONTAINER).

Once the files are copied, the local copy is moved to a folder inside the container (environment variables: AURA_KPIS_STORE_COMMON_CONTAINER/ AURA_KPI_UPLOADER_PROCESSED_FOLDER) and kept there during a time, for recovering issues.

Types of KPIDimensionType:

export enum KPIDimensionTypes {
    KPIS_DIMENSIONS_CHANNEL = 'CHANNEL',
    KPIS_DIMENSIONS_RECOGNIZER = 'RECOGNIZER',
    KPIS_DIMENSIONS_SKILL = 'SKILL',
    KPIS_DIMENSIONS_APP = 'APP',
    KPIS_DIMENSIONS_PRESETS = 'PRESETS',
    KPIS_DIMENSIONS_COMPONENT = 'COMPONENT'
}

The path and filename should follow these patterns:

• Path: AURA_KPIS_ENTITIES_CONTAINER/avro/dimensional/<avro_schema: name>/<avro_schema: x-fp-version>
• Filename: <OB>_DIM_<DIM_NAME>_YYYYMMDDTHH0000Z.<AURA_KPI_FILES_EXTENSION>

The filename of the file is obtained as follows:

export function getDimensionFileName(KPIDimensionType: string) {
    return \`${getCountry().toUpperCase()}_\` +
        'DIM_' + \`${KPIDimensionType}_\` +
        \`${moment()
            .toISOString()
            .substring(0, 13)
            .replace(/-/g, '')
            .replace(/:/g, '') + '0000Z'}\` +
        \`${ConfigurationManager.instance.environmentConfiguration.AURA_KPI_FILES_EXTENSION}\`;
}

Channel type dimensions

aura-kpis-uploader-cli gets the list of channels configured in the environment through the aura-configuration-api channels endpoint.

It generates a file with the following schema:

• Channel Schema

The Adapter format:

 {
        "name": "D_Aura_Channel",
        "schema": "dimensional",
        "avroSchema": "aura-channel-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "CHANNEL"
        },
        "targetType": "avro",
        "fields": {
            "AURA_CHANNEL_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_CHANNEL_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "AURA_CHANNEL_SHORT_NAME": {
                "sourceName": "prefix",
                "targetType": "string"
            },
            "AURA_SKILL_ID": {
                "sourceName": "skillId",
                "targetType": "string",
                "defaultValue": "NO_SKILL"
            },
            "AURA_NLP_STAGES_ARRAY": {
                "sourceName": "nlp.stages",
                "targetType": "array",
                "nullable": "true"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "GBL_CONTACT_CHANNEL_ID": {
                "sourceName": "contact",
                "targetType": "string"
            },
            "BRAND_ID": {
                "sourceName": "brand",
                "targetType": "string"
            }
        }
    }

Recognizers type dimensions

aura-kpis-uploader-cli gets the recognizers configuration through the file configured in the project: aura-kpis-uploader/src/dimensions/resource/recognizers.json

It generates a file with the following schema:

• Recognizer Schema

The Adapter format:

  {
        "version": "1.0.0",
        "name": "D_Aura_Recognizer",
        "schema": "dimensional",
        "avroSchema": "aura-recognizer-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "RECOGNIZER"
        },
        "targetType": "avro",
        "fields": {
            "AURA_RECOGNIZER_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_RECOGNIZER_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Skill type dimensions

aura-kpis-uploader-cli gets the skill configuration through the aura-configuration-api Skills endpoint.

It generates a file with the following schema:

• Skill Schema

The Adapter format:

{
        "version": "1.0.0",
        "name": "D_Aura_Skill",
        "schema": "dimensional",
        "avroSchema": "aura-skill-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "SKILL"
        },
        "targetType": "avro",
        "fields": {
            "AURA_SKILL_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_SKILL_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTERNAL_SKILL_IND": {
                "sourceName": "external",
                "targetType": "boolean"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Components type dimensions

aura-kpis-uploader-cli gets the components configuration through the aura-configuration-api Components endpoint.

It generates a file with the following schema:

• Component Schema

The Adapter format:

{
        "version": "1.0.0",
        "name": "D_Aura_Component",
        "schema": "dimensional",
        "avroSchema": "aura-component-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "COMPONENT"
        },
        "targetType": "avro",
        "fields": {
            "AURA_COMPONENT_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_COMPONENT_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Presets type dimensions

aura-kpis-uploader-cli gets the presets configuration through the atria-model-gateway presets endpoint.

It generates a file with the following schema:

• Preset Schema

The Adapter format:

    {
        "version": "1.0.0",
        "name": "D_Aura_Preset",
        "schema": "dimensional",
        "avroSchema": "aura-preset-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "PRESETS"
        },
        "targetType": "avro",
        "fields": {
            "AURA_PRESET_ID": {
                "sourceName": "preset.id",
                "targetType": "string"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "preset.name",
                "targetType": "string"
            },
            "AURA_MODEL_ID": {
                "sourceName": "model.id",
                "targetType": "string"
            },
            "AURA_PRESET_GROUP_NAME_CD": {
                "sourceName": "preset.group",
                "targetType": "enum",
                "symbols": [
                    "enriched_ai",
                    "simple_ai"
                ]
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Applications type dimensions

aura-kpis-uploader-cli gets the applications configuration through the aura-configuration-api Applications endpoint.

It generates a file with the following schema:

• Application Schema

The Adapter format:

    {
        "version": "1.0.0",
        "name": "D_Aura_App",
        "schema": "dimensional",
        "avroSchema": "aura-app-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "APP"
        },
        "targetType": "avro",
        "fields": {
            "AURA_APP_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_APP_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "AURA_NLP_CHANNEL_ID": {
                "sourceName": "nlp.channeId",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_PRESET_NAMES_ARRAY": {
                "sourceName": "models.presets",
                "targetType": "array",
                "nullable": "true"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "BRAND_ID": {
                "sourceName": "brand",
                "targetType": "string"
            }
        }
    }

2.5 - Aura entities processing

Aura entities processing

Introduction

Currently, Aura entities processing is done both for CSV and Avro entities, that coexist in Aura.

These two processes are described in the following sections:

• Aura entities processing
  • Introduction
  • Aura entities processing for CSV files
  • Aura entities processing for AVRO files
    • aura-gateway-api Message Entity KPI
    • AUDIT Entity KPI

Aura entities processing for CSV files

Check Aura entities definition in CSV format here

aura-kpis-uploader-cli component aims to copy Aura’s entities files into Telefónica Kernel data storage.

It iterates all the folders where Aura servers store the entities files ( the aura-kpis container in the source Azure Storage account), copying them into the destination Azure Storage account (usually a Telefónica Kernel container).

Afterwards, the processed files are moved to the processed folder within the source container and, finally, the original files are removed.

sequenceDiagram
Title: Normal flow for a source component of Aura entities
    actor Cronjob
    participant uploadEntities
    participant processKPIS
    participant processBlobs
    participant processRemoteKPI
    participant StorageFileManager
    participant Azure Storage
    Cronjob ->> uploadEntities: Start process
    uploadEntities ->> processKPIS: Launch upload KPIS
    loop
      processKPIS ->> StorageFileManager: Get container client and blobs
      StorageFileManager ->> Azure Storage: 
      Azure Storage ->> StorageFileManager: OK
      StorageFileManager ->> processKPIS: OK
      processKPIS ->> processBlobs: Send all blobs to process
      processBlobs ->> processRemoteKPI: Filter and format blobs
      loop
        processRemoteKPI ->> StorageFileManager: Copy source blobs to destination container
        StorageFileManager ->> Azure Storage: 
        Azure Storage ->> StorageFileManager: OK
        StorageFileManager ->> processRemoteKPI: OK
        processRemoteKPI ->> StorageFileManager: Copy source blobs to source processed folder
        StorageFileManager ->> Azure Storage: 
        Azure Storage ->> StorageFileManager: OK
        StorageFileManager ->> processRemoteKPI: OK
        processRemoteKPI ->> StorageFileManager: Delete source blobs
        StorageFileManager ->> Azure Storage: 
        Azure Storage ->> StorageFileManager: OK
        StorageFileManager ->> processRemoteKPI: OK
      end
    end

The Adapter CSV entities definitions:

{
        "version": "1.0.0",
        "name": "E_Aura_BOT",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "BOT"
        },
        "targetType": "csv",
        "fields": {}
    },
    {
        "version": "1.0.0",
        "name": "E_Aura_CLF",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "CLF"
        },
        "targetType": "csv",
        "fields": {}
    },
    {
        "version": "1.0.0",
        "name": "E_Aura_GROOT",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "GROOT"
        },
        "targetType": "csv",
        "fields": {}
    },
    {
        "version": "1.0.0",
        "name": "E_Aura_NLP",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "NLP"
        },
        "targetType": "csv",
        "fields": {}
    }