Categories:
Aura entities processing
How Aura entities files are processed by aura-kpis-uploader and uploaded to Telefónica Kernel storage
Introduction
Currently, Aura entities processing is done both for CSV and Avro entities, that coexist in Aura.
These two processes are described in the following sections:
Aura entities processing for CSV files
Check Aura entities definition in CSV format here
aura-kpis-uploader-cli component aims to copy Aura’s entities files into Telefónica Kernel data storage.
It iterates all the folders where Aura servers store the entities files ( the aura-kpis container in the source Azure Storage account), copying them into the destination Azure Storage account (usually a Telefónica Kernel container).
Afterwards, the processed files are moved to the processed folder within the source container and, finally, the original files are removed.
sequenceDiagram
Title: Normal flow for a source component of Aura entities
actor Cronjob
participant uploadEntities
participant processKPIS
participant processBlobs
participant processRemoteKPI
participant StorageFileManager
participant Azure Storage
Cronjob ->> uploadEntities: Start process
uploadEntities ->> processKPIS: Launch upload KPIS
loop
processKPIS ->> StorageFileManager: Get container client and blobs
StorageFileManager ->> Azure Storage:
Azure Storage ->> StorageFileManager: OK
StorageFileManager ->> processKPIS: OK
processKPIS ->> processBlobs: Send all blobs to process
processBlobs ->> processRemoteKPI: Filter and format blobs
loop
processRemoteKPI ->> StorageFileManager: Copy source blobs to destination container
StorageFileManager ->> Azure Storage:
Azure Storage ->> StorageFileManager: OK
StorageFileManager ->> processRemoteKPI: OK
processRemoteKPI ->> StorageFileManager: Copy source blobs to source processed folder
StorageFileManager ->> Azure Storage:
Azure Storage ->> StorageFileManager: OK
StorageFileManager ->> processRemoteKPI: OK
processRemoteKPI ->> StorageFileManager: Delete source blobs
StorageFileManager ->> Azure Storage:
Azure Storage ->> StorageFileManager: OK
StorageFileManager ->> processRemoteKPI: OK
end
end
The Adapter CSV entities definitions:
{
"version": "1.0.0",
"name": "E_Aura_BOT",
"schema": "entity",
"avroSchema": "",
"source": {
"data": "csv",
"id": "BOT"
},
"targetType": "csv",
"fields": {}
},
{
"version": "1.0.0",
"name": "E_Aura_CLF",
"schema": "entity",
"avroSchema": "",
"source": {
"data": "csv",
"id": "CLF"
},
"targetType": "csv",
"fields": {}
},
{
"version": "1.0.0",
"name": "E_Aura_GROOT",
"schema": "entity",
"avroSchema": "",
"source": {
"data": "csv",
"id": "GROOT"
},
"targetType": "csv",
"fields": {}
},
{
"version": "1.0.0",
"name": "E_Aura_NLP",
"schema": "entity",
"avroSchema": "",
"source": {
"data": "csv",
"id": "NLP"
},
"targetType": "csv",
"fields": {}
}
Aura entities processing for AVRO files
Check Aura entities definition in Avro format here
There are two ways in which KPIs entities are created:
- One is through a component, for example, aura-gateway-api generates its KPIs while it is running.
- The other way is with the aura-kpis-uploader itself through the transformations of other KPIS already stored, for example
AUDIT, which is generated through the aura-gateway-api count in AVRO format and through the aura-groot KPI, which is in CSV format.
When generating KPIS in AVRO for entities, it is not done in the default folder for each entity AURA_KPIS_ENTITIES_CONTAINER/avro/entity/<avro_schema: name>/<avro_schema: x-fp-version>, but a temporary folder named current is used: AURA_KPIS_ENTITIES_CONTAINER/avro/current/entity/<avro_schema: name>/<avro_schema: x-fp-version>. This is done to separate the files already generated from those currently being generated. When the KPIS Uploader is run, it moves the files that have already been generated to the default folder and converts them to BlockBlob.
aura-gateway-api Message Entity KPI
It generates a file with the following schema:
Aura Gateway API Schema Definition
First the CSV data must be transformed into avro format:
{
"version":"1.0.1",
"name": "Aura_Gateway_Message_converter",
"stopWithErrors": true,
"order": 1,
"schema": "entity",
"avroSchema": "aura-gateway-message-asvc.json",
"versionSchema": "6.0.0",
"source": {
"data": "partialCsv",
"id": "GATEWAY",
"csvFolder": "gwapi",
"entityName": "GATEWAYMESSAGE",
"removeAfterProcessing": true
},
"targetType": "avro",
"fields": {
"MESSAGE_ID": {
"sourceName": "MESSAGE_ID",
"targetType": "string",
"defaultValue": "NO_MESSAGE_ID"
},
"MESSAGE_TM": {
"sourceName": "MESSAGE_TM",
"targetType": "string",
"preCalculated": "DATE_ISO_8691"
},
"AURA_APP_ID": {
"sourceName": "AURA_APP_ID",
"targetType": "string",
"nullable": "true"
},
"COUNTRY_3_ALPHA_CD": {
"sourceName": "COUNTRY_3_ALPHA_CD",
"targetType": "string",
"preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
},
"AURA_PRESET_NAME": {
"sourceName": "AURA_PRESET_NAME",
"targetType": "string",
"nullable": "true"
},
"DAY_DT": {
"sourceName": "MESSAGE_TM",
"targetType": "string",
"preCalculated": "CUSTOM_DATE_FORMAT",
"format": "YYYY-MM-DD"
},
"BRAND_ID": {
"sourceName": "BRAND_ID",
"targetType": "string"
},
"AURA_NLP_IND": {
"sourceName": "AURA_NLP_IND",
"targetType": "boolean"
},
"CORR_ID": {
"sourceName": "CORR_ID",
"targetType": "string",
"defaultValue": "NO_CORR_ID"
}
}
}
The next adapter process the Gateway from current folder to entity destination folder.
{
"version": "1.0.0",
"name": "Aura_Gateway_Message",
"schema": "entity",
"avroSchema": "aura-gateway-message-asvc.json",
"versionSchema": "6.0.0",
"source": {
"data": "object",
"id": "GATEWAY"
},
"targetType": "avro",
"fields": {}
}
AUDIT Entity KPI
It generates a file with the following schema: Audit Schema Definition
The Adapters format is shown below:
From current folder to entity destination folder:
{
"version": "1.0.0",
"name": "Aura_Audit",
"schema": "entity",
"avroSchema": "aura-audit-asvc.json",
"versionSchema": "6.0.0",
"source": {
"data": "object",
"id": "AUDIT",
"useDefaultTimeFilter": true
},
"targetType": "avro",
"fields": {}
}
From aura-gateway-api Message Entity KPI:
{
"version": "1.0.0",
"name": "Aura_Audit",
"stopWithErrors": true,
"schema": "entity",
"avroSchema": "aura-audit-asvc.json",
"versionSchema": "6.0.0",
"source": {
"data": "avro",
"id": "AUDIT",
"avroSchema": "aura-gateway-message-asvc.json",
"avroSchemaVersion": "6.0.0"
},
"targetType": "avro",
"fields": {
"INTERACTION_ID": {
"sourceName": "CORR_ID",
"targetType": "string"
},
"AURA_TM": {
"sourceName": "MESSAGE_TM",
"targetType": "string",
"preCalculated": "DATE_ISO_8691"
},
"AURA_CHANNEL_ID": {
"sourceName": "AURA_CHANNEL_ID",
"targetType": "string",
"nullable": "true"
},
"AURA_APP_ID": {
"sourceName": "AURA_APP_ID",
"targetType": "string"
},
"COUNTRY_3_ALPHA_CD": {
"sourceName": "COUNTRY_3_ALPHA_CD",
"targetType": "string",
"preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
},
"AURA_SERVICE_NAME_CD": {
"sourceName": "AURA_PRESET_NAME",
"targetType": "string",
"defaultValue": "nlpaas",
"valueIfExist" : "ai"
},
"AURA_PRESET_NAME": {
"sourceName": "AURA_PRESET_NAME",
"targetType": "string",
"nullable": "true"
},
"AURA_COMPONENT_ID": {
"sourceName": "AURA_COMPONENT_ID",
"targetType": "string"
},
"AURA_COMPONENT_HOST_ID": {
"sourceName": "AURA_COMPONENT_HOST_ID",
"targetType": "string"
},
"AURA_VERSION_ID": {
"sourceName": "AURA_VERSION_ID",
"targetType": "string"
},
"DAY_DT": {
"sourceName": "DAY_DT",
"targetType": "string",
"preCalculated": "CUSTOM_DATE_FORMAT",
"format": "YYYY-MM-DD"
},
"BRAND_ID": {
"sourceName": "BRAND_ID",
"targetType": "string"
}
}
}
From aura-groot Message Entity KPI:
{
"version": "1.0.0",
"name": "Aura_Audit",
"stopWithErrors": true,
"schema": "entity",
"avroSchema": "aura-audit-asvc.json",
"versionSchema": "6.0.0",
"source": {
"data": "partialCsv",
"id": "AUDIT",
"csvFolder": "groot",
"entityName":"GROOTMESSAGE"
},
"targetType": "avro",
"fields": {
"INTERACTION_ID": {
"sourceName": "CORR_ID",
"targetType": "string"
},
"AURA_TM": {
"sourceName": "MSG_DT",
"targetType": "string",
"preCalculated": "DATE_ISO_8691"
},
"AURA_CHANNEL_ID": {
"sourceName": "CHANNEL_ID",
"targetType": "string",
"nullable": "true"
},
"AURA_APP_ID": {
"sourceName": "AURA_APP_ID",
"targetType": "string",
"nullable": "true"
},
"COUNTRY_3_ALPHA_CD": {
"sourceName": "COUNTRY_CD",
"targetType": "string",
"preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
},
"AURA_SERVICE_NAME_CD": {
"sourceName": "AURA_SERVICE_NAME_CD",
"targetType": "string",
"defaultValue": "message"
},
"AURA_PRESET_NAME": {
"sourceName": "AURA_PRESET_NAME",
"targetType": "string",
"nullable": "true"
},
"AURA_COMPONENT_ID": {
"sourceName": "AURA_COMPONENT_ID",
"targetType": "string"
},
"AURA_COMPONENT_HOST_ID": {
"sourceName": "AURA_COMPONENT_HOST_ID",
"targetType": "string"
},
"AURA_VERSION_ID": {
"sourceName": "VERSION_ID",
"targetType": "string"
},
"DAY_DT": {
"sourceName": "MSG_DT",
"targetType": "string",
"preCalculated": "CUSTOM_DATE_FORMAT",
"format": "YYYY-MM-DD"
},
"BRAND_ID": {
"sourceName": "BRAND_ID",
"targetType": "string"
}
}
}