Aura – entity

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura entities definition

Aura entities definition versions:

Version 5.0.0 for those entities using CSV format
Version 6.0.0 for those entities using Avro format

Introduction

Aura entities definition includes the different entities currently written in Aura and required to calculate Aura KPIs and generate Aura’s invoices.

Entities are organized by types, as shown in the table below, together with the definition, how they are generated and its format, currently, CSV or Avro.

Entities types	Description	Generation	Format
Dimensions	They are Aura’s configuration settings that are needed to understand Aura’s entities	They are generated directly by aura-kpis-uploader from the content of the aura-configuration-api.	Avro
Message	Message entity is used to store information about the messages handled by aura-bot and the actions performed on them.	They are created in aura-bot.	CSV
Recognizer	Recognizer entity is used to store the request to any of the recognizers during the utterance recognition phase of the messages.	They are created by aura-groot, aura-bot and aura-nlp.	CSV
Extended Message	Extended Message entity is used to store extra information of a Message.	They are created in aura-bot.	CSV
Groot Message	Groot Message entity is used to store information about the messages handled by aura-groot.	They are created in aura-groot.	CSV
Gateway Message	Gateway Message entity is used to store information about the messages handled by aura-gateway-api.	They are created by aura-gateway-api.	Avro
Audit	Audit entity is used to store information about the interactions handled by Aura components. They will be used to generate Aura’s invoices.	They are created by aura-kpis-uploader based on the information received in Groot Message and Gateway Message.	Avro

Entities generation

Until Aura’s release 9.3.0 (Gwen Stefani), all entities in Aura were generated in CSV format.
The full procedure is explained in Aura Entities in CSV format.

In release 9.4.0 (Hannah Montana), the process to migrate Aura entities format from CSV to full-URM-compliant Avro format started. It is going to be a phased process, with entities being migrated in small groups to minimize the impact on services already consuming Aura entities in CSV.
The full procedure is explained in Aura Entities in Avro format.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura entities definition in Avro format

Aura entities definition version 6.0.0 (entities using Avro format), URM-compliant.

Introduction

The Aura entities definition includes the different entities currently written in Aura in Avro format, URM-compliant.

For further information regarding Kernel Avro datasets access the following links:

Entities generated in Avro URM-compliant format

Entities types	Description	Generation
Aura_Atria_Message	Summary of Aura’s interactions handled by aura-rag-server, aura-groot and aura-bot	Generated by aura-gateway-api
Aura_Audit	Entity used to store information about the interactions handled by Aura components. It will be used to generate Aura’s invoices.	Created by aura-kpis-uploader based on the information received in Groot Message and Gateway Message
Aura_Gateway_Message	Entity used to store information about the messages handled by aura-gateway-api.	Created by aura-gateway-api.
Aura_Message	Entity that contains the summary of Aura’s interactions handled by aura-bridge, aura-groot and aura-bot	Created directly by aura-groot and aura-bot
D_Aura_Dimensionals	They are Aura’s configuration settings that are needed to understand Aura’s entities	Generated directly by aura-kpis-uploader from the content of the aura-configuration-api.

Entities generation

Avro Entities tables nomenclature

Position: Field order in the Avro schema definition.
Field: Name of the field being defined. Usually, capitalized, although more aliases will be available in the schema definition.
PK: Parameter that indicates if the field is a Primary Key or not, that is, a value that uniquely identifies this field.
Referenced Entity: When the field references a value from another dataset, indicating that a join must be defined in the schema, this parameter contains the name of the dataset.
Referenced Entity Field: When the field references a value from another dataset, indicating that a join must be defined in the schema, this parameter contains the name of the field of the other dataset.
Type: Type of the field. It can be one of Avro logical types defined in Kernel.
Doc: Brief description of the field.
Nullable doc: If the field can be null, it should contain the cases when a null value is valid.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura_Atria_Message v6.0.0

Description of entities included in the entity type Aura_Atria_Message

Under implementation

This entity contains the summary of Aura’s interactions handled by aura-rag-server, aura-groot and aura-bot

Avro schema definition published in Kernel - TO BE COMPLETED

POSITION	FIELD	PK	Referenced Entity	Referenced Entity Field	TYPE	DESCRIPTION	NULLABLE_DES
1	MESSAGE_ID	YES			“type”:[“string”, “x-fp-unique-constraint”]	Unique ID of the current message.
2	MESSAGE_TM				“type”: “string” “logicalType”: “datetime”	Timestamp when the message is processed.
3	AURA_APP_ID		D_Aura_App	AURA_APP_ID	“type”: “string”	Identifier of the Aura application used by the user to interact with Aura. If due to an error, the application sent a wrongly formatted string without `app_id`, it will be set to null.	Null if due to an error, we are not capable of getting the incoming application.
4	COUNTRY_3_ALPHA_CD		D_Gbl_Country	COUNTRY_3_ALPHA_CD	“type”: “string”	Country of the operator running the current Aura deployment Three letters (alpha-3) code element of the country name (e.g. DEU) defined in ISO 3166-1
5	CORR_ID				“type”: “string”	Cross service transaction identifier that allows tracking a request through all the services and components.
6	AURA_PRESET_NAME		D_Aura_Preset	AURA_PRESET_NAME	“type”: “string”	Configuration used by the app to access Aura AI services.
7	SESSION_ID				“type”: “string”	Session identifier.
8	STAGE_NAME				“type”: [“null”, “string”]	Name of RAG stage	Null in the full request.
9	STAGE_EXECUTION_NUM				“type”: “int”	Stage execution counter.
10	STAGE_SEQUENCE_NUM				“type”: “int”	Stage execution order.
11	COMPLETION_TOKENS_NUM				“type”: “int”	Number of tokens the model used to respond to your request.
12	PROMPTS_TOKENS_NUM				“type”: “int”	Tokens used to represent the text sent in the request.
13	INPUT_QUERY_DESC				“type”: “string”	Stage input text.
14	OUTPUT_QUERY_DESC				“type”: [“null”, “string”]	Stage output text.
15	DURATION_QT				“type”: “string” logicalType: “duration”	Time spent per stage.	Null in error cases.
16	STATUS_CD				“type”: “string”	Aura status code. Values: `SUCCESS` or `ERROR`.
17	AURA_COMPONENT_ID		D_Aura_Component	AURA_COMPONENT_ID	“type”: “string”	Identifier of the Aura component that handled the current request, usually the name of the deployment running the corresponding Aura service. This name is consistent over time.
18	AURA_VERSION_ID				“type”: “string”	Aura platform version that produces this data.
19	DAY_DT				“type”: “string”, “logicalType”: “iso-date”	Year, month and day of the interaction.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura_Audit v6.0.0

Within the different entities currently used in Aura, this document describes those included in the entity type Aura_Audit

This entity contains the summary of Aura’s interactions intended for accountability purposes.

Avro schema definition published in Kernel

Position	Field	PK	Referenced entity	Referenced entity field	Type	Doc	Nullable Doc
1	INTERACTION_ID	YES			string; x-fp-unique-constraint:1	Unique ID of the current interaction. First part of the compound primary key of the dataset.
2	AURA_TM				string;datetime	Timestamp when the interaction happens.
3	AURA_CHANNEL_ID		D_Aura_Channel	AURA_CHANNEL_ID	string;null	Identifier of the Aura channel used by the user to interact with Aura.	If due to an error the channel sent a wrongly formatted string without `channelId` or `auraId`, it will be set to null.
4	AURA_APP_ID		D_Aura_App	AURA_APP_ID	string;null	Identifier of the Aura application used by the user to interact with Aura. If due to an error the application sent a wrongly formatted string without `app_id`, it will be set to null.	It will be null if due to an error, we are not capable of getting the incoming application. Also, if the request goes through aura-groot or aura-bridge
5	COUNTRY_3_ALPHA_CD		D_Gbl_Country	COUNTRY_3_ALPHA_CD	string	Three letters (alpha-3) code element of the country name (e.g. DEU) defined in ISO 3166-1
6	AURA_SERVICE_NAME_CD				enum;[message,nlpaas,ai,other]	Specific service consumed during the interaction. `message`: if the interaction happens in aura-groot `ai`: if the interaction calls generative services `nlpaas`: if the interaction calls Aura’s NLP as a service endpoint `other`: future use, for extra cases.
7	AURA_PRESET_NAME		D_Aura_Preset	AURA_PRESET_NAME	string;null	Preset used for the interaction, if using an AI service	It will be null for interactions not using AI services
8	AURA_COMPONENT_ID		D_Aura_Component	AURA_COMPONENT_ID	string;x-fp-unique-constraint:1	Identifier of the Aura component handling the incoming request of the user. Second part of the compound primary key of the dataset.
9	AURA_COMPONENT_HOST_ID				string	An identifier of the Aura component host that handled the current request, usually the pod identifier running the corresponding Aura service. This name changes every time a pod is reinitiated.
10	AURA_VERSION_ID				string	Aura platform version that produces this data.
11	DAY_DT				string;iso-date;partition-key	Year, month and day of the interaction.
12	BRAND_ID		D_Gbl_Brand	GBL_BRAND_ID	string	Brand id. Join with D_Gbl_Brand.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura_Gateway_Message v6.0.0

Within the different entities currently used in Aura, this document describes those included in the entity type Aura_Gateway_Message

This entity contains the summary of Aura’s interactions handled by aura-gateway-api.

Avro schema definition published in Kernel

Position	Field	PK	Referenced entity	Referenced entity field	Type	Doc	Nullable Doc
1	MESSAGE_ID	YES			string; x-fp-unique-constraint	Unique ID of the current message.
2	MESSAGE_TM				string;datetime	Timestamp when the message is processed
3	AURA_APP_ID		D_Aura_App	AURA_APP_ID	string;null	Identifier of the Aura application used by the user to interact with Aura. If due to an error the application sent a wrongly formatted string without app_id, it will be set to null.	It will be null if due to an error we are not capable of getting the incoming application.
4	COUNTRY_3_ALPHA_CD		D_Gbl_Country	COUNTRY_3_ALPHA_CD	string	Three letters (alpha-3) code element of the country name (e.g. DEU) defined in ISO 3166-1
5	CORR_ID				string	Cross service transaction identifier that allows tracking a request through all the services and components.
6	AURA_NLP_IND				boolean	Flag indicating whether or not this interaction goes to ATRIA NLP as a Service.
7	AURA_PRESET_NAME		D_Aura_Preset	AURA_PRESET_NAME	string;null	Configuration used by the app to access Aura AI services	If it is a NLP access no preset will be defined
8	AURA_COMPONENT_HOST_ID				string	An identifier of the aura component host that handled the current request, usually the pod identifier running the corresponding aura service. This name changes every time a pod is reinitiated.
9	AURA_COMPONENT_ID		D_Aura_Component	AURA_COMPONENT_ID	string	An identifier of the aura component that handled the current request, usually the name of the deployment running the corresponding aura service. This name is consistent over time.
10	AURA_VERSION_ID				string	Aura platform version that produces this data.
11	DAY_DT				string;iso-date;partition-key	Year, month and day of the interaction
12	BRAND_ID		D_Gbl_Brand	GBL_BRAND_ID	string	Brand id. Join with D_Gbl_Brand

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura_Gateway_Message v6.0.0

Within the different entities currently used in Aura, this document describes those included in the entity type Aura_Gateway_Message

NOTE: This entity is not used in production environments.

This entity contains the summary of Aura’s interactions handled by aura-gateway-api.

Position	Field	PK	Referenced entity	Referenced entity field	Type	Doc	Nullable Doc
1	MESSAGE_ID	YES			string; x-fp-unique-constraint	Unique ID of the current message.
2	MESSAGE_TM				string;datetime	Timestamp when the message is processed
3	AURA_APP_ID		D_Aura_App	AURA_APP_ID	string;null	Identifier of the Aura application used by the user to interact with Aura. If due to an error the application sent a wrongly formatted string without app_id, it will be set to null.	It will be null if due to an error we are not capable of getting the incoming application.
4	COUNTRY_3_ALPHA_CD		D_Gbl_Country	COUNTRY_3_ALPHA_CD	string	Three letters (alpha-3) code element of the country name (e.g. DEU) defined in ISO 3166-1
5	CORR_ID				string	Cross service transaction identifier that allows tracking a request through all the services and components.
6	AURA_NLP_IND				boolean	Flag indicating whether or not this interaction goes to ATRIA NLP as a Service.
7	AURA_PRESET_NAME		D_Aura_Preset	AURA_PRESET_NAME	string;null	Configuration used by the app to access Aura AI services	If it is a NLP access no preset will be defined
8	AURA_COMPONENT_HOST_ID				string	An identifier of the aura component host that handled the current request, usually the pod identifier running the corresponding aura service. This name changes every time a pod is reinitiated.
9	AURA_COMPONENT_ID		D_Aura_Component	AURA_COMPONENT_ID	string	An identifier of the aura component that handled the current request, usually the name of the deployment running the corresponding aura service. This name is consistent over time.
10	AURA_VERSION_ID				string	Aura platform version that produces this data.
11	DAY_DT				string;iso-date;partition-key	Year, month and day of the interaction
12	BRAND_ID		D_Gbl_Brand	GBL_BRAND_ID	string	Brand id. Join with D_Gbl_Brand
13	USER_ID				string	Identifier of the user sending the message	YES

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura_Message v6.0.0

Within the different entities currently used in Aura, this document describes those included in the entity type Aura_Message

Under implementation

This entity contains the summary of Aura’s interactions handled by aura-bridge, aura-groot and aura-bot

Avro schema definition published in Kernel - TO BE COMPLETED

Position	Field	PK	Referenced Entity	Referenced Entity	Type	Description	Nullable_Des
1	MESSAGE_ID	Yes			“type”: “string” {x-fp-unique-constraint}	Unique ID of the current message.
2	MESSAGE_TM				“type”:“string” “logicalType”: “datetime”	Timestamp when the message happens.
3	ACTION_CD				“type”:“enum” “values”: [‘send’, ‘receive’, ’event’, ‘audit’]	Code of the action that produces the data.
4	AURA_ID				“type”: [“null”, “string”] “x-fp-data-protection”: “pseudonymize”	Unique user login ID in Aura. User might have a new `AURA_ID` with each logging in the channel or when the previous authorization in Kernel expires.	Null if the channel fails to send it
5	AURA_CHANNEL_ID		D_Aura_Channel	AURA_CHANNEL_ID	“type”: [“null”, “string”]	Identifier of the Aura channel used by the user to interact with Aura.	Null if the channel wrongly sends a string without channelId or auraId
6	COUNTRY_3_ALPHA_CD		D_Gbl_Country	COUNTRY_3_ALPHA_CD	“type”:“string”	Country of the operator running the current Aura deployment.
7	CORR_ID				“type”: “string”	Cross-service transaction ID for tracking requests through all services and components.
8	AURA_COMPONENT_HOST_ID				“type”: “string”	Identifier of the Aura component host handling the request, usually the pod identifier running the corresponding Aura service. This name changes with each pod restart.
9	AURA_COMPONENT_ID		D_Aura_Component	AURA_COMPONENT_ID	“type”: “string”	Identifier of the Aura component handling the request (usually, the deployment running the corresponding Aura service). This name is consistent over time.
10	AURA_VERSION_ID				“type”: “string”	Aura platform version that produces this data.
11	DURATION_QT				type": “string” logicalType: “duration” default: “PT0S”	Duration of the action to handle the current message. For inbound, it is `0` (it only logs that message is in the system); For Outbound, it logs the number of milliseconds to process message and send response back to the channel.
12	AURA_STATUS_CD				“type”: “string”	Aura status code. “SUCCESS” for inbound; For outbound, it sends the corresponding status. For v1 requests, if no status nor error is provided, then SUCCESS is sent.
13	AURA_SKILL_ID		D_Aura_Skill	AURA_SKILL_ID	“type”: “string”, null	Identifier of the skill handling the message.	Null if no skill-bot is available to handle the message due to an error.
14	AURA_CHANNEL_CONVERSATION_ID				“type”: “string”	Identifier of the conversation in the channel.
15	AURA_SKILL_CONVERSATION_ID				“type”: “string”,null	Identifier of the conversation in the skill.	Null for incoming messages or outgoing ones in case of error.
16	WIN_RECOGNIZER_ID		D_Aura_Recognizer	AURA_RECOGNIZER_ID	“type”: “string”,null	Identifier of the recognizer that returns the top score.	Null for incoming messages.
17	WIN_RECOGNIZER_SCORE_QT				“type”:“float”	Score of the recognizer with higher score for this message.
18	RESULT_INTENT_NAME				“type”: “string”,null	Intent resolved by the recognizer, that is, the Aura action that resolves the intention of the user, related with the use case triggered by the user utterance. As example, in the sentence: “I want to see a movie of Lars von Trier”, NLP would return “intent.tv.play” as user intention.	Null for incoming messages.
19	RESULT_INTENT_ENTITIES_ARRAY				“type”: “array[RESULT_INTENT_ENTITIES_ARRAY_ITEM]”, null	Recognized entities, as part of the action in Aura to resolve the user’s intention. For instance, in TV use cases, recognizable entities can be: movies, actors, directors, series. As example, in the sentence: “I want to see a movie of Lars von Trier”, NLP would return two entities: one of type `audiovisual_content` with “movie” as name, and one of type `movie_director` with “lars von trier” as name.	Null for incoming messages and for those intents that do not define them.
19.1	RESULT_INTENT_ENTITIES_ARRAY_ITEM.INTENT_ENTITY_NAME				“type”: “string”	Entity name handled by Aura services. It corresponds with the string extracted from the user’s utterance that matches any of the defined entities of any of the types. See example above.
19.2	RESULT_INTENT_ENTITIES_ARRAY_ITEM.INTENT_ENTITY_TYPE_DES				“type”: “string”	Entity type handled by Aura services. It corresponds with the type of the entity extracted from the user’s utterance that matches any of the defined entities of any of the types. See example above.
19.3	RESULT_INTENT_ENTITIES_ARRAY_ITEM.INTENT_ENTITY_CANON_NAME				“type”: “string”	Entity type handled by Aura services. It corresponds with the canonical value of the entity extracted from the user’s utterance that matches any of the defined entities of any of the types. In the previous example: entity of type audiovisual_content: “movie” as name and “film” as canon; entity of type “movie_director”: “lars von trier” as name and “Lars von Trier” as canon.
19.4	RESULT_INTENT_ENTITIES_ARRAY_ITEM.INTENT_ENTITY_LABEL_NAME				type: string, null	Entity type handled by Aura services. It corresponds with a common alias of the entity type extracted from the user’s utterance that matches any of the defined entities of any of the types. In the previous example: entity of type audiovisual_content (“movie” as name, “movie” as canon and “MOV” as label; entity of type “movie_director”: “lars von trier” as name,“Lars von Trier” as canon, and without label).	Labels for entities must have been introduced in the training set.
20	ENRICHMENT_ARRAY				“type”: “array[ENRICHMENT_ARRAY_ITEM]”, null	Contextual information to enrich the request to get a more efficient recognition.	Null for incoming messages and for those messages where no enrichment pipeline was executed.
20.1	ENRICHMENT_ARRAY_ITEM.ENRICHMENT_TYPE_NAME				`string`	Type of data added as contextual information. For instance: sentiment, profile, language (not a closed list).
20.2	ENRICHMENT_ARRAY_ITEM.ENRICHMENT_PROVIDER_NAME				type: string	Specific component executed that provides meaningful information for this request. For instance, regarding language, a provider to detect the language or to translate it.
20.3	ENRICHMENT_ARRAY_ITEM.ENRICHMENT_VALUE_DES				type: string	Specific value returned by the enricher. For instance, in sentiment analysis: happy, sad, angry.
20.4	ENRICHMENT_ARRAY_ITEM.ENRICHMENT_VALUE_DURATION_QT				“type”: “string”, “logicalType”: “duration”,default: “PT0S”	Duration of the request execution to obtain this contextual information.
21	MESSAGE_DES				“type”: “string”, null	Incoming/outgoing information sent by the user or by Aura. Incoming: it can contain: text sent by the user, for simpler cases; auraCommand sent by the channel; prompt option clicked or selected by the user when responding to a prompt; response of any asynchronous API called by aura-bot, such as when calling handover systems. Outgoing: text sent by Aura, titles, texts and button values of the cards sent as attachments, name of files sent as attachments.	Null if no message is sent back and forth.
22	MODALITY_CD				“type”: “enum” “values”: [“form”, “text”, “voice”]	The form used by the user to send the interaction to Aura. Values: “form”, if the user clicks a button to send the message; “text”, if the input is a written text; “voice”, if the user sends an utterance, regardless of whether it was passed to text before calling Aura.
23	CHANNEL_DATA_VERSION_NUM				“type”: “integer”	channelData request and response version used by the channel. If none is sent, the default version handled by Aura is assumed.
24	AURA_GLOBAL_ID				“type”: “string” “x-fp-data-protection”: “pseudonymize”	It identifies the user id logged in Kernel (USER_4P_ID) with the same authentication method. If there is no information regarding the `AURA_ID`, a value based on the default `AURA_ID` will be set, but calculated with the same method as any other `AURA_GLOBAL_ID`
25	USER_4P_ID				“type”: “string” “x-fp-user-id”: true, “x-fp-data-protection”: “pseudonymize”	Unique identifier of the user (data owner) in Kernel platform. This identifier must coincide with the `user_id` parameter of the Kernel APIs, as they both refer to the same concept. Example values: “3706277557884218994”, “-6189348075566519429”.
26	USER_IDENTIFICATION_CD				“type”: “enum” “values”: [“anonymous”, “identifiable”, “authenticated”]	Flag to indicate if the user can be identified univocally. If she is already authenticated (her `user_4p_id` is a a real one), or it is not authenticated, (the `user_4p_id` is generated based on her `aura_id` and, in this case, shows if the `aura_id` is completely random (anonymous, for example, based on a cookie of a webpage) or corresponds to a unique user (identifiable, for example, based on the `whatsapp_id` of a user).
27	ADMINISTRATIVE_NUMBER				“type”: “string”, null	Identifier of customer in video platform. A user may have more than one administrative number, if she has more than one video product.
28	EXT_APP_ID				“type”: “string”, null	Identifier of the application that is calling Aura.	It will be null if the application does not send it to Aura.
29	EXT_APP_SESSION_ID				“type”: “string”, null	Identifier of the session handled by the application that is calling Aura.	Null if the application does not send it to Aura.
30	DAY_DT				“type”: “string” “logicalType”:“iso-date” partition key	Year, month and day of the interaction.
31	BRAND_ID		D_Gbl_Brand	GBL_BRAND_ID	“type”: “string”	Commercial brand global identifier (codified according to `D_Gbl_Brand`) used to differentiate among different brands in the same OB.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Dimensional entities

Within the different entities currently used in Aura, this document describes those included as dimensions, that hold the configuration of each Aura instance.

D_Aura_App schema definition v6.0.0

This entity contains the list of possible Apps defined in Aura.

Avro schema definition published in Kernel

Position	Field	PK	Referenced entity	Referenced entity field	Type	Doc	Nullable Doc
1	AURA_APP_ID	YES			string; x-fp-unique-constraint:1	Unique identifier of the Aura application.
2	AURA_APP_NAME				string; x-fp-unique-constraint:2	Name of the Aura application.
3	AURA_NLP_CHANNEL_ID		D_Aura_Channel	AURA_CHANNEL_ID	string;null	Identifier of the Aura channel configured for this application.	It will be null if the application does not count on nlpaas configuration.
4	AURA_PRESET_NAMES_ARRAY		D_Aura_Preset	AURA_PRESET_NAME	Array[string];null	List of presets available for the current application.	It will be null if the application does not count on LLM configuration.
5	EXTRACTION_TM				string; datetime	Timestamp of the data.
6	BRAND_ID		D_Gbl_Brand	GBL_BRAND_ID	string	Brand id. Join with D_Gbl_Brand

D_Aura_Channel v6.0.0

This entity contains the list of possible channels defined in Aura

Avro schema definition published in Kernel

Position	Field	PK	Referenced entity	Referenced entity field	Type	Doc	Nullable Doc
1	AURA_CHANNEL_ID	YES			string; x-fp-unique-constraint:1	Identifier of the Aura channel
2	AURA_CHANNEL_NAME				string; x-fp-unique-constraint:2	Name of the Aura channel
3	AURA_CHANNEL_SHORT_NAME				string	Short name of the Aura channel
4	AURA_SKILL_ID		D_Aura_SKill	AURA_SKILL_ID	string	Identifier of the skill that handles this channel
5	AURA_NLP_STAGES_ARRAY				Array[string];null	List of the nlp stages configured for this channel	The value can be null if the channel does not count on nlp configuration
6	GBL_CONTACT_CHANNEL_ID		D_Gbl_Contact_Channel	GBL_CONTACT_CHANNEL_ID	string	Identifier of the contact channel. Join with D_Gbl_Contact_Channel v5
7	BRAND_ID		D_Gbl_Brand	GBL_BRAND_ID	string	Brand id. Join with D_Gbl_Brand v6
5	EXTRACTION_TM				string; datetime	Timestamp of the data.

D_Aura_Component v6.0.0

This entity contains the list of possible components defined by Aura

Avro schema definition published in Kernel

Position	Field	PK	Type	Doc
1	AURA_COMPONENT_ID	YES	string; x-fp-unique-constraint:1	Aura identifier for each component available in Aura.
2	AURA_COMPONENT_NAME		string; x-fp-unique-constraint:2	Aura component name
3	EXTRACTION_TM		string; datetime	Timestamp of the data.

D_Aura_Preset v6.0.0

This entity contains the list of possible presets defined by Aura

Avro schema definition published in Kernel

Position	Field	PK	Type	Doc
1	AURA_PRESET_ID	YES	string; x-fp-unique-constraint:1	Unique identifier of the Aura preset.
2	AURA_PRESET_NAME		string; x-fp-unique-constraint:2	Aura preset name
3	AURA_MODEL_ID		string	Identifier of the model that will run with this preset
4	AURA_PRESET_GROUP_NAME_CD		enum;[simple_ai,enriched_ai]	Type of the preset that means the underlying models and stages that will run with this preset
5	EXTRACTION_TM		string; datetime	Timestamp of the data.

D_Aura_Recognizer v6.0.0

This entity contains the list of possible recognizers defined by Aura

Avro schema definition published in Kernel

Position	Field	PK	Type	Doc
1	AURA_RECOGNIZER_ID	YES	string; x-fp-unique-constraint:1	Identifier of the exact recognizer
2	AURA_RECOGNIZER_NAME		string; x-fp-unique-constraint:2	Recognizer code
3	EXTRACTION_TM		string; datetime	Timestamp of the data.

D_Aura_Skill v6.0.0

This entity contains the list of possible skills defined by Aura

Avro schema definition published in Kernel

Position	Field	PK	Type	Doc
1	AURA_SKILL_ID	YES	string; x-fp-unique-constraint:1	Unique identifier
2	AURA_SKILL_NAME		string; x-fp-unique-constraint:2	Skill code
3	EXTERNAL_SKILL_IND		boolean	Flag to indicate whether it is an internal skill based in aura-bot or an external skill, done using any technology.
4	EXTRACTION_TM		string; datetime	Timestamp of the data.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura KPIs uploader user guide

Guidelines including the orderly steps to use aura-kpis-uploader

1. Prepare your `CONFIG_FILE`

First, review and prepare your environment variables in order to create your CONFIG_FILE.

An example is shown below:

AURA_AUTHORIZATION_HEADER=${AURA_AUTHORIZATION_HEADER}
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT=${AURA_CHANNELS_CONFIGURATION_API_ENDPOINT}
AURA_DEFAULT_LOCALE=${AURA_DEFAULT_LOCALE}
AURA_ENVIRONMENT_NAME=${AURA_ENVIRONMENT_NAME}
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY}
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT}
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION}
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION}
AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION}
AURA_VERSION=${AURA_VERSION}

2. Launch Aura KPIs uploader

There are two methods for launching the aura-kpis-uploader

Directly with the source code: to launch it in your local environment
As a docker container: to use it with the already generated Docker image.

2.1. Launch the tool directly with the source code

$ cd aura-kpis-uploader
$ npm install
$ export CONFIG_FILE=[PATH_TO_MY_CONFIG_FILE]
$ npm run start

2.2. Launch the tool as a docker container

2.2.1. Pull image from repository

This method needs login first.

docker pull auraregistry.azurecr.io/aura/aura-kpis-uploader

2.2.2. Run Docker container

There are two options for running the Docker container:

With .env file:

Change $VERSION accordingly, exporting it or changing the string in the commands.

If $VERSION is not present, it will use the latest default tag.

docker run --rm -e CONFIG_FILE=/opt/aura-kpis-uploader/env/config_file.env \
        -v $(pwd):/opt/aura-kpis-uploader/env \
        auraregistry.azurecr.io/aura/aura-kpis-uploader:$VERSION

With environmental variables in command:

Note that host.docker.internal only works for MACOS/Windows systems:

docker run -e AURA_AUTHORIZATION_HEADER=${AURA_AUTHORIZATION_HEADER} \
        -e AURA_CHANNELS_CONFIGURATION_API_ENDPOINT=${AURA_CHANNELS_CONFIGURATION_API_ENDPOINT} \
        -e AURA_DEFAULT_LOCALE=${AURA_DEFAULT_LOCALE} \
        -e AURA_ENVIRONMENT_NAME=${AURA_ENVIRONMENT_NAME} \
        -e AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY} \
        -e AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT} \
        -e AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION} \
        -e AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION} \
        -e AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION} \
        -e AURA_VERSION=${AURA_VERSION} \
        -v ${HOST_DIRECTORY}:/opt/aura-kpis-uploader/logs \
        auraregistry.azurecr.io/aura/aura-kpis-uploader:$VERSION

3. Execution Dependencies

With the addition of the new component aura-databricks-jobs, now for Avro entities, the execution of aura-kpis-uploader depends on aura-databricks-jobs being executed correctly.

This is done to avoid duplicating KPIs when they are processed by aura-databricks-jobs, and an error has occurred.

3.1 Evaluation of conditions for execution

If avro-to-dataset-job-cli has been executed correctly, it will write a file to the default avro folder containing a date. If this file exists and the date it contains does not exceed the time set for the scheduled run in Databricks, aura-kpis-uploader-cli will run its processes.

If avro-to-dataset-job-cli has generated any error in its execution, it will generate a file in the same folder containing internally the error(s) that have occurred.

Environment variables involved:

AURA_KPIS_AVRO_DESTINATION_PATH: Default value is avro.
AURA_DATABRICKS_ERROR_FILENAME: Default value is databricks.ERROR.
AURA_DATABRICKS_EXECUTION_PERIOD: Default value is 24 hours.

The flow that Aura KPIs Uploader follows to validate if it is going to be executed is as follows:

4 Generate Reports

By default, aura-kpis-uploader generates reports if any error has occurred in the upload process.

These reports are available in the Azure Storage defined in AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT, path aura-kpis/reports/.

If you want to change the behavior and generate reports of all uploaded files or disable their generation, you can do it by changing the environment variable AURA_KPIS_REPORTS_MODE. If the value is set to all, it will generate a report for each of the processed files and if it is set to none, it will not generate any report. The default value is error.

4.1 Report upload Files

When an error occurs, the system will send an event to Prometheus to alert about the error.

 this.prometheusHandler.addSummary(
                        { 
                        metricId: PrometheusMetrics.auraKpisUploaderReportErrorSummary, 
                        start: this.startTimer 
                        },
                        { 
                        reportError: reportLink },
                        this.corr
                        );

The event will contain the metricId auraKpisUploaderReportErrorSummary, when an error occurs and a link to the report containing the error.

4.1.1 Report Model

A report will contain the following template in JSON format:

numberOfFiles: Number of files processed.
numberOfFilesSkipped: Number of files that have been skipped because they have not yet been processed.
NumberOfFilesMovedToProcessed: Number of files that have been moved to the processed folder.
numberOfFilesUploaded : Number of files that have been successfully uploaded.
NumberOfFilesDeleted: Number of files that have been deleted from the main folder.
numberOfFilesUploadValidated: Number of files that have been verified as successfully uploaded.

If errors have occurred, it will contain an errors property with the files that have failed, arranged as properties:

key: Name of the file that has failed.
step: It indicates in which phase it has failed. The possible ones are “UPLOADING”, “CHECK_IF_UPLOADED”, “MOVING_TO_PROCESSED” AND “REMOVING”.
error: Error message obtained.
corr: Associated correlator.

Example:

{
 "numberOfFiles": 43,
 "numberOfFilesSkipped": 0,
 "numberOfFilesMovedToProcessed": 41,
 "numberOfFilesUploaded": 41,
 "numberOfFilesDeleted": 41,
 "numberOfFilesUploadValidated": 41,
 "errors": {
   "BOT_da5df187-8aa9-55b7-94d6-50586bd6b0b8_CR_API_20240216T080000Z.txt": {
     "step": "UPLOADING",
     "error": "getaddrinfo ENOTFOUND aurautils-test.blob.core.windows.net",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   },
   "BOT_b8efccfd-f2bd-5ef0-a19d-b2414ab94dc8_CR_API_20240215T160000Z.txt": {
     "step": "UPLOADING",
     "error": "getaddrinfo ENOTFOUND aurautils-test.blob.core.windows.net",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   }
 }
}

4.2 Report Generating Avro KPIs

When an error occurs, the system will send an event to Prometheus to alert about the error.

 this.prometheusHandler.addSummary(
                        { 
                        metricId: PrometheusMetrics.auraKpisGeneratedReportErrorSummary, 
                        start: this.startTimer 
                        },
                        { 
                        reportError: reportLink },
                        this.corr
                        );

The event will contain the metricId auraKpisGeneratedReportErrorSummary, when an error occurs and a link to the report containing the error.

4.2.1 Report Model

A report will contain the following template in JSON format.

numberBlobsLoaded: Number of blobs loaded.
numberDataRowsLoaded: Number of total lines loaded from blobs.
numberDataRowsGenerated: Number of Avro registers generated.
numberEntitiesGenerated: Number of entities generated.
numberDimensionalGenerated: Number of dimensional files generated.

If errors have occurred, it will contain an errors property with the files that have failed, arranged as properties:

key: The id of the source. Example: ‘AUDIT’.
error: Error message obtained.
corr: Associated correlator.

Example:

{
 "numberBlobsLoaded": 13,
 "numberDataRowsLoaded": 0,
 "numberDataRowsGenerated": 141,
 "numberEntitiesGenerated": 2,
 "numberDimensionalGenerated": 7,
 "errors": {
   "GATEWAY": {
     "error": "Error parsing data. Field name does not exist in source",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   },
   "AUDIT": {
     "error": "Error parsing data. Field id cannot be empty",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   }
 }
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura KPIs Adapter Manager

Module to manage the copy of CSV files to Kernel and to generate Avro dimensional files and Avro Entity files.

⚠️ Although the definition of the module refers to Avro files, it also handles CSV files, until all CSV files are migrated to Avro in upcoming releases

Definition

Set of classes that will be in charge of processing the necessary operations to copy the KPIs in CSV format to Kernel and to process or generate the KPIs of the entities and dimensions in Avro format.

This is done from a definition file that, by default, is called aura-avro-adapter.json and is defined in the AURA_SOURCE_PATH_AVRO_ADAPTERS environment variable. This file is remotely loaded from the Storage AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT and the path AURA_KPIS_ENTITIES_CONTAINER/schemas/AURA_SOURCE_PATH_AVRO_ADAPTERS. If the file does not exist remotely, a local one is loaded by default.

Adapter Definition File Model

For each KPI file to be managed, a model is defined containing an array with all the adaptation operations to be performed. The content format of this file is JSON.

Name	Type	Definition
name	string	Name of the adapter. To manage Avro files, this field contains the name of the Avro schema itself
schema	AvroAdapterSchema	Type of KPI, entity or dimensional
avroSchema	string	If the destination file is an Avro file, this field contains the Avro schema to generate the file.
source	AvroAdapterSource	Contains the necessary data from where you are going to copy or transform the data, if applicable.
fields	AvroAdapterField	Contains a set of properties with a model defined in AvroAdapterFieldModel
targetType	AvroAdapterTargetType	Type file to generate. Values: `csv` or `avro`.
versionSchema	string	If the file to generate is Avro, this field contains the version of its schema
stopWithErrors	boolean	When `true`, if another adapter has generated an error, this one will not be executed
expiresAt	date	If set, it indicates which date the intended KPI will no longer be processed. This is used when there are different versions of AVRO templates and you want to keep both versions until the end of a cycle, which is usually monthly.
order	number	Contains the execution order, the order 1 will be the first to execution. By default the order is MAX_SAFE_INTEGER.

AvroAdapterSchema

Name	type	Definition
dimensional	string	Type of KPI is dimensional.
entity	string	Type of KPI is an entity.

AvroAdapterSource

Name	type	Definition
data	AvroAdapterSourceDataType	Type of file to use as source.
id	any	KPI identifier, for example: `GATEWAY`, `AUDIT`, `BOT`, etc.
avroSchema	string	Name of the schema file. Required if the source is an Avro file, as its schema needs to read it.
avroSchemaVersion	string	Version of the schema, required if the source is an Avro file.
csvFolder	string	Name of the folder where this CSV is stored, required if the source is a partial CSV.
entityName	string	Entity name of the CSV to load, required if the source is a partial CSV.
useDefaultTimeFilter	boolean	If `true`, the adapter will ignore the pre-calculated date range at the beginning of the process and will use the default one, which is calculated based on `AURA_DAYS_INTERVAL` as the start date and the end date by subtracting `AURA_KPIS_TIME_FILE_IS_IN_USE_AMOUNT` * `AURA_KPIS_TIME_FILE_IS_IN_USE_UNIT` from the current date. The default is 30 days prior to the current time.
version	string	Version of the adapter. It is used to propagate changes in a release and to update the remote file where the adapter models are stored in `aura-avro-adapter.json`. Format: x.y.z
removeAfterProcessing	boolean	Remove source data after processing. The data won’t store in processed data folder.

AvroAdapterSourceDataType

Name	type	Definition
avro	string	Name of the source file for Avro.
object	string	Name of the source file for the Javascript object.
csv	string	Name of the source file for CSV.
partialCsv	string	Name of the source file when it is referred to specific data from a CSV file.

AvroAdapterField

Name	type	Definition
any	AvroAdapterFieldModel	Name of the field.

AvroAdapterFieldModel

Name	type	Definition
sourceName	string	Name of the source field from which data is obtained.
nullable	boolean	It indicates whether the target field can be null or not.
preCalculated	AvroPreCalculatedType	String that indicates the type of calculation needed. For example, the required format for dates or numbers.
targetType	Avrofieldbasetype	Type of data for the target data.
defaultValue	string	number
symbols	any[]	Array with the values searched in the origin data. If the data does not match one of those values, it will generate an error.
format	string	Specific format to be sent, for precalculated fields.
description	string	Human-readable description of the field.
symbolsValues	string	If we want to change the value of the destination based on the values defined in the `symbols` field, we can define it in this field in the form of an object.
valueIfExist	string	If the source data exists, it will be changed to the one defined in this field.

AvroPreCalculatedType

Name	type	Definition
DATE_ISO_8691	string	Format used for dates. The input date will be converted to this format. If there is no input date, the current date is obtained.
CUSTOM_DATE_FORMAT	string	Custom format for dates. Example: MM/DD/YYY
DURATION_ISO_8601	string	Format used to set a duration.
COUNTRY_ISO_3166_ALPHA_3	string	Format to indicate a country that complies with the ISO 3166 format.

Avrofieldbasetype

Name	type	Definition
string	string	Type string.
number	string	Type number.
enum	string	Type enumerable.
array	string	Type array.
boolean	string	Type boolean.

AvroAdapterTargetType

Name	type	Definition
csv	string	CSV file.
avro	string	Avro file.

Types of adapters

The Adapter manager can perform several types of processes, that will be explained in the following sections.

Copy CSV files

This adapter will copy the CSV files generated from the Aura servers (aura-bot, aura-groot, aura-nlp, etc.) to the Kernel storage.

{
        "version": "1.0.0",
        "name": "E_Aura_BOT",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "BOT"
        },
        "targetType": "csv",
        "fields": {}
    }

Generate Dimensional KPIs in Avro

This adapter will generate an Avro file with Aura available RECOGNIZER data, as they are defined only in aura-kpis-uploader and not in aura-configuration-api.

  {
        "version": "1.0.0",
        "name": "D_Aura_Recognizer",
        "schema": "dimensional",
        "avroSchema": "aura-recognizer-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "RECOGNIZER"
        },
        "targetType": "avro",
        "fields": {
            "AURA_RECOGNIZER_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_RECOGNIZER_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Copy Avro files

This adapter copies the KPIs generated in Avro by aura-gateway-api to be processed by aura-databricks-jobs.

    {
        "version": "1.0.0",
        "name": "Aura_Gateway_Message",
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "GATEWAY"
        },
        "targetType": "avro",
        "fields": {}
    }

Generate Avro from another Avro

This is an adapter that generates an AURA_AUDIT file based on the data from the AVRO file of the aura-gateway-api KPIs.

{       "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "avro",
            "id": "AUDIT",
            "avroSchema": "aura-gateway-message-asvc.json",
            "avroSchemaVersion": "6.0.0"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "AURA_CHANNEL_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "defaultValue": "nlpaas",
                "valueIfExist" : "ai"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "AURA_VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "DAY_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

Generate Avro from data in CSV file

This is an adapter that generates an AURA_AUDIT file based on the data from the CSV file of the aura-groot Message KPIs.

{
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "order": 2,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "AUDIT",
            "csvFolder": "groot",
            "entityName":"GROOTMESSAGE"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "CHANNEL_ID",
                "targetType": "string",
                  "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_SERVICE_NAME_CD",
                "targetType": "string",
                "defaultValue": "message"

            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

Another example, deleting CSV source file data when the adapter finalize its execution.

{
        "version":"1.0.1",
        "name": "Aura_Gateway_Message_converter",
        "stopWithErrors": true,
        "order": 1,
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "GATEWAY",
            "csvFolder": "gwapi",
            "entityName": "GATEWAYMESSAGE",
            "removeAfterProcessing": true
        },
        "targetType": "avro",
        "fields": {
                "MESSAGE_ID": {
                "sourceName": "MESSAGE_ID",
                "targetType": "string",
                "defaultValue": "NO_MESSAGE_ID"
            },
            "MESSAGE_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "DAY_DT": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            },
            "AURA_NLP_IND": {
                "sourceName": "AURA_NLP_IND",
                "targetType": "boolean"
            },
            "CORR_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string",
                "defaultValue": "NO_CORR_ID"
            }
        }
    }

Use filters in Avro files

With this model, it is possible to filter records from the Avro files that should not be loaded into these files:

export interface AvroAdapterSource {
    data: AvroAdapterSourceDataType;
    id: any;
    avroSchema?: string;
    avroSchemaVersion?: string;
    csvFolder: string;
    entityName: string;
    filter?: (SourceFilter | LogicalFilter);

}
export interface SourceFilter {
    field: string;
    value: any;
    operator: FilterOperator;
}
export interface LogicalFilter {
    logic: LogicalOperator.and | LogicalOperator.or;
    filters: (SourceFilter | LogicalFilter)[];
}
export enum FilterOperator {
    equal = 'equal',
    notEqual = 'notEqual',
    greaterThan = 'greaterThan',
    lessThan = 'lessThan'
}
export enum LogicalOperator {
    and = 'and',
    or = 'or'
}

These filters could be configured in aura-avro-adapter.json in two ways:

With logical filter, with several conditions. For example:

    {
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "AUDIT",
            "csvFolder": "groot",
            "entityName": "GROOTMESSAGE",
            "filter": {
                "logic": "and",
                "filters": [
                    {
                        "field": "CHANNEL_ID",
                        "value": "",
                        "operator": "notEqual"
                    },
                    {
                        "field": "ACTION_CD",
                        "value": "receive",
                        "operator": "equal"
                    }
                ]
            }
        }

Or it could be configured in aura-avro-adapter.json with simple filter. For example:

{
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "avro",
            "id": "AUDIT",
            "avroSchema": "aura-gateway-message-asvc.json",
            "avroSchemaVersion": "6.0.0",
            "filter": {
                "field": "AURA_APP_ID",
                "value": null,
                "operator": "notEqual"
            }
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "AURA_CHANNEL_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "defaultValue": "nlpaas",
                "valueIfExist": "ai"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "AURA_VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "DAY_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    },

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura entities definition in CSV format

Aura entities definition in CSV format version 5.0.0.

Introduction

The Aura entities definition includes the different entities currently written in Aura in CSV and required to calculate Aura KPIs or by any other team in their data processes.

Entities types	Description	Generation
Message	Message entity is used to store information about the messages handled by aura-bot and the actions performed on them.	They are created in aura-bot.
Recognizer	Recognizer entity is used to store the request to any of the recognizers during the utterance recognition phase of the messages.	They are created by aura-groot, aura-bot and aura-nlp.
Extended Message	Extended Message entity is used to store extra information of a Message.	They are created in aura-bot.
Groot Message	Groot Message entity is used to store information about the messages handled by aura-groot.	They are created in aura-groot.

Entities generation

Until release 9.3.0 (Gwen Stefani), all entities in Aura were generated in CSV format. They were also uploaded into Kernel storage in CSV format. Although using a script provided by and running in Kernel, some of them are converted to Avro entities (in a deprecated Avro format, not URM compliant).

In particular, the following conversions are provided:

Recognizer:
- Aura Authenticated Users’ Recognizer
- Aura Anonymous Users’ Recognizer

As can be seen, the entities in Avro are duplicated to hold authenticated and anonymous users interactions separately, because the field USER_4P_ID cannot be null if it exists in a dataset definition.

CSV files format

All entities’ files must be created following these rules:

File format: UNIX. UTF8 without BOM
Date format: ISO8601
- Date: 2018-05-02
- Datetime: 2018-05-02T15:18:11Z => Always UTC

Usually, these kinds of files are stored in .txt and zipped, before uploading.

Entities used for the calculation of KPIs are stored in a Kernel bucket. Each entity must be included in a different folder, with files distributed by months, in the path:
[OB]/[ENTITY]/YYYYMM
Files generated in aura-bot can be stored as needed.
Filename: BOT_[HOST_ID]_[OB]_[ENTITY]_YYYYMMDDTHH0000Z.txt
Files generated in aura-services (authentication) can be stored as needed.
Filename: SERVICES_[HOST_ID]_[OB]_[ENTITY]_YYYYMMDDTHH0000Z.txt
Files generated in Aura NLP components can be stored as needed:
Filename: NLP_<HOST_ID>_<OB>_<ENTITY>_YYYYMMDDTHH0000Z.txt
Dimensions entities:
Path: [OB]/DIMENSIONS/YYYYMM
Filename: [OB]_DIM_[DIM_NAME]_YYYYMMDDTHH0000Z.txt
For example: ES_DIM_CHANNEL_20180612T160000Z.txt

All the files are refreshed every day.

CSV Entities tables nomenclature

#: Field ID
FIELD: Specific field of the entity type
PK: Parameter that indicates if the field is a Primary Key or not, that is, a value that uniquely identifies this field.
NULLABLE: Parameter that indicates if a field is allowed to have a null value or not.
TYPE: Type of the field. It can be one of: text, date, number, boolean
DESCRIPTION: Brief description of the field
FORMAT: Field mandatory format, if applicable
ALLOWED VALUES: Prefixed values permitted for this field
EXAMPLE: Example of application

The following considerations must be taken into account:

Numeric values are rounded to two decimal positions
The amount of money must be included in local currency

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura KPIs uploader

Aura KPIs uploader is the component in charge of the management of Aura entities and KPIs dimensions

Introduction

aura-kpis-uploader is the component responsible for handling entities and KPIs dimensions in Aura.

Currently, it manages both CSV entities and AVRO ones, that coexist in Aura.

aura-kpis-uploader-cli is an executable script that uploads KPIs in the storage location indicated in the destination config. It is a cron-job deployed in Aura’s cluster that is executed every hour (although it is configurable in the range from 1h to 24h). It is developed with Node 14.

Detailed information regarding aura-kpis-uploader-cli is found in the following documents:
. Architecture and main components
. How does Aura KPIs uploader work?
. How to use Aura KPIs uploader?
. Environment variables
. KPIs dimensions and Aura entities processing
. Troubleshooting

Aura KPIs uploader architecture

In the following diagram the architecture of aura-kpis-uploader-cli is represented, including its main components, which are described in the following sections.

Aura KPIs uploader components

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a channel configuration or environment variables.

Main process

Main process is referred to the process that a cron-job executes in Aura’s cluster.

KPI Upload

It contains the logic to perform the upload of KPIs dimensions and Aura entities in parallel mode to Kernel Azure Storage Account.

This process is in charge of converting Aura entity files to Avro format, so that they can be imported in Kernel DataLake and be consumed by third-party teams, such as Network Tokenization or BI of each OB. Aura DataSet Importer can only handle blobs of type Block Blob.

KPIs Processes

The processes are defined in a configuration file configured in AURA_SOURCE_PATH_AVRO_ADAPTERS and they are classified into:

KPIs Dimensions AVRO
Currently, KPIs dimensions are generated in Avro format
KPIs Entities
- KPIs Entities CSV
- KPIs Entities AVRO

Storage File Manager

This module is used to download and upload files from and to Azure Storage.

Aura KPIs uploader operation

The execution flowchart of aura-kpis-uploader-cli is shown in the following image:

aura-kpis-uploader-cli is responsible for copying the CSV files in Aura KPIs container (environment variables: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT) to the correspondent container in Kernel Azure Storage (environment variable: AURA_KPI_UPLOADER_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION).

Once the files are copied, the local copy is moved to a folder inside the container (environment variables: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT/AURA_KPI_UPLOADER_PROCESSED_FOLDER) and kept there during a fixed time, for recovering purposes.

The configured time by default is 7 days and is set in installer aurak8s in the template lifecycle.json.j2:

    {
      "name": "retention-kpis-processed-policy",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": [ "blockBlob" ],
          "prefixMatch": [ "{{ kpi_blob_container_name_processed }}" ]
        },
        "actions": {
          "baseBlob": {
            "delete": { "daysAfterModificationGreaterThan": {{ backup_retention_time | default(7) }} }
          }
        }
      }
    }

Independently of when it runs, aura-kpis-uploader-cli always performs the same process:

It gets all the files in KPIs container (environment variable: AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT).
Firstly, it verifies the number of files to process in order not to overload the memory. This is given by the environment variable AURA_KPI_NUM_MAX_FILES_FOR_EXECUTION. If the number of files exceeds this number, it will adapt the dates until it gets a range that fulfills the condition. The initial date is determined by AURA_DAYS_INTERVAL and the final date is calculated by subtracting the hours indicated in AURA_KPI_HOURS_TO_SUBTRACT_TO_GET_NUM_MAX_FILES.
When the number of files condition is fulfilled, all the aura-kpis-uploader processes will use that date range, except for the generation of the AUDIT file, since this file is generated at run time and must be moved to the entities folder if everything has proceeded successfully, with the current date.

It can also be launched manually to process old files that have not been processed in the regular execution, due to lack of communication with Kernel or any other error.

Furthermore, aura-kpis-uploader-cli generates for dimensions of entity type files: Channels, Skills, Presets, Applications, Components and Recognizers. You can see more information in KPIs dimensions.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura KPIs Uploader environment variables

List of environment variables handled by Aura KPIs uploader

Properties marked in bold are mandatory.
Properties marked in italics are optional.

Property	Type	Description	Modifiable by OB?
AURA_AUTHORIZATION_HEADER	string	Complete authorization header to be sent to `AURA_CHANNELS_CONFIGURATION_API_ENDPOINT`, with the following format: `APIKEY xxxxxx`	YES, but only if the previous APIKey was deprecated
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT	string	Complete URL where aura-bot should get the configuration of all the channels available in the environment.	YES, but only if the generated SAS token was deprecated.
AURA_DAYS_INTERVAL	number	Interval of days to get the processed KPIs files. By default: `30`	NO
AURA_DEFAULT_LOCALE	string	Culture code to be used by default in the current deployment: `de-de`, `en-gb`, `es-es`, `pt-br`.	NO
AURA_ENVIRONMENT_NAME	string	Name of the environment where aura-kpis-uploader is deployed. For example: `ap-next`, `es-dev`, `de-pre`	NO
AURA_FILES_PREFIX	string	Paths where aura-bot, aura-nlp or aura-authentication-api entity KPIs files are stored.	NO
AURA_FOLDER_DESTINATION	string	Root folder name in destination where the KPIs files are stored. By default: `AURA_DATA`.	NO
AURA_KPIS_ENTITIES_CONTAINER	string	Name of the Azure Blob container to store Aura entities files. By default, `aura-kpis`. It MUST be the same than the `AURA_KPIS_STORE_CONTAINER` configured in aura-bot, aura-authentication-api, aura-kpis and aura-nlp.	NO
AURA_KPI_FILES_EXTENSION	string	Extension to be used in KPIs files. Default: `txt`. It MUST be the same than the `AURA_KPI_TO_DSV_EXTENSION` configured in aura-bot, aura-authentication-api, aura-kpis and aura-nlp.	NO
AURA_KPI_FILES_NOT_TO_UPLOAD	string	String separated by comma with the files to not upload to the destination folder with format Component:Entity. For example: `BOT:MESSAGE,BOT:API`	NO
AURA_KPIS_TIME_FILE_IS_IN_USE_AMOUNT	number	Contains the amount of time it considers a file to be in use and does not process it. By default: `1`
AURA_KPIS_TIME_FILE_IS_IN_USE_UNIT	string	Time unit that together with AURA_KPIS_TIME_FILE_IS_IN_USE_AMOUNT determines whether or not a file should be processed: Values: “year”, “years”, “y”, “month”, “months”, “M”, “week”, “weeks”, “w”, “day”, “days”, “d”, “hour”, “hours”, “h”, “minute”, “minutes”, “m”. By default: `hour`
AURA_LOGGING_FORMAT	string	Format to be used in monitoring logs: `json` or `dev` (which is a more visual format). By default: `json`.	NO. Only for development, set it to `dev`.
AURA_LOGGING_LEVEL	string	Level to be used in monitoring logs, from more to less verbose: `'TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL', 'OFF'`. By default: `'INFO'`.	YES, for development environment, set it to debug. In pre/production environment, it should be ‘INFO’ or ‘ERROR’. For analysis of an issue in pre/production, it may be changed to ‘DEBUG’.
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY	string	Microsoft Storage password of the source deployment.	NO. Only if Operations Team changes it.
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT	string	Microsoft Storage account of the source environment.	NO. Only if Operations Team changes it.
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION	string	Microsoft Storage password of destination.	NO
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION	string	Microsoft Storage account of destination.	NO
AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION	string	Microsoft Storage container of destination.	NO
AURA_VERSION	string	Number of Aura release being executed.	NO
PUSHGATEWAY_ENDPOINT	string	Prometheus push gateway endpoint.	NO
PROMETHEUS_JOB_NAME	string	Prometheus job name. By default: `aura_kpi_uploader_job`.	NO
AURA_SOURCE_PATH_AVRO_ADAPTERS	string	Relative path to file with the dimensions and entities to transform Avro.	NO
AURA_KPIS_AVRO_DESTINATION_PATH	string	Path to store the Avro files in Aura Azure Container. Default: ‘avro’	YES
AURA_KPIS_REPORTS_DESTINATION_PATH	string	Path to store the reports in Aura Azure Container. Default: ‘reports’	YES
AURA_KPIS_REPORTS_MODE	string	Behavior of aura-kpis-uploader regarding the generation of reports. Possible values: `all`: a report is generated for each processed file; `none`: it will not generate any report; `error`: it generates a report if an error has occurred. By default: `error`.	NO
AURA_SOURCE_PATH_AVRO_ADAPTERS	string	Adapters to transform data, ‘/schemas/aura-csv-adapter.json’ for CSV transform and ‘/schemas/aura-avro-adapter.json’ to transform in CSV and Avro. By default: `schemas/aura-csv-adapter.json`.	YES
AURA_SAS_STORAGE_FILE_TTL	number	TTL in minutes for SAS when generates URL to File Storage. By default: `15`.	YES
AURA_KPIS_BLOB_TIME_WAIT_IF_EXISTS	number	Time to wait in milliseconds if the KPIs blob exists to avoid duplicate headers. By default: `1000`.	YES
AURA_KPIS_REPORTS_SAS_EXPIRATION	number	Time to expiration in minutes for the report SAS URL generated when an error occurs. Default: `43200` (30 days).	YES
AURA_DATABRICKS_OK_FILENAME	string	Name of the file to validate if DataBricks process was successful. Inside the file, there is a date to validate last success execution. Default: `databricks.OK`	YES
AURA_DATABRICKS_ERROR_FILENAME	string	Name of the file to validate if DataBricks process was failed. Default: `databricks.ERROR`	YES
AURA_DATABRICKS_EXECUTION_PERIOD	number	Period of DataBricks execution in hours. By default: `24`.	YES
AURA_KPIS_AVRO_SIZE_REPORT_FILENAME	string	Contains the name of the file to store size of AVRO files. By default: `sizeReport.json`.	YES
AURA_KPI_MAX_BLOCK_SIZE_FOR_APPEND_BLOB	number	Maximum buffer size for append blob uploading. Default: `100000000` (100MiB)	YES. It can be changed by a value lower than the maximum one established by default
AURA_KPI_NUM_MAX_FILES_FOR_EXECUTION	number	Number of files by entity to process in a single execution. Default: `12`	YES
AURA_KPI_HOURS_TO_SUBTRACT_TO_GET_NUM_MAX_FILES	number	Number of hours to subtract in order to get the number of files to process in a single execution. Default: `10`	YES

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura KPIs dimensions

Processing of Aura KPIs dimensions generated in aura-kpis-uploader

All of them will be generated in Avro format.

Introduction

aura-kpis-uploader-cli component generates the following types of KPIs dimensions:
. Channel type dimensions
. Recognizers type dimensions
. Skill type dimensions
. Components type dimensions
. Presets type dimensions
. Applications type dimensions

All of them are stored into the correspondent container in Aura Common Azure Storage (environment variable: AURA_KPIS_STORE_COMMON_CONTAINER).

Once the files are copied, the local copy is moved to a folder inside the container (environment variables: AURA_KPIS_STORE_COMMON_CONTAINER/ AURA_KPI_UPLOADER_PROCESSED_FOLDER) and kept there during a time, for recovering issues.

Types of KPIDimensionType:

export enum KPIDimensionTypes {
    KPIS_DIMENSIONS_CHANNEL = 'CHANNEL',
    KPIS_DIMENSIONS_RECOGNIZER = 'RECOGNIZER',
    KPIS_DIMENSIONS_SKILL = 'SKILL',
    KPIS_DIMENSIONS_APP = 'APP',
    KPIS_DIMENSIONS_PRESETS = 'PRESETS',
    KPIS_DIMENSIONS_COMPONENT = 'COMPONENT'
}

The path and filename should follow these patterns:

Path: AURA_KPIS_ENTITIES_CONTAINER/avro/dimensional/<avro_schema: name>/<avro_schema: x-fp-version>
Filename: <OB>_DIM_<DIM_NAME>_YYYYMMDDTHH0000Z.<AURA_KPI_FILES_EXTENSION>

The filename of the file is obtained as follows:

export function getDimensionFileName(KPIDimensionType: string) {
    return \`${getCountry().toUpperCase()}_\` +
        'DIM_' + \`${KPIDimensionType}_\` +
        \`${moment()
            .toISOString()
            .substring(0, 13)
            .replace(/-/g, '')
            .replace(/:/g, '') + '0000Z'}\` +
        \`${ConfigurationManager.instance.environmentConfiguration.AURA_KPI_FILES_EXTENSION}\`;
}

Channel type dimensions

aura-kpis-uploader-cli gets the list of channels configured in the environment through the aura-configuration-api channels endpoint.

It generates a file with the following schema:

Channel Schema

The Adapter format:

 {
        "name": "D_Aura_Channel",
        "schema": "dimensional",
        "avroSchema": "aura-channel-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "CHANNEL"
        },
        "targetType": "avro",
        "fields": {
            "AURA_CHANNEL_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_CHANNEL_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "AURA_CHANNEL_SHORT_NAME": {
                "sourceName": "prefix",
                "targetType": "string"
            },
            "AURA_SKILL_ID": {
                "sourceName": "skillId",
                "targetType": "string",
                "defaultValue": "NO_SKILL"
            },
            "AURA_NLP_STAGES_ARRAY": {
                "sourceName": "nlp.stages",
                "targetType": "array",
                "nullable": "true"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "GBL_CONTACT_CHANNEL_ID": {
                "sourceName": "contact",
                "targetType": "string"
            },
            "BRAND_ID": {
                "sourceName": "brand",
                "targetType": "string"
            }
        }
    }

Recognizers type dimensions

aura-kpis-uploader-cli gets the recognizers configuration through the file configured in the project: aura-kpis-uploader/src/dimensions/resource/recognizers.json

It generates a file with the following schema:

Recognizer Schema

The Adapter format:

  {
        "version": "1.0.0",
        "name": "D_Aura_Recognizer",
        "schema": "dimensional",
        "avroSchema": "aura-recognizer-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "RECOGNIZER"
        },
        "targetType": "avro",
        "fields": {
            "AURA_RECOGNIZER_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_RECOGNIZER_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Skill type dimensions

aura-kpis-uploader-cli gets the skill configuration through the aura-configuration-api Skills endpoint.

It generates a file with the following schema:

Skill Schema

The Adapter format:

{
        "version": "1.0.0",
        "name": "D_Aura_Skill",
        "schema": "dimensional",
        "avroSchema": "aura-skill-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "SKILL"
        },
        "targetType": "avro",
        "fields": {
            "AURA_SKILL_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_SKILL_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTERNAL_SKILL_IND": {
                "sourceName": "external",
                "targetType": "boolean"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Components type dimensions

aura-kpis-uploader-cli gets the components configuration through the aura-configuration-api Components endpoint.

It generates a file with the following schema:

Component Schema

The Adapter format:

{
        "version": "1.0.0",
        "name": "D_Aura_Component",
        "schema": "dimensional",
        "avroSchema": "aura-component-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "COMPONENT"
        },
        "targetType": "avro",
        "fields": {
            "AURA_COMPONENT_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_COMPONENT_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Presets type dimensions

aura-kpis-uploader-cli gets the presets configuration through the atria-model-gateway presets endpoint.

It generates a file with the following schema:

Preset Schema

The Adapter format:

    {
        "version": "1.0.0",
        "name": "D_Aura_Preset",
        "schema": "dimensional",
        "avroSchema": "aura-preset-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "PRESETS"
        },
        "targetType": "avro",
        "fields": {
            "AURA_PRESET_ID": {
                "sourceName": "preset.id",
                "targetType": "string"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "preset.name",
                "targetType": "string"
            },
            "AURA_MODEL_ID": {
                "sourceName": "model.id",
                "targetType": "string"
            },
            "AURA_PRESET_GROUP_NAME_CD": {
                "sourceName": "preset.group",
                "targetType": "enum",
                "symbols": [
                    "enriched_ai",
                    "simple_ai"
                ]
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Applications type dimensions

aura-kpis-uploader-cli gets the applications configuration through the aura-configuration-api Applications endpoint.

It generates a file with the following schema:

Application Schema

The Adapter format:

    {
        "version": "1.0.0",
        "name": "D_Aura_App",
        "schema": "dimensional",
        "avroSchema": "aura-app-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "APP"
        },
        "targetType": "avro",
        "fields": {
            "AURA_APP_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_APP_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "AURA_NLP_CHANNEL_ID": {
                "sourceName": "nlp.channeId",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_PRESET_NAMES_ARRAY": {
                "sourceName": "models.presets",
                "targetType": "array",
                "nullable": "true"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "BRAND_ID": {
                "sourceName": "brand",
                "targetType": "string"
            }
        }
    }

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Message

Within the different entities currently used in Aura, this document describes those included in the entity type Message

#	FIELD	PK	NULLABLE	TYPE	DESCRIPTION	FORMAT	ALLOWED VALUES	EXAMPLE
1	USER_ID	NO	NO	Text	Unique User ID in the OB Systems or unique generated one for anonymous users	Format depends on the OB	—
2	MSG_DT	NO	NO	Date	Timestamp of the data in UTC time	ISO 8601	—
3	MSG_ID	YES	NO	Text	Unique ID of the message	UUID	—
4	ACTION_CD	NO	NO	Text	Code of the action that produces the data	—	`send`, `receive`, `event`	—
5	AURA_ID	NO	YES	Text	User logging ID in Aura Starting in this version, the user will have a new `aura_id` each time she logs in Aura	UUID	—
6	PHONE_ID	NO	YES	Text	Phone number of the user	Phone number including international prefix	—
7	CHANNEL_CD	NO	YES	Text	Code of the channel where the action happened	—	Same values as `AURA_CHANNEL_NAME` field in the dimensional entity D_Aura_Channel
8	SUBSCRIPTION_CD	NO	YES	Text	Code of the subscription type of the user in the OB	—	`PREPAID`, `POSTPAID`, `CONTROL`	—
9	DOMAIN_CD	NO	YES	Text	Code of the domain where the action happened ⚠️ Future use	—	—
10	CATEGORY_CD	NO	YES	Text	Code of the category where the action happened ⚠️ Future use	—	—
11	COUNTRY_CD	NO	NO	Text	ISO code of the country	ISO 3166	—
12	CORR_ID	NO	NO	Text	Correlator ID of the request that produces the data	UUID	—
13	IS_CACHED	NO	NO	Boolean	It shows if the entity content was already cached or not	—	`true`/`false`	—
14	STATUS_CD	NO	YES	Text	Status code of the action, if meaningful	HTTP status code of the response	—
15	REASON	NO	YES	Text	Result of the action in error case: error code	—	—
16	VERSION_ID	NO	NO	Text	Aura version that produces this data	X.Y.Z	—	8.2.0.
17	LANG_CD	NO	YES	Text	Language configured by the user for the communication	- ISO 639 two-letter lowercase culture code - ISO 3166 two-letter lowercase subculture code associated with a country or region	—
18	TZ_CD	NO	NO	Text	Timezone where the communication happened	UTC offset of the TZ. TZ codes	—
19	DURATION_NU	NO	YES	Number	Duration of the action in milliseconds	Natural number	—
20	MESSAGE	NO	NO	Text	Content of the message	—	—
21	DIALOG_ID	NO	YES	Text	Id of the dialog where the message happens	—	—
22	CONVERSATION_ID	NO	NO	Text	Id of the conversation where the message happens	UUID	—
23	WIN_RECOGNIZER_CD	NO	YES	Text	Code of the recognizer that wins for this message		Same values as `AURA_RECOGNIZER_NAME` field in the dimensional entity D_Aura_Recognizer
24	WIN_RECOGNIZER_SCORE_NU	NO	YES	Number	Score of the recognizer that wins for this message	—	—
25	INTENT	NO	YES	Text	Selected intent	—	—
26	ENTITIES	NO	YES	Text	List of entities determined by the recognizer	—	—
27	MODALITY_CD	NO	YES	Text	It explains how the user communicates with Aura	—	`text`, `voice`, `form`	—
28	AURA_ID_GLOBAL	NO	YES	Text	It identifies the same `user_id` logged with the same authentication method	—	—
29	ACCOUNT_NUMBER	NO	YES	Text	Unique account number of the user Mandatory only for Spain. But it can be null also in Spain because it is only available in aura-bot, not in aura-services	—	—

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura entities processing

How Aura entities files are processed by aura-kpis-uploader and uploaded to Telefónica Kernel storage

Introduction

Currently, Aura entities processing is done both for CSV and Avro entities, that coexist in Aura.

These two processes are described in the following sections:

Aura entities processing

Aura entities processing for CSV files

Check Aura entities definition in CSV format here

aura-kpis-uploader-cli component aims to copy Aura’s entities files into Telefónica Kernel data storage.

It iterates all the folders where Aura servers store the entities files ( the aura-kpis container in the source Azure Storage account), copying them into the destination Azure Storage account (usually a Telefónica Kernel container).

Afterwards, the processed files are moved to the processed folder within the source container and, finally, the original files are removed.

sequenceDiagram
Title: Normal flow for a source component of Aura entities
    actor Cronjob
    participant uploadEntities
    participant processKPIS
    participant processBlobs
    participant processRemoteKPI
    participant StorageFileManager
    participant Azure Storage
    Cronjob ->> uploadEntities: Start process
    uploadEntities ->> processKPIS: Launch upload KPIS
    loop
      processKPIS ->> StorageFileManager: Get container client and blobs
      StorageFileManager ->> Azure Storage: 
      Azure Storage ->> StorageFileManager: OK
      StorageFileManager ->> processKPIS: OK
      processKPIS ->> processBlobs: Send all blobs to process
      processBlobs ->> processRemoteKPI: Filter and format blobs
      loop
        processRemoteKPI ->> StorageFileManager: Copy source blobs to destination container
        StorageFileManager ->> Azure Storage: 
        Azure Storage ->> StorageFileManager: OK
        StorageFileManager ->> processRemoteKPI: OK
        processRemoteKPI ->> StorageFileManager: Copy source blobs to source processed folder
        StorageFileManager ->> Azure Storage: 
        Azure Storage ->> StorageFileManager: OK
        StorageFileManager ->> processRemoteKPI: OK
        processRemoteKPI ->> StorageFileManager: Delete source blobs
        StorageFileManager ->> Azure Storage: 
        Azure Storage ->> StorageFileManager: OK
        StorageFileManager ->> processRemoteKPI: OK
      end
    end

The Adapter CSV entities definitions:

{
        "version": "1.0.0",
        "name": "E_Aura_BOT",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "BOT"
        },
        "targetType": "csv",
        "fields": {}
    },
    {
        "version": "1.0.0",
        "name": "E_Aura_CLF",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "CLF"
        },
        "targetType": "csv",
        "fields": {}
    },
    {
        "version": "1.0.0",
        "name": "E_Aura_GROOT",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "GROOT"
        },
        "targetType": "csv",
        "fields": {}
    },
    {
        "version": "1.0.0",
        "name": "E_Aura_NLP",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "NLP"
        },
        "targetType": "csv",
        "fields": {}
    }

Aura entities processing for AVRO files

Check Aura entities definition in Avro format here

There are two ways in which KPIs entities are created:

One is through a component, for example, aura-gateway-api generates its KPIs while it is running.
The other way is with the aura-kpis-uploader itself through the transformations of other KPIS already stored, for example AUDIT, which is generated through the aura-gateway-api count in AVRO format and through the aura-groot KPI, which is in CSV format.

When generating KPIS in AVRO for entities, it is not done in the default folder for each entity AURA_KPIS_ENTITIES_CONTAINER/avro/entity/<avro_schema: name>/<avro_schema: x-fp-version>, but a temporary folder named current is used: AURA_KPIS_ENTITIES_CONTAINER/avro/current/entity/<avro_schema: name>/<avro_schema: x-fp-version>. This is done to separate the files already generated from those currently being generated. When the KPIS Uploader is run, it moves the files that have already been generated to the default folder and converts them to BlockBlob.

aura-gateway-api Message Entity KPI

It generates a file with the following schema:

Aura Gateway API Schema Definition

First the CSV data must be transformed into avro format:

{
        "version":"1.0.1",
        "name": "Aura_Gateway_Message_converter",
        "stopWithErrors": true,
        "order": 1,
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "GATEWAY",
            "csvFolder": "gwapi",
            "entityName": "GATEWAYMESSAGE",
            "removeAfterProcessing": true
        },
        "targetType": "avro",
        "fields": {
                "MESSAGE_ID": {
                "sourceName": "MESSAGE_ID",
                "targetType": "string",
                "defaultValue": "NO_MESSAGE_ID"
            },
            "MESSAGE_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "DAY_DT": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            },
            "AURA_NLP_IND": {
                "sourceName": "AURA_NLP_IND",
                "targetType": "boolean"
            },
            "CORR_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string",
                "defaultValue": "NO_CORR_ID"
            }
        }
    }

The next adapter process the Gateway from current folder to entity destination folder.

{
        "version": "1.0.0",
        "name": "Aura_Gateway_Message",
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "GATEWAY"
        },
        "targetType": "avro",
        "fields": {}
    }

AUDIT Entity KPI

It generates a file with the following schema: Audit Schema Definition

The Adapters format is shown below:

From current folder to entity destination folder:

    {
        "version": "1.0.0",
        "name": "Aura_Audit",
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "AUDIT",
            "useDefaultTimeFilter": true
        },
        "targetType": "avro",
        "fields": {}
    }

From aura-gateway-api Message Entity KPI:

 {
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "avro",
            "id": "AUDIT",
            "avroSchema": "aura-gateway-message-asvc.json",
            "avroSchemaVersion": "6.0.0"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "AURA_CHANNEL_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "defaultValue": "nlpaas",
                "valueIfExist" : "ai"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "AURA_VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "DAY_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

From aura-groot Message Entity KPI:

{
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "AUDIT",
            "csvFolder": "groot",
            "entityName":"GROOTMESSAGE"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "CHANNEL_ID",
                "targetType": "string",
                  "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_SERVICE_NAME_CD",
                "targetType": "string",
                "defaultValue": "message"

            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Recognizer

Within the different entities currently used in Aura, this document describes those included in the entity type Recognizer

#	FIELD	PK	NULLABLE	TYPE	DESCRIPTION	FORMAT	ALLOWED VALUES	EXAMPLE
1	USER_ID	NO	NO	Text	Unique User ID in the OB Systems or unique generated one for anonymous users	Format depends on the OB	—
2	RECOGNIZER_DT	NO	NO	Date	Timestamp of the data in UTC time	ISO 8601	—
3	RECOGNIZER_ID	YES	NO	Text	Unique ID of the recognizer	UUID	—
4	ACTION_CD	NO	NO	Text	Code of the action that produces the data	—	`recognize`	—
5	AURA_ID	NO	YES	Text	User logging ID in Aura Starting in this version, the user will have a new `aura_id` each time she logs in Aura.	UUID	—
6	PHONE_ID	NO	YES	Text	Phone number of the user	Phone number including international prefix	—
7	CHANNEL_CD	NO	YES	Text	Code of the channel where the action happened	—	Same values as `AURA_CHANNEL_NAME` field in the dimensional entity D_Aura_Channel
8	DOMAIN_CD	NO	YES	Text	Code of the domain where the action happened ⚠️ Future use	—	—
9	CATEGORY_CD	NO	YES	Text	Code of the category where the action happened ⚠️ Future use	—	—
10	COUNTRY_CD	NO	NO	Text	ISO code of the country	ISO 3166	—
11	CORR_ID	NO	NO	Text	Correlator ID of the request that produces the data	UUID	—
12	IS_CACHED	NO	NO	Boolean	It shows if the entity content was already cached or not	—	`true`/`false`	—
13	STATUS_CD	NO	YES	Text	Status code of the action, if meaningful	HTTP status code of the response	—
14	REASON	NO	YES	Text	Result of the action in error case: error code	—	—
15	VERSION_ID	NO	NO	Text	Aura version that produces the data	X.Y.Z	—	8.2.0.
16	LANG_CD	NO	YES	Text	Language configured by the user for the communication	- ISO 639 two-letter lowercase culture code - ISO 3166 two-letter lowercase subculture code associated with a country or region	—
17	TZ_CD	NO	YES	Text	Timezone where the communication happened	UTC offset of the TZ. TZ codes	—
18	DURATION_NU	NO	NO	Number	Duration in milliseconds of the action	Natural number	—
19	SCORE_NU	NO	NO	Number	Score returned by the recognizer	Real number from 0 to 1	—
20	INPUT	NO	YES	Text	User input sent to the recognizer `null` if incoming message is an auraCommand	—	—
21	OUTPUT	NO	YES	Text	Complete output generated by the recognizer	—	—
22	INTENT	NO	YES	Text	Intent returned by the recognizer	Format depends on the OB	—
23	ENTITIES	NO	YES	Text	Entities returned by the recognizer due to the intent	—	—
24	COMMON_THRESHOLD_NU	NO	YES	Number	Common threshold used to determine the best answer of all recognizers	Real number from 0 to 1	—
25	THRESHOLD	NO	YES	Number	Established threshold for the recognizer	Real number from 0 to 1	—
26	EXPECTED_INTENT	NO	YES	Text	Intent expected to be returned by the recognizer	—	—
27	EXPECTED_ENTITIES	NO	YES	Text	Entities expected to be returned by the recognizer due to the intent	—	—
28	AURA_ID_GLOBAL	NO	YES	Text	It identifies the same `user_id` logged with the same authentication method	—	—
29	ACCOUNT_NUMBER	NO	YES	Text	Unique account number of the user Mandatory only for Spain. But it can be null also in Spain because it is only available in aura-bot, not in aura-services	—	—

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura KPIs uploader troubleshooting

Most common errors in aura-kpis-uploader together with the generated logs and recommendations for errors fixing

Required environment variables

Situation produced due to missing configuration of the mandatory environment variables.

If any of the mandatory environment variables is missing, an error message appears in the aura-kpis-uploader logs similar to the one shown below:

{"module":"Orchestrator","corr":"aura-system","error":"\"AURA_AUTHORIZATION_HEADER\" is required. 
\"AURA_CHANNELS_CONFIGURATION_API_ENDPOINT\" is required. 
\"AURA_DEFAULT_LOCALE\" is required. 
\"AURA_ENVIRONMENT_NAME\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY\" is required.
\"AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT\" is required.
\"AURA_VERSION\" is required: 
ValidationError: 
\"AURA_AUTHORIZATION_HEADER\" is required.
\"AURA_CHANNELS_CONFIGURATION_API_ENDPOINT\" is required. 
\"AURA_DEFAULT_LOCALE\" is required. 
\"AURA_ENVIRONMENT_NAME\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY\" is required. 
\"AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT\" is required. 
\"AURA_VERSION\" is required"
,"stck":{},"version":"not-reachable","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-17T21:31:49.850Z","lvl":"ERROR","msg":"Error initializing configuration manager"}

Errors in origin

The failure in the source Azure account will cause errors in the KPIs entities files loading process, but will not affect the loading of KPIs dimensions files.

Three types of errors can be generated:

Error in the Azure Blob container that stores KPIs entities files

The value of AURA_KPIS_ENTITIES_CONTAINER environment variable is not correct, as the container does not exist.

In the aura-kpis-uploader logs, an error message similar to this will appear:

{"module":"StorageFileManager","corr":"578543a2-73ce-430a-b949-af8a054dce85","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:36:58.165Z","lvl":"INFO","msg":"Try to connect to container-error"}
{"module":"StorageFileManager","error":"The specified container does not exist.\nRequestId:77c28402-101e-0039-63f6-e2cdd3000000\nTime:2022-10-18T13:36:58.1896965Z, stck: RestError: The specified container does not exist.\nRequestId:77c28402-101e-0039-63f6-e2cdd3000000\nTime:2022-10-18T13:36:58.1896965Z","corr":"578543a2-73ce-430a-b949-af8a054dce85","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:36:58.205Z","lvl":"ERROR","msg":"Container container-error doesn't exist."}
{"module":"AuraKpisUploaderStorageManager","error":"Container container-error doesn't exist.","stck":{},"corr":"578543a2-73ce-430a-b949-af8a054dce85","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:36:58.206Z","lvl":"ERROR","msg":"Error in getBlobsList: container-error/services"}
{"module":"AuraKpisUploaderUploadKPIS","containerName":"container-error","blobName":"services","corr":"578543a2-73ce-430a-b949-af8a054dce85","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:36:58.206Z","lvl":"INFO","msg":"No blobs found"}

Error in the source Microsoft Storage account

The value of AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT environment variable is not correct.

In the aura-kpis-uploader logs, an error message similar to this will appear, where aura-kpis is the default value for AURA_KPIS_ENTITIES_CONTAINER:

{"module":"StorageFileManager","error":"request to https://auraapnext4bbfcc3773error.blob.core.windows.net/aura-kpis?restype=container failed, reason: getaddrinfo ENOTFOUND auraapnext4bbfcc3773error.blob.core.windows.net, stck: RestError: request to https://auraapnext4bbfcc3773error.blob.core.windows.net/aura-kpis?restype=container failed, reason: getaddrinfo ENOTFOUND auraapnext4bbfcc3773error.blob.core.windows.net","corr":"91d31b7a-fe0b-44f9-8ce9-1da9e5705d6f","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:41:38.536Z","lvl":"ERROR","msg":"Error getting remote container: aura-kpis"}
{"module":"AuraKpisUploaderStorageManager","error":"Error getting remote container: aura-kpis","stck":{},"corr":"91d31b7a-fe0b-44f9-8ce9-1da9e5705d6f","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:41:38.536Z","lvl":"ERROR","msg":"Error in getContainerClient: aura-kpis"}

Error in the source Microsoft Storage password

The value of AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY environment variable is not correct.

In the aura-kpis-uploader logs, an error message similar to this will appear:

{"module":"StorageFileManager","error":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:b120fd76-a01e-0013-1cf7-e212c3000000\nTime:2022-10-18T13:43:27.8675929Z, stck: RestError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:b120fd76-a01e-0013-1cf7-e212c3000000\nTime:2022-10-18T13:43:27.8675929Z","corr":"700a4fab-6ec3-487b-adec-de558b08fd45","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:43:27.916Z","lvl":"ERROR","msg":"Error getting remote container: aura-kpis"}
{"module":"AuraKpisUploaderStorageManager","error":"Error getting remote container: aura-kpis","stck":{},"corr":"700a4fab-6ec3-487b-adec-de558b08fd45","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:43:27.917Z","lvl":"ERROR","msg":"Error in getContainerClient: aura-kpis"}

Errors in destination

This failure in the destination Azure account will cause the failure both in the loading of the KPIs files of entities and dimensions.

Error in the destination Azure Blob container

The value of AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION environment variable is not correct, as the container does not exist.

In the aura-kpis-uploader logs, an error message similar to this will appear, where aura-kpis-dest is the value of the environment variable AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION:

{"module":"StorageFileManager","error":"The specified container does not exist.\nRequestId:2da2f410-701e-005d-2bdb-e23c4b000000\nTime:2022-10-18T10:22:13.3171521Z, stck: RestError: The specified container does not exist.\nRequestId:2da2f410-701e-005d-2bdb-e23c4b000000\nTime:2022-10-18T10:22:13.3171521Z","corr":"fb07d24e-abb8-44bb-9a22-62907d571bf6","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T10:22:13.359Z","lvl":"ERROR","msg":"Container aura-kpis-dest doesn't exist."}

Error in the destination Microsoft Storage account

The value of AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION environment variable is not correct.

In the aura-kpis-uploader logs, an error message similar to this will appear:

{"module":"StorageFileManager","error":"request to https://auraapcurrent81dc0acde7.blob.core.windows.net/aura-kpis-dest?restype=container failed, reason: getaddrinfo ENOTFOUND auraapcurrent81dc0acde7.blob.core.windows.net, stck: RestError: request to https://auraapcurrent81dc0acde7.blob.core.windows.net/aura-kpis-dest?restype=container failed, reason: getaddrinfo ENOTFOUND auraapcurrent81dc0acde7.blob.core.windows.net","corr":"e29992b5-0421-43ad-9dde-c3a7f523c934","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T09:18:14.980Z","lvl":"ERROR","msg":"Error getting remote container: aura-kpis-dest"}

Error in destination Microsoft Storage password

The value of AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION environment variable is not correct.

In the aura-kpis-uploader logs, an error message similar to this will appear to upload entities files or to obtain dimensions files:

{"module":"StorageFileManager","error":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:4866760e-f01e-006c-7df8-e2dd58000000\nTime:2022-10-18T13:50:59.4208890Z, stck: RestError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:4866760e-f01e-006c-7df8-e2dd58000000\nTime:2022-10-18T13:50:59.4208890Z","corr":"e98b397e-c0a7-41e6-a8eb-d9a1ddc1bff6","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:50:59.432Z","lvl":"ERROR","msg":"Error getting remote container: aura-kpis-dest"}
{"module":"StorageFileManager","error":"Error getting remote container: aura-kpis-dest","stck":{},"corr":"e98b397e-c0a7-41e6-a8eb-d9a1ddc1bff6","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:50:59.433Z","lvl":"ERROR","msg":"Error copyFromBlobToBlob file from aura-kpis/services/SERVICES_1bc69ce0-4ebd-11ed-91cc-e3d43fca9661_CR_USER_20221018T130000Z.txt to aura-kpis-dest/AURA-DATA/CR/USER/202210/SERVICES_1bc69ce0-4ebd-11ed-91cc-e3d43fca9661_CR_USER_20221018T130000Z.txt"}
{"module":"AuraKpisUploaderStorageManager","error":"Error getting remote container: aura-kpis-dest","stck":{},"corr":"e98b397e-c0a7-41e6-a8eb-d9a1ddc1bff6","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T13:50:59.433Z","lvl":"ERROR","msg":"Error in copyRemote: aura-kpis/services/SERVICES_1bc69ce0-4ebd-11ed-91cc-e3d43fca9661_CR_USER_20221018T130000Z.txt"}

The URL stored in AURA_CHANNELS_CONFIGURATION_API_ENDPOINT environment is not correct or endpoint is not reachable.

In the aura-kpis-uploader logs, an error message similar to this will appear:

{"module":"Orchestrator","corr":"aura-system","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T09:13:53.486Z","lvl":"DEBUG","msg":"AuraChannelsConfiguration starting"}
{"module":"channels-configuration","error":"getaddrinfo ENOTFOUND aura-configuration-api.aura-ap-next.svc.cluster.local","stck":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"aura-configuration-api.aura-ap-next.svc.cluster.local"},"version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T09:13:58.507Z","lvl":"DEBUG","msg":"Get channels configuration failed."}
{"module":"Orchestrator","corr":"aura-system","error":"An error occurred while loading the channel information","stck":{},"version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T09:13:58.508Z","lvl":"DEBUG","msg":"Module AuraChannelsConfiguration has not started."}
{"module":"Orchestrator","corr":"aura-system","error":"An error occurred while loading the channel information","stck":{},"version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T09:13:58.509Z","lvl":"ERROR","msg":"Server cannot start"}
{"module":"Orchestrator","corr":"aura-system","version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-18T09:13:58.509Z","lvl":"DEBUG","msg":"Server closing process started"}

The value of AURA_AUTHORIZATION_HEADER environment variable with authorization header to be sent to AURA_CHANNELS_CONFIGURATION_API_ENDPOINT is not correct.

In the aura-kpis-uploader logs, an error message similar to this will appear:

{"module":"channels-configuration","error":"Unauthorized","stck":{"status":401,"response":{"req":{"method":"GET","url":"http://localhost:8999/aura-services/v2/configuration/channels?excludeFields=dialogLibraries%2Cmetadata&includeFields&legacyMode=false","headers":{"content-type":"application/json","accept":"application/json","authorization":"APIKEY {{your apikey}}","correlator":"aura-system"}},"header":{"content-security-policy":"default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests","x-dns-prefetch-control":"off","expect-ct":"max-age=0","x-frame-options":"SAMEORIGIN","strict-transport-security":"max-age=15552000; includeSubDomains","x-download-options":"noopen","x-content-type-options":"nosniff","x-permitted-cross-domain-policies":"none","referrer-policy":"no-referrer","x-xss-protection":"0","correlator":"aura-system","content-type":"application/json; charset=utf-8","content-length":"58","etag":"W/\"3a-LdGaUpp2yAiBlUgLhIWTUTcDhfM\"","date":"Wed, 19 Oct 2022 08:54:31 GMT","connection":"close"},"status":401,"text":"{\"code\":\"401\",\"message\":\"Given credentials are not valid\"}"}},"version":"7.4.0","app":"aura-kpis-uploader","host":"PC-516378","time":"2022-10-19T08:54:31.661Z","lvl":"DEBUG","msg":"Get channels configuration failed."}

Days intervals configuration

The value of AURA_DAYS_INTERVAL has an incorrect format.

In the aura-kpis-uploader logs, an error message similar to this will appear:

ERROR Error initializing configuration manager {
        module: 'Orchestrator',
        corr: 'aura-system',
        error: '"AURA_DAYS_INTERVAL" must be a number: ValidationError: "AURA_DAYS_INTERVAL" must be a number',
        stck: Error: "AURA_DAYS_INTERVAL" must be a number: ValidationError: "AURA_DAYS_INTERVAL" must be a number
            at Function.validateConfiguration (/home/cx02114/programacion/git/aura-kpis-uploader/lib/config/configuration-manager.js:104:19)
            at Function.init (/home/cx02114/programacion/git/aura-kpis-uploader/lib/config/configuration-manager.js:56:22)
            at Orchestrator.prepareConfigurationManager (/home/cx02114/programacion/git/aura-kpis-uploader/node_modules/@telefonica/aura-orchestrator/lib/orchestrator.js:33:49)
            at /home/cx02114/programacion/git/aura-kpis-uploader/lib/index.js:24:31
            at Object.<anonymous> (/home/cx02114/programacion/git/aura-kpis-uploader/lib/index.js:34:3)
            at Module._compile (internal/modules/cjs/loader.js:1085:14)
            at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
            at Module.load (internal/modules/cjs/loader.js:950:32)
            at Function.Module._load (internal/modules/cjs/loader.js:790:12)
            at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12),
        version: 'not-reachable',
        app: 'aura-kpis-uploader',
        host: 'PC-516378'
      }
ERROR Error in Orchestrator {
        module: 'StartService',
        error: '"AURA_DAYS_INTERVAL" must be a number: ValidationError: "AURA_DAYS_INTERVAL" must be a number',
        stck: Error: "AURA_DAYS_INTERVAL" must be a number: ValidationError: "AURA_DAYS_INTERVAL" must be a number
            at Function.validateConfiguration (/home/cx02114/programacion/git/aura-kpis-uploader/lib/config/configuration-manager.js:104:19)
            at Function.init (/home/cx02114/programacion/git/aura-kpis-uploader/lib/config/configuration-manager.js:56:22)
            at Orchestrator.prepareConfigurationManager (/home/cx02114/programacion/git/aura-kpis-uploader/node_modules/@telefonica/aura-orchestrator/lib/orchestrator.js:33:49)
            at /home/cx02114/programacion/git/aura-kpis-uploader/lib/index.js:24:31
            at Object.<anonymous> (/home/cx02114/programacion/git/aura-kpis-uploader/lib/index.js:34:3)
            at Module._compile (internal/modules/cjs/loader.js:1085:14)
            at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
            at Module.load (internal/modules/cjs/loader.js:950:32)
            at Function.Module._load (internal/modules/cjs/loader.js:790:12)
            at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12),
        corr: 'aura-system',
        version: 'not-reachable',
        app: 'aura-kpis-uploader',
        host: 'PC-516378'
      }

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Extended Message

Within the different entities currently used in Aura, this document describes those included in the entity type extended message

#	FIELD	PK	NULLABLE	TYPE	DESCRIPTION	FORMAT	ALLOWED VALUES	EXAMPLE
1	USER_ID	NO	NO	Text	Unique User ID in the OB Systems or unique generated one for anonymous users	Format depends on the OB	—
2	MSG_DT	NO	NO	Date	Timestamp of the data in UTC time	ISO 8601	—
3	MSG_ID	YES	NO	Text	Unique ID of the message	UUID	—
4	ACTION_CD	NO	NO	Text	Code of the action that produces data	—	`send`, `receive`, `event`	—
5	AURA_ID	NO	YES	Text	User logging ID in Aura Starting in this version, the user will have a new `aura_id` each time she logs in Aura	UUID	—
6	PHONE_ID	NO	YES	Text	Phone number of the user	Phone number including international prefix	—
7	CHANNEL_CD	NO	YES	Text	Code of the channel where the action happened	—	Same values as `AURA_CHANNEL_NAME` field in the dimensional entity D_Aura_Channel	—
8	SUBSCRIPTION_CD	NO	YES	Text	Code of the subscription type of the user in the OB	—	`PREPAID`, `POSTPAID`, `CONTROL`	—
9	DOMAIN_CD	NO	YES	Text	Code of the domain where the action happened ⚠️ Future use	—	—
10	CATEGORY_CD	NO	YES	Text	Code of the category where the action happened ⚠️ Future use	—	—
11	COUNTRY_CD	NO	NO	Text	ISO code of the country	ISO 3166	—
12	CORR_ID	NO	NO	Text	Correlator ID of the request that produces the data	UUID	—
13	IS_CACHED	NO	NO	Boolean	It shows if the entity content was already cached or not	—	`true`/`false`	—
14	STATUS_CD	NO	YES	Text	Status code of the action, if meaningful	HTTP status code of the response	—
15	REASON	NO	YES	Text	Result of the action in error case: error code	—	—
16	VERSION_ID	NO	NO	Text	Aura version that produces this data	X.Y.Z	—	8.2.0.
17	LANG_CD	NO	YES	Text	Language configured by the user for the communication	- ISO 639 two-letter lowercase culture code - ISO 3166 two-letter lowercase subculture code associated with a country or region	—
18	TZ_CD	NO	NO	Text	Timezone where the communication happened	UTC offset of the TZ. TZ codes	—
19	DURATION_NU	NO	YES	Number	Duration of the action in milliseconds	Natural number	—
20	MESSAGE	NO	NO	Text	Content of the message	—	—
21	DIALOG_ID	NO	YES	Text	Id of the dialog where the message happens	—	—
22	CONVERSATION_ID	NO	NO	Text	Id of the conversation where the message happens	UUID	—
23	WIN_RECOGNIZER_CD	NO	YES	Text	Code of the recognizer that wins for this message		Same values as `AURA_RECOGNIZER_NAME` field in the dimensional entity D_Aura_Recognizer
24	WIN_RECOGNIZER_SCORE_NU	NO	YES	Number	Score of the recognizer that wins for this message	—	—
25	INTENT	NO	YES	Text	Selected intent	—	—
26	ENTITIES	NO	YES	Text	List of entities determined by the recognizer	—	—
27	MODALITY_CD	NO	YES	Text	It explains how the user communicates with Aura	—	`text`, `voice`, `form`	—
28	AURA_ID_GLOBAL	NO	YES	Text	It identifies the same `user_id` logged with the same authentication method	—	—
29	ACCOUNT_NUMBER	NO	YES	Text	Unique account number of the user Mandatory only for Spain. But it can be null also in Spain because it is only available in aura-bot, not in aura-services	—	—

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura entities

Description of entities in Aura and components in charge of managing these entities

Aura entities belongs to both Aura Virtual Assistant and ATRIA

Introduction

Aura entities are files stored by different Aura components that contain relevant information related to key system processes or actions: user messages, Aura components interactions, message handling, applications, etc.

They are useful to measure and evaluate the performance of the system against defined objectives through the generation of KPIs, processes tracking, identification of issues and decision-making.

Each Aura component generates a series of entities which are uploaded into Azure Storage for different purposes.

aura-kpis-uploader is the component in charge of the management of entities for every Aura component. On an hourly basis, it uploads all the generated files to Kernel datasets. Once there, specific algorithms are executed to calculate the KPIs of each instance or for other purposes.

Detailed information regarding Aura entities is found in the following documents:

Types of logs in Aura: Operational logs and KPIs entities logs.
Description of Aura KPIs blob container.
Aura Databricks Jobs: Component that imports Avro-formatted files into Kernel datasets.
Aura KPIs uploaderComponent in charge of handling entities and dimensions.
KPI entity handler: aura-bot module in charge of managing Aura entities.
Status codes stored in KPIs entities: List of response codes stored in KPIs by aura-bot.
Aura entities definition: Database that includes the different entities currently used in Aura.

Types of logs in Aura

aura-bot writes two different types of logs:

Operational logs

Operational logs are written using AuraLogger that writes each row, by default, in JSON format in the standard output of the POD running each instance of aura-bot.

These logs are used to monitor or debug aura-bot. The standard output of each POD is aggregated to be stored in an ElasticSearch cluster, to make it available in Kibana.
Entities logs

An entity is a specific definition of one of the actors involved in the processing of each activity in aura-bot. So, the bot is in charge of writing the corresponding rows for each activity on each entity.

This section aims to describe how the entities are handled by aura-bot and how the rows are written and included in the entity files.

Logs are written in a blob file: a blob container in Azure Storage using an internal library aura-kpi-handler, that provides classes and utilities to decouple Aura components: how the rows are written from how the needed information is gathered.

The database Aura entities definition includes the different entities currently used in Aura. Entities are generated by different Aura’s modules, each of them in charge of performing a different task: aura-bot, aura-groot, aura-services, etc.

Currently, these logs are used to generate KPIs for the measurement of Aura performance, tracking processes or identifying issues.

Aura KPIs blob container

The following figures show the Aura KPIs blob container and an example of content of the blob container for aura-bot.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Groot message

Within the different entities currently used in Aura, this document describes those included in the entity type groot message

#	FIELD	PK	NULLABLE	TYPE	DESCRIPTION	FORMAT	ALLOWED VALUES	EXAMPLE
1	USER_ID	NO	NO	Text	Unique User ID in the OB Systems or unique generated one for anonymous users	Format depends on the OB	—
2	MSG_DT	NO	NO	Date	Timestamp of the data in UTC time	ISO 8601	—
3	MSG_ID	YES	NO	Text	Unique ID of the message	UUID	—
4	ACTION_CD	NO	NO	Text	Code of the action that produces data	—	`send`, `receive`, `event`	—
5	AURA_ID	NO	YES	Text	User logging ID in Aura Starting in this version, the user will have a new `aura_id` each time she logs in Aura	UUID	—
6	PHONE_ID	NO	YES	Text	Phone number of the user	Phone number including international prefix	—
7	CHANNEL_CD	NO	YES	Text	Code of the channel where the action happened	—	Same values as `AURA_CHANNEL_NAME` field in the dimensional entity D_Aura_Channel	—
8	DOMAIN_CD	NO	YES	Text	Code of the domain where the action happened ⚠️ Future Use	—	—
9	CATEGORY_CD	NO	YES	Text	Code of the category where the action happened ⚠️ Future Use	—	—
10	COUNTRY_CD	NO	NO	Text	ISO code of the country	ISO 3166	—
11	CORR_ID	NO	NO	Text	Correlator ID of the request that produces data	UUID	—
12	IS_CACHED	NO	NO	Boolean	It shows if the entity content was already cached or not	—	`true`/`false`	—
13	STATUS_CD	NO	YES	Text	Status code of the API request	HTTP status of the response	—
14	REASON	NO	YES	Text	Result of the action in error case: error code	—	—
15	VERSION_ID	NO	NO	Text	Aura version that produces data	X.Y.Z	—	8.2.0.
16	LANG_CD	NO	YES	Text	Language configured by the user for the communication	- ISO 639 two-letter lowercase culture code - ISO 3166 two-letter lowercase subculture code associated with a country or region	—
17	TZ_CD	NO	YES	Text	Timezone where the communication happened	UTC offset of the TZ. TZ codes	—
18	DURATION_NU	NO	NO	Number	Duration in milliseconds of the action	Natural number
19	MESSAGE	NO	NO	Text	Content of the message	—	—
20	CHANNEL_CONVERSATION_CD	NO	NO	Text	Identifier of the conversation in the channel (`conversationId` in aura-groot)	—	—
21	SKILL_CONVERSATION_CD	NO	NO	Text	Identifier of the conversation in the skill	—	—-
22	WIN_RECOGNIZER_CD	NO	YES	Text	Code of the recognizer that wins for this message	—	Same values as `AURA_RECOGNIZER_NAME` field in the dimensional entity D_Aura_Recognizer
23	WIN_RECOGNIZER_SCORE_NU	NO	YES	Number	Score of the recognizer that wins for this message	—	—
24	AURA_ID_GLOBAL	NO	YES	Text	It identifies the same `user_id` logged with the same authentication method	—	—
25	ACCOUNT_NUMBER	NO	YES	Text	Unique account number of the user Mandatory only for Spain, but it can be null also in this country because it is only available in aura-bot, not in aura-services	—	—
26	SKILL_CD	No	No	Text	Identifier of the skill	—	Same values as `AURA_SKILL_NAME` field in the dimensional entity D_Aura_Skill
27	AURA_COMPONENT_ID	No	No	Text	Identifier of the Aura component	—	Same values as `AURA_COMPONENT_NAME` field in the dimensional entity D_Aura_Component
28	AURA_COMPONENT_HOST_ID	No	No	Text	Identifier of the host running the specific component	—
29	BRAND_ID	No	No	Text	Identifier of the Telefonica’s brand	—	Same values as BRAND_ID field in D_GBL_BRAND

Aura – entity

Docs:

Aura entities definition

Introduction

Entities generation

Docs:

Aura entities definition in Avro format

Introduction

Entities generated in Avro URM-compliant format

Entities generation

Avro Entities tables nomenclature

Docs:

Aura_Atria_Message v6.0.0

Docs:

Aura_Audit v6.0.0

Docs:

Aura_Gateway_Message v6.0.0

Docs:

Aura_Gateway_Message v6.0.0

Docs:

Aura_Message v6.0.0

Docs:

Aura Dimensional entities

D_Aura_App schema definition v6.0.0

D_Aura_Channel v6.0.0

D_Aura_Component v6.0.0

D_Aura_Preset v6.0.0

D_Aura_Recognizer v6.0.0

D_Aura_Skill v6.0.0

Docs:

Aura KPIs uploader user guide

1. Prepare your CONFIG_FILE

2. Launch Aura KPIs uploader

2.1. Launch the tool directly with the source code

2.2. Launch the tool as a docker container

2.2.1. Pull image from repository

2.2.2. Run Docker container

3. Execution Dependencies

3.1 Evaluation of conditions for execution

4 Generate Reports

4.1 Report upload Files

4.1.1 Report Model

4.2 Report Generating Avro KPIs

4.2.1 Report Model

Docs:

Aura KPIs Adapter Manager

Definition

Adapter Definition File Model

AvroAdapterSchema

AvroAdapterSource

AvroAdapterSourceDataType

AvroAdapterField

AvroAdapterFieldModel

AvroPreCalculatedType

Avrofieldbasetype

AvroAdapterTargetType

Types of adapters

Copy CSV files

Generate Dimensional KPIs in Avro

Copy Avro files

Generate Avro from another Avro

Generate Avro from data in CSV file

Use filters in Avro files

Docs:

Aura entities definition in CSV format

Introduction

Entities generation

CSV files format

CSV Entities tables nomenclature

Docs:

Aura KPIs uploader

Introduction

Aura KPIs uploader architecture

Aura KPIs uploader components

ConfigurationManager

Main process

KPI Upload

KPIs Processes

Storage File Manager

Aura KPIs uploader operation

1. Prepare your `CONFIG_FILE`