Tags:
Categories:

Aura KPIs Adapter Manager

Module to manage the copy of CSV files to Kernel and to generate Avro dimensional files and Avro Entity files.

⚠️ Although the definition of the module refers to Avro files, it also handles CSV files, until all CSV files are migrated to Avro in upcoming releases

Definition

Set of classes that will be in charge of processing the necessary operations to copy the KPIs in CSV format to Kernel and to process or generate the KPIs of the entities and dimensions in Avro format.

This is done from a definition file that, by default, is called aura-avro-adapter.json and is defined in the AURA_SOURCE_PATH_AVRO_ADAPTERS environment variable. This file is remotely loaded from the Storage AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT and the path AURA_KPIS_ENTITIES_CONTAINER/schemas/AURA_SOURCE_PATH_AVRO_ADAPTERS. If the file does not exist remotely, a local one is loaded by default.

Adapter Definition File Model

For each KPI file to be managed, a model is defined containing an array with all the adaptation operations to be performed. The content format of this file is JSON.

Name Type Definition
name string Name of the adapter. To manage Avro files, this field contains the name of the Avro schema itself
schema AvroAdapterSchema Type of KPI, entity or dimensional
avroSchema string If the destination file is an Avro file, this field contains the Avro schema to generate the file.
source AvroAdapterSource Contains the necessary data from where you are going to copy or transform the data, if applicable.
fields AvroAdapterField Contains a set of properties with a model defined in AvroAdapterFieldModel
targetType AvroAdapterTargetType Type file to generate. Values: csv or avro.
versionSchema string If the file to generate is Avro, this field contains the version of its schema
stopWithErrors boolean When true, if another adapter has generated an error, this one will not be executed
expiresAt date If set, it indicates which date the intended KPI will no longer be processed. This is used when there are different versions of AVRO templates and you want to keep both versions until the end of a cycle, which is usually monthly.
order number Contains the execution order, the order 1 will be the first to execution. By default the order is MAX_SAFE_INTEGER.

AvroAdapterSchema

Name type Definition
dimensional string Type of KPI is dimensional.
entity string Type of KPI is an entity.

AvroAdapterSource

Name type Definition
data AvroAdapterSourceDataType Type of file to use as source.
id any KPI identifier, for example: GATEWAY, AUDIT, BOT, etc.
avroSchema string Name of the schema file. Required if the source is an Avro file, as its schema needs to read it.
avroSchemaVersion string Version of the schema, required if the source is an Avro file.
csvFolder string Name of the folder where this CSV is stored, required if the source is a partial CSV.
entityName string Entity name of the CSV to load, required if the source is a partial CSV.
useDefaultTimeFilter boolean If true, the adapter will ignore the pre-calculated date range at the beginning of the process and will use the default one, which is calculated based on AURA_DAYS_INTERVAL as the start date and the end date by subtracting AURA_KPIS_TIME_FILE_IS_IN_USE_AMOUNT * AURA_KPIS_TIME_FILE_IS_IN_USE_UNIT from the current date. The default is 30 days prior to the current time.
version string Version of the adapter. It is used to propagate changes in a release and to update the remote file where the adapter models are stored in aura-avro-adapter.json. Format: x.y.z
removeAfterProcessing boolean Remove source data after processing. The data won’t store in processed data folder.

AvroAdapterSourceDataType

Name type Definition
avro string Name of the source file for Avro.
object string Name of the source file for the Javascript object.
csv string Name of the source file for CSV.
partialCsv string Name of the source file when it is referred to specific data from a CSV file.

AvroAdapterField

Name type Definition
any AvroAdapterFieldModel Name of the field.

AvroAdapterFieldModel

Name type Definition
sourceName string Name of the source field from which data is obtained.
nullable boolean It indicates whether the target field can be null or not.
preCalculated AvroPreCalculatedType String that indicates the type of calculation needed. For example, the required format for dates or numbers.
targetType Avrofieldbasetype Type of data for the target data.
defaultValue string number
symbols any[] Array with the values searched in the origin data. If the data does not match one of those values, it will generate an error.
format string Specific format to be sent, for precalculated fields.
description string Human-readable description of the field.
symbolsValues string If we want to change the value of the destination based on the values defined in the symbols field, we can define it in this field in the form of an object.
valueIfExist string If the source data exists, it will be changed to the one defined in this field.

AvroPreCalculatedType

Name type Definition
DATE_ISO_8691 string Format used for dates. The input date will be converted to this format. If there is no input date, the current date is obtained.
CUSTOM_DATE_FORMAT string Custom format for dates. Example: MM/DD/YYY
DURATION_ISO_8601 string Format used to set a duration.
COUNTRY_ISO_3166_ALPHA_3 string Format to indicate a country that complies with the ISO 3166 format.

Avrofieldbasetype

Name type Definition
string string Type string.
number string Type number.
enum string Type enumerable.
array string Type array.
boolean string Type boolean.

AvroAdapterTargetType

Name type Definition
csv string CSV file.
avro string Avro file.

Types of adapters

The Adapter manager can perform several types of processes, that will be explained in the following sections.

Copy CSV files

This adapter will copy the CSV files generated from the Aura servers (aura-bot, aura-groot, aura-nlp, etc.) to the Kernel storage.

{
        "version": "1.0.0",
        "name": "E_Aura_BOT",
        "schema": "entity",
        "avroSchema": "",
        "source": {
            "data": "csv",
            "id": "BOT"
        },
        "targetType": "csv",
        "fields": {}
    }

Generate Dimensional KPIs in Avro

This adapter will generate an Avro file with Aura available RECOGNIZER data, as they are defined only in aura-kpis-uploader and not in aura-configuration-api.

  {
        "version": "1.0.0",
        "name": "D_Aura_Recognizer",
        "schema": "dimensional",
        "avroSchema": "aura-recognizer-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "RECOGNIZER"
        },
        "targetType": "avro",
        "fields": {
            "AURA_RECOGNIZER_ID": {
                "sourceName": "id",
                "targetType": "string"
            },
            "AURA_RECOGNIZER_NAME": {
                "sourceName": "name",
                "targetType": "string"
            },
            "EXTRACTION_TM": {
                "sourceName": "EXTRACTION_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            }
        }
    }

Copy Avro files

This adapter copies the KPIs generated in Avro by aura-gateway-api to be processed by aura-databricks-jobs.

    {
        "version": "1.0.0",
        "name": "Aura_Gateway_Message",
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "object",
            "id": "GATEWAY"
        },
        "targetType": "avro",
        "fields": {}
    }

Generate Avro from another Avro

This is an adapter that generates an AURA_AUDIT file based on the data from the AVRO file of the aura-gateway-api KPIs.

{       "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "avro",
            "id": "AUDIT",
            "avroSchema": "aura-gateway-message-asvc.json",
            "avroSchemaVersion": "6.0.0"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "AURA_CHANNEL_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "defaultValue": "nlpaas",
                "valueIfExist" : "ai"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "AURA_VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "DAY_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

Generate Avro from data in CSV file

This is an adapter that generates an AURA_AUDIT file based on the data from the CSV file of the aura-groot Message KPIs.

{
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "order": 2,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "AUDIT",
            "csvFolder": "groot",
            "entityName":"GROOTMESSAGE"
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "CHANNEL_ID",
                "targetType": "string",
                  "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_SERVICE_NAME_CD",
                "targetType": "string",
                "defaultValue": "message"

            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "MSG_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    }

Another example, deleting CSV source file data when the adapter finalize its execution.

{
        "version":"1.0.1",
        "name": "Aura_Gateway_Message_converter",
        "stopWithErrors": true,
        "order": 1,
        "schema": "entity",
        "avroSchema": "aura-gateway-message-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "GATEWAY",
            "csvFolder": "gwapi",
            "entityName": "GATEWAYMESSAGE",
            "removeAfterProcessing": true
        },
        "targetType": "avro",
        "fields": {
                "MESSAGE_ID": {
                "sourceName": "MESSAGE_ID",
                "targetType": "string",
                "defaultValue": "NO_MESSAGE_ID"
            },
            "MESSAGE_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string",
                "preCalculated": "COUNTRY_ISO_3166_ALPHA_3"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "DAY_DT": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            },
            "AURA_NLP_IND": {
                "sourceName": "AURA_NLP_IND",
                "targetType": "boolean"
            },
            "CORR_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string",
                "defaultValue": "NO_CORR_ID"
            }
        }
    }

Use filters in Avro files

With this model, it is possible to filter records from the Avro files that should not be loaded into these files:

export interface AvroAdapterSource {
    data: AvroAdapterSourceDataType;
    id: any;
    avroSchema?: string;
    avroSchemaVersion?: string;
    csvFolder: string;
    entityName: string;
    filter?: (SourceFilter | LogicalFilter);

}
export interface SourceFilter {
    field: string;
    value: any;
    operator: FilterOperator;
}
export interface LogicalFilter {
    logic: LogicalOperator.and | LogicalOperator.or;
    filters: (SourceFilter | LogicalFilter)[];
}
export enum FilterOperator {
    equal = 'equal',
    notEqual = 'notEqual',
    greaterThan = 'greaterThan',
    lessThan = 'lessThan'
}
export enum LogicalOperator {
    and = 'and',
    or = 'or'
}

These filters could be configured in aura-avro-adapter.json in two ways:

  • With logical filter, with several conditions. For example:

        {
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "partialCsv",
            "id": "AUDIT",
            "csvFolder": "groot",
            "entityName": "GROOTMESSAGE",
            "filter": {
                "logic": "and",
                "filters": [
                    {
                        "field": "CHANNEL_ID",
                        "value": "",
                        "operator": "notEqual"
                    },
                    {
                        "field": "ACTION_CD",
                        "value": "receive",
                        "operator": "equal"
                    }
                ]
            }
        }

  • Or it could be configured in aura-avro-adapter.json with simple filter. For example:

    {
        "version": "1.0.0",
        "name": "Aura_Audit",
        "stopWithErrors": true,
        "schema": "entity",
        "avroSchema": "aura-audit-asvc.json",
        "versionSchema": "6.0.0",
        "source": {
            "data": "avro",
            "id": "AUDIT",
            "avroSchema": "aura-gateway-message-asvc.json",
            "avroSchemaVersion": "6.0.0",
            "filter": {
                "field": "AURA_APP_ID",
                "value": null,
                "operator": "notEqual"
            }
        },
        "targetType": "avro",
        "fields": {
            "INTERACTION_ID": {
                "sourceName": "CORR_ID",
                "targetType": "string"
            },
            "AURA_TM": {
                "sourceName": "MESSAGE_TM",
                "targetType": "string",
                "preCalculated": "DATE_ISO_8691"
            },
            "AURA_CHANNEL_ID": {
                "sourceName": "AURA_CHANNEL_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_APP_ID": {
                "sourceName": "AURA_APP_ID",
                "targetType": "string",
                "nullable": "true"
            },
            "COUNTRY_3_ALPHA_CD": {
                "sourceName": "COUNTRY_3_ALPHA_CD",
                "targetType": "string"
            },
            "AURA_SERVICE_NAME_CD": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "defaultValue": "nlpaas",
                "valueIfExist": "ai"
            },
            "AURA_PRESET_NAME": {
                "sourceName": "AURA_PRESET_NAME",
                "targetType": "string",
                "nullable": "true"
            },
            "AURA_COMPONENT_ID": {
                "sourceName": "AURA_COMPONENT_ID",
                "targetType": "string"
            },
            "AURA_COMPONENT_HOST_ID": {
                "sourceName": "AURA_COMPONENT_HOST_ID",
                "targetType": "string"
            },
            "AURA_VERSION_ID": {
                "sourceName": "AURA_VERSION_ID",
                "targetType": "string"
            },
            "DAY_DT": {
                "sourceName": "DAY_DT",
                "targetType": "string",
                "preCalculated": "CUSTOM_DATE_FORMAT",
                "format": "YYYY-MM-DD"
            },
            "BRAND_ID": {
                "sourceName": "BRAND_ID",
                "targetType": "string"
            }
        }
    },
Last modified November 11, 2025: feat: Clean up of Living Apps related stuff #AURA-30761 [RTM] (c97ca748)