Aura KPIs uploader user guide

1. Prepare your CONFIG_FILE

First, review and prepare your environment variables in order to create your CONFIG_FILE.

An example is shown below:

AURA_AUTHORIZATION_HEADER=${AURA_AUTHORIZATION_HEADER}
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT=${AURA_CHANNELS_CONFIGURATION_API_ENDPOINT}
AURA_DEFAULT_LOCALE=${AURA_DEFAULT_LOCALE}
AURA_ENVIRONMENT_NAME=${AURA_ENVIRONMENT_NAME}
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY}
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT}
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION}
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION}
AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION}
AURA_VERSION=${AURA_VERSION}

2. Launch Aura KPIs uploader

There are two methods for launching the aura-kpis-uploader

• Directly with the source code: to launch it in your local environment
• As a docker container: to use it with the already generated Docker image.

2.1. Launch the tool directly with the source code

$ cd aura-kpis-uploader
$ npm install
$ export CONFIG_FILE=[PATH_TO_MY_CONFIG_FILE]
$ npm run start

2.2. Launch the tool as a docker container

2.2.1. Pull image from repository

This method needs login first.

docker pull auraregistry.azurecr.io/aura/aura-kpis-uploader

2.2.2. Run Docker container

There are two options for running the Docker container:

• With .env file:

  Change $VERSION accordingly, exporting it or changing the string in the commands.

  If $VERSION is not present, it will use the latest default tag.

  docker run --rm -e CONFIG_FILE=/opt/aura-kpis-uploader/env/config_file.env \
          -v $(pwd):/opt/aura-kpis-uploader/env \
          auraregistry.azurecr.io/aura/aura-kpis-uploader:$VERSION
  

• With environmental variables in command:

  Note that host.docker.internal only works for MACOS/Windows systems:

  docker run -e AURA_AUTHORIZATION_HEADER=${AURA_AUTHORIZATION_HEADER} \
          -e AURA_CHANNELS_CONFIGURATION_API_ENDPOINT=${AURA_CHANNELS_CONFIGURATION_API_ENDPOINT} \
          -e AURA_DEFAULT_LOCALE=${AURA_DEFAULT_LOCALE} \
          -e AURA_ENVIRONMENT_NAME=${AURA_ENVIRONMENT_NAME} \
          -e AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY} \
          -e AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT=${AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT} \
          -e AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY_DESTINATION} \
          -e AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT_DESTINATION} \
          -e AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION=${AURA_MICROSOFT_AZURE_STORAGE_CONTAINER_DESTINATION} \
          -e AURA_VERSION=${AURA_VERSION} \
          -v ${HOST_DIRECTORY}:/opt/aura-kpis-uploader/logs \
          auraregistry.azurecr.io/aura/aura-kpis-uploader:$VERSION
  

3. Execution Dependencies

With the addition of the new component aura-databricks-jobs, now for Avro entities, the execution of aura-kpis-uploader depends on aura-databricks-jobs being executed correctly.

This is done to avoid duplicating KPIs when they are processed by aura-databricks-jobs, and an error has occurred.

3.1 Evaluation of conditions for execution

If avro-to-dataset-job-cli has been executed correctly, it will write a file to the default avro folder containing a date. If this file exists and the date it contains does not exceed the time set for the scheduled run in Databricks, aura-kpis-uploader-cli will run its processes.

If avro-to-dataset-job-cli has generated any error in its execution, it will generate a file in the same folder containing internally the error(s) that have occurred.

Environment variables involved:

• AURA_KPIS_AVRO_DESTINATION_PATH: Default value is avro.
• AURA_DATABRICKS_ERROR_FILENAME: Default value is databricks.ERROR.
• AURA_DATABRICKS_EXECUTION_PERIOD: Default value is 24 hours.

The flow that Aura KPIs Uploader follows to validate if it is going to be executed is as follows:

4 Generate Reports

By default, aura-kpis-uploader generates reports if any error has occurred in the upload process.

These reports are available in the Azure Storage defined in AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT, path aura-kpis/reports/.

If you want to change the behavior and generate reports of all uploaded files or disable their generation, you can do it by changing the environment variable AURA_KPIS_REPORTS_MODE. If the value is set to all, it will generate a report for each of the processed files and if it is set to none, it will not generate any report. The default value is error.

4.1 Report upload Files

When an error occurs, the system will send an event to Prometheus to alert about the error.

 this.prometheusHandler.addSummary(
                        { 
                        metricId: PrometheusMetrics.auraKpisUploaderReportErrorSummary, 
                        start: this.startTimer 
                        },
                        { 
                        reportError: reportLink },
                        this.corr
                        );

The event will contain the metricId auraKpisUploaderReportErrorSummary, when an error occurs and a link to the report containing the error.

4.1.1 Report Model

A report will contain the following template in JSON format:

• numberOfFiles: Number of files processed.
• numberOfFilesSkipped: Number of files that have been skipped because they have not yet been processed.
• NumberOfFilesMovedToProcessed: Number of files that have been moved to the processed folder.
• numberOfFilesUploaded : Number of files that have been successfully uploaded.
• NumberOfFilesDeleted: Number of files that have been deleted from the main folder.
• numberOfFilesUploadValidated: Number of files that have been verified as successfully uploaded.

If errors have occurred, it will contain an errors property with the files that have failed, arranged as properties:

• key: Name of the file that has failed.
• step: It indicates in which phase it has failed. The possible ones are “UPLOADING”, “CHECK_IF_UPLOADED”, “MOVING_TO_PROCESSED” AND “REMOVING”.
• error: Error message obtained.
• corr: Associated correlator.

Example:

{
 "numberOfFiles": 43,
 "numberOfFilesSkipped": 0,
 "numberOfFilesMovedToProcessed": 41,
 "numberOfFilesUploaded": 41,
 "numberOfFilesDeleted": 41,
 "numberOfFilesUploadValidated": 41,
 "errors": {
   "BOT_da5df187-8aa9-55b7-94d6-50586bd6b0b8_CR_API_20240216T080000Z.txt": {
     "step": "UPLOADING",
     "error": "getaddrinfo ENOTFOUND aurautils-test.blob.core.windows.net",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   },
   "BOT_b8efccfd-f2bd-5ef0-a19d-b2414ab94dc8_CR_API_20240215T160000Z.txt": {
     "step": "UPLOADING",
     "error": "getaddrinfo ENOTFOUND aurautils-test.blob.core.windows.net",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   }
 }
}

4.2 Report Generating Avro KPIs

When an error occurs, the system will send an event to Prometheus to alert about the error.

 this.prometheusHandler.addSummary(
                        { 
                        metricId: PrometheusMetrics.auraKpisGeneratedReportErrorSummary, 
                        start: this.startTimer 
                        },
                        { 
                        reportError: reportLink },
                        this.corr
                        );

The event will contain the metricId auraKpisGeneratedReportErrorSummary, when an error occurs and a link to the report containing the error.

4.2.1 Report Model

A report will contain the following template in JSON format.

• numberBlobsLoaded: Number of blobs loaded.
• numberDataRowsLoaded: Number of total lines loaded from blobs.
• numberDataRowsGenerated: Number of Avro registers generated.
• numberEntitiesGenerated: Number of entities generated.
• numberDimensionalGenerated: Number of dimensional files generated.

If errors have occurred, it will contain an errors property with the files that have failed, arranged as properties:

• key: The id of the source. Example: ‘AUDIT’.
• error: Error message obtained.
• corr: Associated correlator.

Example:

{
 "numberBlobsLoaded": 13,
 "numberDataRowsLoaded": 0,
 "numberDataRowsGenerated": 141,
 "numberEntitiesGenerated": 2,
 "numberDimensionalGenerated": 7,
 "errors": {
   "GATEWAY": {
     "error": "Error parsing data. Field name does not exist in source",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   },
   "AUDIT": {
     "error": "Error parsing data. Field id cannot be empty",
     "corr": "ec4978ee-0021-4633-a28f-a933759d0250"
   }
 }
}