Aura Analytics 1.1. architecture

Architecture description

The following figure shows a full overview of Aura Analytics Dashboard architecture and operation, which is also described below:

1.  Aura logs generated in local instance are converted to datasets and transferred to local Kernel via the standard procedure and with the established frequency (typically, daily).

2.  Once there, the “Active listening” process flow fires up daily. Through a specialized process that runs on an Aura local instance and with access to the stored datasets in the Kernel local storage space:

   • PII (Personally Identifiable Information) is removed or encrypted.
   • The result is transferred to a bucket/blob set up for this task and managed by Global Aura team.
   • Here, the PPDs (Privacy-Preserving Datasets) are created. Currently, only MESSAGE, RECOGNIZER and API datasets are involved in this process.

   In order to convert PII data to PPD, every field in these datasets can be:

   • a. Not transferred.
   • b. Pseudo-anonymized. In this situation, the field is transformed through a cryptographic hashing process using a secret set up by the OB.
   • c. Anonymized fragments of the field (e.g., credit card number, email, etc.). The field is processed to detect specific patterns and replaces them with a specific tag (idemail, idpassport, etc.). The list of anonymization strings is agreed with each OB.
   • d. Transferred as is.

3.  After that, the Raw PPD Datasets stored in bucket/blog managed by the Global Team are processed generating clean PPD Datasets in order to adapt them to the analytics tools.

4.  From that space, the clean PPD Datasets can be:

• Accessed by the Aura Global Team that use them for several tasks, with the purpose of evaluating Aura quality and taking the best decisions regarding to product evolution:

  • Perform analytics on Aura behavior and prototype Analytics Dashboard features
  • Improve Aura Platform capabilities (e.g., adapting machine learning models)

• Accessed by a Local Aura Team, ingesting the data to a dedicated server managed by the OB with analytics and data visualization capabilities. In order to do that, the Aura Global Team provides a component with the ELK (elasticsearch, logstash & kibana) preconfigured with a set of dashboards that can be deployed and adapted by the OB team.

All the code involved in this process can be found in Github. Particularly:

• PPD RAW creation package

• Conversion from PPD RAW to PPD Clean

• Pseudo-anonymization function for identifiers

• Utterance anonymization (agreed individually for ES and UK)