Aura –

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura operational flowcharts

The current section includes certain key operational flows in Aura

Introduction

Find the following flowcharts for Aura operational processes:

Basic Direct Line flowcharts for anonymous and authenticated users
User Authentication flowcharts, that show how users’ authentication is handled depending on how the channel is integrated in Aura
Use cases flowcharts for basic use case execution flows
WhatsApp flowcharts for basic processes, Terms & Conditions management and authentication in this channel
Auraline flowcharts, containing the messages flow for this communication protocol

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Hot swapping in Aura

Description of Aura processes that can be executed without service outage

Introduction

The objective of the current documents is to describe the available actions that can be carried out over Aura through hot swapping processes, that is, without interrupting its operation. These actions include the update, insertion or removal of specific components or resources. The modifications resulting from these processes can be available immediately without the need to wait for the deployment of a subsequent Aura Platform release.

Despite the previous definition, performing a specific hot swapping process within a component can impact other components. In such cases, it becomes necessary to either restart the affected component or carry out a particular action to mitigate any potential issues or conflicts.

Available hot swapping processes and associated impact

The following table illustrates the available hot swapping processes against key Aura components, the impact of the process over each of them and the consequent actions that must be executed afterwards.

Each hot swapping process in the table is fully described in the document Hot swapping processes.

|——-|————-|———————-|————|———-|———-|————-|————|————————-|——————|———————|——————|———|————————| | 1. Update locales | Restart | Restart | Restart | aura-configuration-updater

Make-up | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | | 2. Update NLP packages | N/A | N/A | N/A | N/A | Restart | NO¹ | N/A | N/A | N/A | N/A | N/A | N/A | N/A | | 3. Modify TV use cases (User helper) | N/A | N/A | N/A | N/A | N/A | Restart | N/A | N/A | N/A | N/A | N/A | N/A | N/A | | 4. Channels handling | Restart | Restart | Reload | Make-up | Restart | Restart | Restart | NO² | Reload| Restart | Restart | N/A | | 5. Skills handling | N/A | N/A | Reload | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | | 6. Applications handling | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | Reload | N/A | N/A | N/A | N/A | | 7. Add new local dialogs in aura-bot | N/A | N/A | N/A | DeployLocal | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | | 8. Update libraries and dialogs configuration | N/A | N/A | N/A | aura-configuration-updater | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | | 9. Add static resources in a uc | N/A | N/A | N/A | aura-configuration-updater | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | | 10. Review indexes definitions | N/A | N/A | Make-up | Make-up | N/A | N/A | Make-up | Make-up | N/A | N/A | Make-up | Force | Make-up |

Legend

N/A: The change does not apply to the component
NO: The change does not apply to the component, unless a condition is met
Restart: the component will be restarted automatically by the config-watcher. The restart is executed pod by pod, with no impact to final users
Reload: the component is not restarted, but its configuration will be reloaded automatically by the config-watcher.
aura-configuration-updater: The hot swapping process must be executed using aura-configuration-updater.
Make-up: the make-up of the component is executed during the deploy phase: both with deploy-core and deploy-local. Take into account that aura-configuration-updater is included into the deploy_local process
DeployLocal: the component is automatically restarted after the execution of its make-up
Force: a restart of the component must be executed manually

Notes

Assuming it doesn’t involve resolution changes ↩︎
Until associated training is added, if applicable ↩︎

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura make-up processes

The current section describes the make-up processes that take place within Aura

Introduction

Make-up is a set of operations to prepare an environment once a module has been deployed, but before it starts. There is a different make-up job per module, that executes the operations needed before the module starts. The operations that a make-up process can perform are the following, although not all modules have to perform all these tasks.

The current document includes general definitions needed to understand this process.
The make-up specific processes for each module are specified in its corresponding document:
Learn how to manage the indexes of a MongoDB database in the make-up index manager

General definitions

Load External Libraries

This process is in charge of loading external modules, called libraries, which contain their own resources, locale, environment variables, code.

Manage MongoDB Indexes

Indexes in MongoDB are managed by a module called Make-up Index Manager documentation.

It relies on a versioned index configuration file to apply the indexes. Only those indexes whose version has not been applied before will be applied. The configuration file can be loaded from a remote repository. By default, all components with MongoDB access have one index configuration file.

There is a tool within each module called makeup-cli to run the make-up process as a stand-alone task.

Prepare Locale Resources

This process is responsible for collecting all the language resources and uploading them to a remote repository, so that they can be changed on the fly if necessary.

If external libraries have been loaded, this process unifies the language resources. See Locale Manager Documentation.

Set Environment Variables

This process is executed when the module loads external libraries that contain its own environment variables, unifying all these environment variables in a single file.

Prepare Remote Channel Config

It is responsible for unifying the different configurations assigned by each external library to the channels.

Process Resources

This process manages the different resources of each external library and promotes them so that they are accessible when the main program is running.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Manage locales in Aura

The aim of this document is both general and specific processes to describe how locales are handled by Aura Bot services

Introduction

Locales in Aura refer to specific content for an OB and a language, in which any special variant preference that an OB wants to see in its user interface is included.

Localization is an important aspect of internationalization and localization (i18n) for the adaptation of internationalized software for a specific region or language by translating text and adding locale-specific components.

The current section includes key documents to manage locales in Aura:

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Change the environment encryption key

Description of the process to change the encryption key without service loss

Introduction

This document describes how to change the encryption key of the Aura environments without service loss. The encryption key in Aura has three main usages:

Generation and validation of the service APIKeys.
Encryption of the secrets stored in the channels configuration settings.
Encryption of temporary params shared with other systems to validate an incoming request. For instance, when authenticating a user using the “Redirect authentication” of Kernel IdP, to be able to validate the incoming request with the user data in the authentication callback, via the state field, which is encrypted by Aura Bot and sent as query param to the authentication callback of Aura.

The mechanism proposed allows to validate APIKeys encrypted with several keys, whilst the servers use just the active encryption key for the rest of the functionalities.

Further information regarding how to configure the new encryption in the installer can be found in the deployment documentation.

Requirements

Docker and python3 installed in your local machine
- Also Nodejs20 if want to try the snippet to generate the encryption keys.
Access to the Aura installer
Access to the config.yml of your environment

Procedure

⚠️ It is important to do this change only during the installation of a new release of Aura, to avoid problems in the encryption of the channels collection and to let the installer generate automatically all the internal APIKeys.

Generate a 32 character encryption key. The length is mandatory because of the encryption algorithm being used. The following snippet contains an example of how to generate it using Nodejs:

const crypto = require('crypto');
const key = crypto.randomBytes(16).toString('hex');
console.log(key);

Encrypt it using ansible-vault:

ansible-vault encrypt_string 'old-encryptionkey,new-encryptionkey' --name 'aura_encryption_key'
New Vault password: 
Confirm New Vault password: 
Encryption successful
aura_encryption_key: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          62393736616536336262613333313866366365396361643930653535633564636430343530303637
          3866643931653739343164303662376439316630633538620a663031306265613162353633323565
          33376662626162663431616132353436366431383064663131643363653636346336343433303637
          3666333534656439390a656132306236336337383761323166666430633463303461316561363632
          65396530323635323363316362343437623065353232339343731643031653961336137343664366
          162386564636561313532633662383366383364653362663530653563623362303164653137653039
          663430356237366439663238346130656432303737

Configure it in the config.yml of your environment:

vim config.yml
...
aura_encryption_key: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          62393736616536336262613333313866366365396361643930653535633564636430343530303637
          3866643931653739343164303662376439316630633538620a663031306265613162353633323565
          33376662626162663431616132353436366431383064663131643363653636346336343433303637
          3666333534656439390a656132306236336337383761323166666430633463303461316561363632
          65396530323635323363316362343437623065353232339343731643031653961336137343664366
          162386564636561313532633662383366383364653362663530653563623362303164653137653039
          663430356237366439663238346130656432303737
...

Now, the installer can be launched. The default encryption key is going to be new-encryptionkey.
- The installer will generate all the internally used APIKeys, such as the one provided in the configuration of the services stored in the environment variable as AURA_AUTHORIZATION_HEADER.
Generate the new APIKeys for all the consumers of Aura’s APIs, using the tool of the installer to do it, following the instructions in generate APIKey section, the APIKeys are going to be created directly with the default one.
The APIKeys to be created depend on the environment, but take into account the following notes to update them all:
- The APIKey is used by all the channels accessing Aura via DirectLine, a new one per channel should be generated.
- The APIKey is used in the following webhooks or Aura APIs configured in Telefónica Kernel:
  - WhatsApp webhooks
  - Deployment of aura-services API
  - Deployment of aura-aiservices API
Once all the channels and Kernel configurations are changed to use the brand-new APIKeys, the environment configuration can be changed to count only with the default one.
To do the change from release to release, at least the encryption key of the latest deployment and the new one should be configured.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Generate an APIKey in Aura

Methods for the generation of an APIKey in Aura

Introduction

All Aura’s public APIs are protected using an APIKey.

The generation of an APIKey can be done using two alternative processes:

Generate an APIKey from Aura installer aurak8s
Follow the instructions in the document Create Aura APIKeys
Generate an APIKey using the aura-api-key-generator utility
- aura-api-key-generator is an Aura utility that generates encrypted APIKeys with a JSON model that will be used to validate the access to Aura’s public APIs.
- It is executed with the configuration of the chosen environment. This utility must be executed by the Operations Team and the generated APIKey must only be shared with the team involved in the integration of the corresponding API client.
- The guidelines are included in the following sections.

Generate an APIKey using the aura-api-key-generator utility

These guidelines correspond to the process that uses the aura-api-key-generator utility as a Docker image.

Prerequisites

Docker 23.0.5 or higher
Access to auraregistry.azurecr.io private docker registry.

The ENCRYPTION_KEY of the environment where the APIKey will be used:

kubectl installed in your local host.
curl installed in your local host.
jq installed in your local host.

To obtain the ENCRYPTION_KEY, just execute the following command:

# substitute {{aura-environment}} with the environment you're configuring
export AURA_ENVIRONMENT={{aura-environment}}

$ kubectl -n $AURA_ENVIRONMENT get secret aura-bot -o json | jq -r ".data.AURA_ENCRYPTION_KEY|@base64d"

YOUR-ENCRYPTION-KEY

Run the tool

Download the Docker image and run the tool by executing the following command:

# Substitute {{aura-encryption-key}} with the ENCRYPTION_KEY obtained in the previous step
$ docker run -e AURA_ENCRYPTION_KEY={{aura-encryption-key}} auraregistry.azurecr.io/aura/aura-api-key-generator:1.0.0 -v 8.3.0 -s aura-services -a kernel -m rw

Usage

> @telefonica/aura-api-key-generator@1.0.0 start
> node lib/index.js --help

process.env.CONFIG_FILE (undefined) not found or not configured, using only process.env

Usage: index -s aura-services:token -e pre -a kernel -v 8.3.0 -m rw

Utility to generate API Key to access the APIs of Aura.  Version: 1.0.0.

Options:
  -V, --version                     Output the version number
  -i, --identifier [identifier]     Unique identifier of the APIKey. Format UUID. If not provided, it will be autogenerated
  -s, --scope [scope]               Comma-separated list of strings containing the API path where this APIKey will be used. Possible values: "aura-services", "aura-services:users,aura-services:token", "aura-services:whatsapp", etc. (By default: "aura-services")
  -e, --enviroment <enviroment>     Environment where the APIKey will be used. Possible values: dev, stg, pre, pro (Choices: "dev", "stg", "pre", "pro". By default: "pro")
  -a, --authorised <client>         Name of the client that will use the APIKey. Usually one of kernel, novum, mh, mp, stb, la, etc. (By default: "kernel")
  -v, --aura-version <auraVersion>  Minimum Aura Platform version to use this APIKey. It should contain the full version of Aura. Example: 8.3.0 (default: "8.3.0")
  -m, --mode <mode>                 Type of access to the API. Possible values: r, w, rw (Choices: "r", "w", "rw". By default: "r")
  -h, --help                        Display help for command

Scope

All public Aura’s APIs are in the aura-services scope, so any APIKey generated with this scope will have access to all Aura’s APIs. But the idea is to use always the minimum scope valid for the specific usage of the client.

For example:

For those clients that are Aura channels and need to get a Direct Line token from Aura, the scope must be aura-services:token.
To configure the WhatsApp webhooks of aura-bridge in Kernel, the scope must be aura-services:whatsapp.
To configure the access from Kernel to aura-services API, the scope must be aura-services:users.
To configure the access from Kernel to aura-gateway-api in aura-services API, the scope must be aura-ai-services:messaging:write or aura-ai-services:nlp-messaging:write.

Environment

As the specific environment is configured via the ENCRYPTION_KEY, the environment parameter is only used to generate the APIKey with the correct environment name.

The environment name is used to validate the APIKey when it is used to access an API. It only contains the type of the environment: dev, stg, pre or pro.

Authorized

It should contain a human-readable name of the client that will use the APIKey. The usual values are: kernel, novum, mh, mp, stb, la, metaverse.

API Version

It contains the major version of the Aura API that the APIKey will be used.

For example, if the APIKey will be used to access the aura-services API, the API version must be 1. It must be coherent with the version of the API published in Aura’s API documentation.

Mode

It indicates the type of access to the API:

r: read-only access, it allows to access the API to execute GET operations.
w: write-only access, it allows to access the API to execute POST, PUT and DELETE operations.
rw: read-write access, it allows to access the API to execute GET, POST, PUT and DELETE operations.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Migrate Aura Groot context (Dire Straits)

Guidelines for the migration of aura-groot database context schema from releases previous to Dire Straits to Dorian release onwards. Additionally, the rollback procedure is included.

Prerequisites

Mandatory requisite: A valid kubeconfig for the environment must be available.
Recommendation: Create a snapshot/backup of the current DB data.

Process for context DB migration

Access MongoDB console

To access and open MongoDB console we can use the kubectl command below to export the intended environment, with MongoDB adminuser password. Use the right environment and database name in the commands.

$ export ATLAS_DEPLOYMENT_NAME=aura-br-pro-70 # Get from kubectl get -n aura-system atlasdeployments.atlas.mongodb.com
$ export ATLAS_DATABASE_NAME=aura-groot-br-pro # Set with the correct DB of your deployment
$ export MONGODB_USER=$(kubectl get -n aura-system atlasdatabaseusers.atlas.mongodb.com mongodb-adminuser-password -o jsonpath='{.spec.username}')
$ export MONGODB_PASSWORD=$(kubectl get -n aura-system secret mongodb-adminuser-password -o go-template='{{ .data.password | base64decode }}')
$ export MONGODB_URI="$(kubectl get -n aura-system atlasdeployments.atlas.mongodb.com $ATLAS_DEPLOYMENT_NAME -o jsonpath='{.status.connectionStrings.standardSrv}'|sed "s|srv://|srv://$MONGODB_USER:$MONGODB_PASSWORD@|g")/${ATLAS_DATABASE_NAME}"
$ kubectl -n aura-system run fix-script -i -q --rm --restart=Never --image=mongo:latest --command mongosh ${MONGODB_URI}

Migrate context collection to new schema

Update documents to the new format:

> db["aura-context"].aggregate([
      {
        $match: {
          _id: {
            $regex: "/conversations/",
          },
          "document.activeSkill.conversationId": {
            $exists: true,
          },
        },
      },
      {
        $lookup:
          {
            from: "aura-context",
            localField:
              "document.activeSkill.conversationId",
            foreignField: "_id",
            as: "document.skillConversationReference",
          },
      },
      {
        $set: {
          "document.skillConversationReference": {
            $arrayElemAt: [
              "$document.skillConversationReference",
              0,
            ],
          },
        },
      },
      {
        $set: {
          "document.skillConversationReference":
            "$document.skillConversationReference.document",
        },
      },
      {
        $merge: "aura-context",
      },
    ])

Check if changes have been applied. The following query should return n documents:

  > db["aura-context"].find({"document.skillConversationReference": { $exists: true }})

Rollback to the previous schema

If a rollback is needed, documents should be modified to have the previous schema and maintain the users’ context data:

> db["aura-context"].aggregate([
    {
      $match:
        {
          "document.activeSkill.conversationId": {
            $exists: true,
          },
          "document.skillConversationReference": {
            $exists: true,
          },
        },
    },
    {
      $project:
        {
          _id: "$document.activeSkill.conversationId",
          document:
            "$document.skillConversationReference",
          expiresAt: "$expiresAt",
        },
    },
    {
      $merge: "aura-context",
    },
  ])

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Purge expired Aura users

Guidelines for the elimination of expired Aura users, in order to minimize the number of documents in aura-users database. This will make data migration lighter.

Prerequirements

This step must be executed if the number of documents in the aura-users database is too large and there are a lot of expired users, i.e., users that accessed more than 6 months ago and did not come back to Aura for a time.
It must be executed on a weekly or monthly basis to avoid long executions. The frequency will depend on the amount of data in your database and how long it takes to remove the expired data.
A valid kubeconfig for the environment must be available
A valid token to be able to pull images from aura docker registry: auraregistry.azurecr.io must be available

Get the variables to access the database:

# substitute {{aura-environment with the environment you're configuring }}
export AURA_ENVIRONMENT={{aura-environment}}

$ kubectl -n $AURA_ENVIRONMENT get cm authentication-api -o json | jq -r ".data.AURA_MONGODB_URI"

{{mongo_uri}}

$ kubectl -n $AURA_ENVIRONMENT get cm authentication-api -o json | jq -r ".data.AURA_MONGODB_USERNAME"

{{mongo_user}}

$ kubectl -n $AURA_ENVIRONMENT get secret authentication-api -o json | jq -r ".data.AURA_MONGODB_PASSWORD|@base64d"

{{mongo_pass}}

$ kubectl -n $AURA_ENVIRONMENT get cm authentication-api -o json | jq -r ".data.AURA_MONGODB_USER_DB"

{{mongo_users_db}}

Process for expired users elimination

Access MongoDB console

This access depends on the deployment configuration:

If the environment database runs in an on-premise MongoDB server, set up the corresponding port-forward to the primary node of the replicaset:

# Review carefully the mongodb pod you're connecting to, it MUST be the primary of the replica set
# Review your local port carefully, to set a free one

$ kubeclt -n aura-system port-forward mongodb-0 27017:27017

# Set the proper mongo_uri to access via the port-forwarded connection
$ mongo_uri=mongodb://localhost:27017

If the environment database runs in Atlas, just connect to the {{mongo_uri}}, read from the environment configuration.

In the end, just open the MongoDB console:

$ mongo -u {{mongo_user}} -p {{mongo_pass}} {{mongo_uri}}

> use {{mongo_users_db}}

Check the number of expired users

Check how many users are expired in the database (Older than 180 days or without lastAccess field)

> db.users.find({ "$or": [ { "lastAccess": { "$exists": false }},{ "lastAccess": { "$lt": new Date(ISODate().getTime() - 1000 * 86400 * 180)} } ]}).count()

⚠️ If Movistar Plus is a valid channel in your deployment, then its users must not be considered in the query, because they are created from a backend and are assumed to be created beforehand.

> db.users.find({ "$and": [ { "$or": [ { "lastAccess": { "$exists": false }},{ "lastAccess": { "$lt": new Date(ISODate().getTime() - 1000 * 86400 * 180)} } ]}, { "channelId": { "$ne": "60f0ffda-e58a-4a96-aad9-d42be70b7b42" }} ]}).count()

Execute database-cleaner tool

The script iterates over all the users in the database, deleting those whose lastAccess is older than 180 days or without lastAccess field (meaning that they come from an old version of Aura where this field does not exist).

It will create some output files with the deleted users that can be used with mongoimport command to restore the deleted users, if needed. The output files with the deleted users will be split in files of 500Mb size each.

If the environment uses MongoDB on premise: Use with port-forwarding to the primary node of the cluster in the same host where the command is running:

# complete the {{mongodb_uri}} with the user and password of the database. You should obtain something like:
# mongodb_full_uri = mongodb://{{mongo_user}}:{{mongo_pass}}@{{mongodb_hosts_path}}

$ docker run -it --net=host -e MONGODB_USERS_DATABASE={{mongo_users_db}} -e MONGODB_URI={{mongodb_full_uri}} -v $(pwd)/output:/opt/cleaner/output auraregistry.azurecr.io/aura-tools/database-cleaner

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Continuous Integration

Find in this section key information related to Aura’s Continuous Integration system

Introduction

Aura uses Jenkins as CI orchestrator, mainly by the use of Jenkinsfiles with some internally developed libraries.

Our Jenkins runs in DCIP instance.

Access to these documents to find key information regarding:

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Event-watcher

Description of the event-watcher component

Event-watcher description

event-watcher is a deployment responsible for the automatic update of the Keyvault Atlas key when it is rotated and a new version is created in the secret handled by the Atlas operator to keep it updated.

The event-watcher is running in aura-system namespace. This component watches for the events published into a Microsoft Storage account queue and updates the key in the secret handled by the Atlas operator to keep it updated.

This new configuration will create a new Event Grid System Topic resource in the common resource group.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Use Config-watcher

Description and use of the config-watcher component

Config-watcher functionality

config-watcher is a deployment, continuously running in a persistent execution.

The objective of the config-watcher is to ensure that deployments are up to date with the most recent configuration adjustments.

To do this, this component gets the configuration from aura-configuration-api endpoints, configmaps or Azure Storage blobs and compares it with certain metadata added to the deployments/pods.

If the configuration is not synchronized, config-watcher will restart or call the endpoint of the pod to refresh its already loaded configuration.