Aura – Catalog of Technical Components

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Core technologies in Aura

DRAFT DOCUMENT

This document summarizes the core technologies used in both Aura Virtual Assistant and ATRIA, that support their different functionalities and services

Other technologies used for specific issues are referenced in the corresponding documentation

Aura deployment

Aura is deployed in Microsoft Azure
Components packaging is done using Docker
Deployment is hosted on Azure Kubernetes
Deployment is orchestrated using Ansible

Security and privacy

Aura ensures compliance with GDPR regulations
Aura installer relies in Python security practices
The use of ARM templates enhance security of our Azure resources

Operational foundations

Aura Virtual Assistant is based on Microsoft Bot skills architecture
Aura Bot is built over Microsoft BotFramework

Data management

We use MongoDB and Redis as service databases
Databricks handles the optimization of data processing

Languages

Aura backend is developed using Typescript and Python
iOS development is based on Swift
Certain scripts are made with Bash
ATRIA aura-manager web interface is built with React and Typescript

Monitoring tools

Aura metrics are generated in Prometheus
Aura monitoring stack includes Elasticsearch, Grafana and Kibana

Testing tools

Functional tests implemented in Python using:
Toolium
Behave
Selenium
Appium

AI-driven technologies

Azure OpenAI: Embeddings, GPT models
Hugging Face models
Microsoft Conversational Language Understanding (CLU)

Releases management

We apply the Release Train Manager methodology
Github is used as the orchestrator for the RTM
All our CI processes run in Jenkins

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Bot as a skill

aura-bot is Aura’s neuronal network in charge of core features in Aura Virtual Assistant. Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

aura-bot is an HTTP server, based on Microsoft Bot Framework. It is built in Typescript using the node.js SDK of Bot Framework. As an HTTP server, it uses Express.js.

aura-bot is Aura’s neuronal network, where it can be considered as a Bot Development Platform itself. In the recent distributed architecture, aura-bot is the only skill on the system that handles all requests, although the new entry point to the system is now the root bot called aura-groot, that redirects all requests to the only skill in order to access Aura through diverse communication channels and interacts with every Aura system or component in order to provide Aura users with the intended answer or requested service.

The following flowchart shows the complete conversational process executed by aura-bot.

Aura Bot functionalities

The main functionalities in charge of aura-bot are users’ sign up, authentication validation process and conversational process.

Apart from them, there are other processes managed by aura-bot that can be executed in order to customize the bot, activate other components, etc.

Find here all the information related to Aura Bot processes.

Aura Bot components

aura-bot components can be classified into different categories:

Aura Bot Platform components, which are components aura-bot has as a BotFramework bot: dialogs, middlewares, recognizers, Bot framework adapter, Activity Handler, etc.
Other components that have been developed within aura-bot to provide it with a specific functionality, such as plugin manager or routing manager.
Moreover, the databases used by aura-bot.

Find detailed information of all of them in Aura Bot components.

Aura Bot Platform repository

The repository aura-bot-platform has a folder layout inside the src folder, to organize the code according to functionality groups.

In the src root folder, only main files should be placed, such as index.ts, or main server file server.ts (it was app.ts in aura-bot).

src/bots

At the beginning, only the file aura-bot.ts is hold here, that is the main ActivityHandler (a kind of replace of previous UniversalBot).

In a near future, more Bot Builder adapters could be located here.

src/config

This folder contains files that load configurations and that are required in other places, such as channel configuration, environment variables, etc.

src/db

This folder contains files that manage and connect to databases, such as MongoDB.

src/events

The files handling events (such as ModuleObserver subclasses) should be placed in this folder.

src/dialogs

This folder contains the main dialog (main.ts) and custom prompts, as use case dialogs will be located in libraries, loaded as dependencies.

src/make

This folder contains the code related with joining all the library-specific data with global one during the make-up process.

src/middlewares

This folder contains abstract middleware base classes, and specific middleware implementation (final classes).

src/middlewares/recognizers

This folder contains all the recognizers, such as NLP recognizer, Aura Command recognizer, etc.

src/models

Files located within this folder will have exported types and interfaces required in different parts of the code (types and interfaces required only within a file could be self contained).

src/modules

This folder contains independent code blocks, that could be exported as a reusable packages if required in different components, such as cache manager, locale manager, etc.

src/plugin

This folder contains the modules in charge of loading plugins: charging dialogs, middlewares, delivery configuration for these components, etc.

src/routing

This folder contains the code related with intent-to-dialog routing, as this is not part of Bot Builder anymore.

src/utils

This folder contains utility classes and methods that are not part or any other block. We should maintain this folder organized and tidy, to avoid lots of unspecific files.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Groot

aura-groot is the router bot that manages the behavior of Aura Root ecosystem functionalities acting as a router for the request to a specific skill.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

aura-groot is the main component in charge of handling the distributed architecture of Aura. It is responsible for the communication between each channel and its corresponding skill, keeping track of all the communication process, but introducing a minimal interference on it.

aura-groot main role includes:

It is the component in charge of making the decision about what is the best bot to answer the user’s request and routing the request to the corresponding skill (bot).
It is able to map configurable input parameters such as the channel that sent the request, user type, previous requests stored in the context, utterance, etc. with a specific domain. The OB will be able to configure the parameters used for the routing process.

Currently, routing is done only by channels or group of channels (no configuration options are available).

Aura Groot components

aura-groot contains two main components:

Middlewares: software components in charge of the message flow.
Recognizers: specific type of middlewares that are executed in a certain stage of the message flow.

Access to the corresponding sections to find detailed information regarding these components.

Aura Groot architecture

The first diagram shows all the components running in aura-groot.

The following diagram includes the messages flow through aura-groot.

Aura Groot configuration and operation

Find out the environment variables included in the aura-groot component here.

The operation of aura-groot is described in the document Aura Groot operational flows.

ConversationId handling

aura-groot uses groot-channel conversation identifier to track the conversation with the user and the use case with the skill. It is a string generated by the channel and is unique for each conversation. The groot-channel is sent in the incoming request to aura-groot in the conversationId field of the message object.

Additionally, it adds conversationReference inside. In this way, aura-groot only has to manage a single conversation and avoids synchronizing two different conversations. Doing so, accesses from aura-groot to the database are reduced.

This change implies a modification in the process the BotFramework works with a conversation, in two different documents which are updated separately. But, on the contrary, the advantage is a most efficient and secure way of working.

sequenceDiagram
title: BEFORE-DIRE-STRAITS

    actor Channel
    participant AuraGroot #ebdff7
    participant AuraBot #ebdff7

    Channel ->> AuraGroot: request message1
    AuraGroot ->> AuraGroot: create targetSkill in groot-channel
    AuraGroot ->> AuraGroot: save conversationReference in groot-channel
    AuraGroot ->> AuraBot: request message1
    AuraBot -->> AuraGroot: response message1
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot -->> Channel: response message1
    AuraBot -->> AuraGroot: response endOfConversation
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot ->> AuraGroot: delete targetSkill

    Channel ->> AuraGroot: request message2
    AuraGroot ->> AuraGroot: create targetSkill in groot-channel
    AuraGroot ->> AuraGroot: save conversationReference in groot-channel
    AuraGroot ->> AuraBot: request message2
    AuraBot -->> AuraGroot: response message2
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot -->> Channel: response message2
    AuraBot -->> AuraGroot: response endOfConversation
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot ->> AuraGroot: delete targetSkill

Aura Groot endpoints

Path	Method	Description
/api/messages/	POST	Incoming requests of channels or aura-bridge
/refresh/	POST	Update skills and channels. Data: `{"changes": ["skills", "channels"]}`. You can use “skills” and/or “channels” or neither of them. In the last case, all will be updated.
/api/skills/*	POST	Endpoints of Direct Line API for skills can exchange messages with aura-groot. The path where the skills will return the answers is configured in the environment variable `AURA_SKILLS_RESPONSE_PATH`, by default, `/api/skills`.
/healthz	GET	Monitoring endpoint. Healthcheck endpoint to validate if the server is up and running, only available from local network
/shutdown	GET	Monitoring endpoint. Wait until shutdown (Kubernetes lifecycle).
/heapSnapshot	GET	Endpoint that must be called internally directly accessing to the aura-groot pod. It stores in Aura Azure Storage Account a copy of the heap memory of the pod. It is useful to debug memory leaks.
/heapStatistics	GET	Endpoint that must be called internally directly accessing to the aura-groot pod. It returns the heap memory statistics of the pod.

These endpoints do not need to be defined in the plugin swagger, since they are supplied by the aura-groot core. Specific routes are defined in https://github.com/Telefonica/aura-distributed-bot/blob/main/master/packages/aura-groot/src/modules/server/routes.ts.

Postman collection

Aura Platform Team has generated a Postman collection for simulations over Aura distributed architecture: Aura distributed Postman collection.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Bridge

aura-bridge is the component in charge of the communication protocol between aura-groot and certain external channels that cannot implement the Direct Line protocol.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction to Aura Bridge

aura-bridge is the component of Aura in charge of adapting the communication protocol, both message format and API calls, between aura-groot and certain external channels, such as Whatsapp, that cannot implement Direct Line protocol (aura-groot native language). It is designed to be extensible, i.e., if any other external channel needs to be integrated with aura-groot, a new adapter for this channel might be added easily to the bridge.

aura-bridge is built as a set of configurable plugins that are deployed on top of aura-bridge core. Each plugin consists of an OpenAPI definition, a controller, a request handler and a converter.

Regarding its implementation, the following technologies are used:

Programming language: Typescript 5
Nodejs engine: nodejs
Message caches: Redis
Server: expressjs
API definition: It is api-first designed, using OpenAPI v3 to provide the API definition.

aura-bridge is divided into two sub-components from the logical point of view, where aura-bridge-outbound is a component which shares the same logical implementation as aura-bridge, with the main distinction lying in the differences in its configuration, which implies a deployment specialization:

Component in charge of adapting the communication from the channel: aura-bridge
Channel communication protocol ➡️ Direct Line protocol
Component in charge of adapting the communication to the channel: aura-bridge-outbound
Direct Line protocol ➡️ Channel communication protocol

aura-bridge documentation is included in different sections, depending on its scope:

aura-bridge descriptive documentation and internal processes:
Practical processes that OBs can carry out over aura-bridge:
- Development of a plugin and activation in Aura bridge
- aura-bridge Postman collection for making simulations in aura-bridge

Communication protocol

aura-groot communication protocol is completely asynchronous, this means that the answer of a request is not included in the HTTP response related to the request, but it is sent as a new HTTP request to the callback configured for each channel. For internal channels, the default callback is Direct Line, but for external channels, the callback is aura-bridge.

But, following the same asynchronous approach in aura-bridge, it must be defined a callback for each channel, too. Therefore, channels connected to aura-bridge must be able to handle asynchronous HTTP communication.

Aura bridge limitations for Whatsapp channels

Facebook does not have Service-Level Agreements (SLA) in their APIs, thus messages managed by aura-bridge could suffer delay in their response times.
If no ACK is received for the message sent, aura-bridge will wait until AURA_QUEUE_MANAGER_SENT_MESSAGE_TTL (in milliseconds) is met and this could cause a delay in the next message response.

If an ACK is received, it can have the following statues:

export enum StatusType {
    // Message the user sent to your business was deleted by the user.
    Deleted = 'deleted',
    // Message sent by your business was delivered to the user's device
    Delivered = 'delivered',
    // Message sent by your business failed to send
    Failed = 'failed',
    // Message sent by your business was read by the user
    Read = 'read',
    // Message sent by your business was received by the server
    Sent = 'sent',
    // Indicates an item in a catalog is not available or does not exist.
    Warning = 'warning'
}

Each one of this states is obtained through a new ACK, and by rule we receive from one to two, but this may change due to its dependency with WhatsApp Business API.

Aura bridge outbound

aura-bridge-outbound is a component which shares the same logical implementation as aura-bridge, with the main distinction lying in the differences in its configuration, which implies a deployment specialization.

Its main mission is focused on returning the responses of aura-groot to the users and leveraging load for aura-bridge

ℹ️ aura-bridge and aura-bridge-outbound share the same make-up process. Further information in aura-bridge make-up

As aura-bridge and aura-bridge-outbound work together, the configuration of both components must be consistent. Further information in aura-bridge environment variables.

Functionality

aura-bridge-outbound is in charge of sending the responses to the corresponding channel callback. For WhatsApp channels, the default callback is Kernel WhatsApp API implementation.

In Aura distributed architecture, this component also works as a translator for aura-groot, but the other way round as aura-bridge does: it translates the Direct Line formatted responses from aura-groot into the corresponding format for each channel (WhatsApp, etc.) and also implements the corresponding protocol for each channel.

ℹ️ aura-bridge and aura-bridge-outbound environment variable AURA_BRIDGE_ENDPOINT should point to internal network (Ex. http://aura-bridge-outbound:8045). Further information about these variables can be found in aura-bridge environment variables.

Sequence diagram

sequenceDiagram
    Channel->>+Aura Bridge: Channel sends a message to bridge.
    Aura Bridge->>+Aura Groot: AURA_BRIDGE_ENDPOINT as serviceUrl variable points to Aura Bridge Outbound, bridge sends the message to groot.
    Aura Groot->>+Aura Bridge Outbound: Groot processes message and sends converted message to outbound.
    Aura Bridge Outbound-->>-Channel: Processed message is sent back to Channel.

ℹ️ Note that message’s information is synchronized between aura-bridge and aura-bridge-outbound due to their shared cache.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP

Aura NLP is the component in charge of processing, analyzing and understanding human natural language. Discover throughout these documents key descriptive documentation regarding this component.

Shared component between Aura Virtual Assistant and ATRIA

Related documents
Use cases development over Aura NLP

What is Aura NLP?

Aura NLP (Natural Language Processing) is the module of Aura Cognitive Services in charge of processing and understanding human natural language in simplified use cases.

Aura’s interaction with users is based on the intent & entity model: a user’s request expressed in natural language is understood by Aura in terms of identifying the user’s intent and the associated entities.

An NLP model contains three basic components: stages, connectors and pipelines. Stages provide different methods for the recognition of intents and entities in the user’s utterance. They are linked through different types of connectors composing an NLP pipeline.

When developing a use case, linguists or NLP experts must build up the NLP model and train it, that is, teach Aura to understand. Afterwards, the model is tested through an ongoing and cyclical process until its accuracy is good enough in terms of recognition of the use case intent and entities.

Throughout this section, you can access to detailed information, both descriptive and practical, regarding Aura NLP:

📄 Aura NLP basic concepts and components. Key concepts that must be known by linguists in order to manage Aura NLP.
📄 Configuration of the NLP system. Description of NLP operational configuration (internal) and introduction to the configuration of NLP stages.
📄 API definition
📄 Moreover, access our practical guidelines for NLP experts and linguists: Train Aura to understand: Use cases development over Aura NLP.

Overview of intent and entities recognition

Aura’s conversational process with the user is composed of three overall stages: the user makes a request to Aura; Aura recognizes the user’s intent and associated entities; Aura provides the user with the requested answer or service.

Two are the main actors in the process: while aura-bot is the component in charge of handling the conversational flow with the user, Aura NLP is responsible for the understanding process, which is schematically shown below.

Aura user asks for a service/request (utterance) through a specific channel.
aura-bot receives the request and handles it. For its understanding, aura-bot summons Aura NLP.
Aura NLP recognizes the intents and associated entities in the user request and sends the information back to aura-bot.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Channels

Discover in this section advanced technical documentation regarding Aura channels

Before facing these documents, we highly recommend reading the document Introduction to Aura channels, that provides a general overview of these components

Channels are shared components between Aura Virtual Assistant and ATRIA

Introduction

An Aura channel is any communication means a Telefónica client may use to interact with Aura, typically to gather information about, as well as to manage, the client’s Telefónica products and services.

As Aura platform is based on Microsoft Azure Bot Service and Bot Framework, it uses the general-purpose communication mechanism called Direct Line API, provided by Microsoft.

In this framework, Aura Virtual Assistant is able to communicate with different communication protocols:

Directly with channels that “talk” Microsoft Direct Line
Directly through communication protocols based on Direct Line, such as Auraline
Through the use of adapters for channels that do not support Direct Line.

To become an Aura-compatible communication means, Aura channels must fulfil a set of requirements as well as support the so-called Aura request-response semantic model. Specifically, it employs one property, channelData, that serves as a crucial bridge, allowing Aura to harness channel-specific data and functionalities, ensuring a tailored and optimized interaction.

Depending on this model, the Aura channel will support distinct levels of complexity regarding the interactions to be implemented as well as the richness of the information provided to customers interacting with Aura.

Moreover, the practical processes for the management of channels are included in Manage channels in Aura, including guidelines for the installation and activation of channels and how to update channels configuration.

Index of documents

Throughout these documents, you can find comprehensive technical information regarding channels in Aura:

Aura communication protocols: Description of Direct Line and Auraline communication protocols.
Channels authentication: How an Aura channel authenticates its users.
Aura channel model: Updated model for the communication between Aura and a channel.
Detailed information regarding key channels available in Aura:
- WhatsApp channel
- Google RCS Business Messaging
Manage channels in Aura: Guidelines for channels management in Aura, including installation and activation of channels and how to update channels configuration.
Use cases development. Guidelines for specific channels:
- Use cases development in WhatsApp
- Use cases development in Google RCS Business Messaging channel

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Global use cases

Discover the Global use cases developed by Aura Platform Team and put at the OB’s disposal and the global libraries and dialogs that contain the logic for different experiences

Aura Virtual Assistant components

Introduction

Global use cases are experiences already designed, developed and tested by Aura Platform Team.

OBs cannot modify their logic but use them as they are or get inspired for deploying their own experiences.

The current section includes:

Global dialogs / libraries, that contain the logic for the management of different experiences:
- Authentication dialogs, that manage different types of authentication in Aura.
- generic-ai-dialogs that allow managing several experiences for TV channels using AI services and complex logic resolution.
- generic-dialog that, currently, allows the implementation of certain experiences in TV channels.
- Other global dialogs grouped as miscellaneous.
Global use cases, as experiences that reach the final users:
Aura video use cases, that provide different experiences for Telefónica customers related to video services.
Use cases for testing Aura: libraries used by Aura Global Team for testing Aura Platform functionalities.

📃 Find guidelines for installing and using global experiences in your Aura system: Use global use cases.

ℹ️ With the introduction of the channelData normalized version (v3), these global dialogs will be progressively adapted to be compatible with this version. This information is included in the documentation of the specific dialogs where two versions (v1 and v3) are already available, together with the changes in the dialogs’ functionality that, for example, affect to video use cases.

Libraries configuration

aura-bot has the control to prepare and manage all configuration variables and make validations to get ready. In case that any validation fails, the set of libraries wrongly loaded will be shown and the bot will not start. If there is a problem with the scheme of an external library, the broken library should be extracted or fixed.

In order to validate schemas of the variables used in external libraries, it is optional that each library has an accessible scheme with the names of THEIR variables, default values, etc.

For example, there is a library that exposes a variable that contains a threshold that can be modified from outside that library. The developer´s library should create a schema (using @hapi/joi) specifying the name of the variable with the prefix of the library name (for example: LIBRARYNAME_THRESHOLD).

It is suggested that at the library level and within the src/ folder, a file named configuration-schema.ts is created, containing the configuration scheme like the one shown below:

import * as joi from 'joi';


const configurationSchema = {
    LIBRARYNAME_DEFAULT_GREETINGS: joi.string().default('hello'),
    LIBRARYNAME_COMMON_COUNT_GREETINGS: joi.number().default(1800)
};

export default configurationSchema;

In this first approach, there is no name validation (checking the prefix name of the variables).

Nomenclature for libraries variables

The naming of local library environment variables must follow these instructions:

Follow the structure: LIBRARYNAME_*.
If the environment variable is an endpoint: LIBRARYNAME_*_ENDPOINT.
If the environment variable is a Kernel endpoint: AURA_FP_*_ENDPOINT.
If the environment variable is a Kernel endpoint and is only used in one library or the use case uses a different API version than the global one: LIBRARYNAME_FP_*_ENDPOINT

Each library in this section includes its own variables.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura request-response semantic model

This section describes the Aura request-response semantic model, which makes it possible for external software components, application and services to speak a language Aura can understand using the Direct Line API

Aura Virtual Assistant component

Introduction

The Direct Line API provides the connectivity between any external software component, service, channel or application and Aura skills.

Anyhow, having a connection means is typically not enough, as a proper understanding and semantics are needed between the communicating parties regarding the data being intercommunicated. In order to improve the communication of Aura with services or channels being integrated into the Microsoft Azure Bot Service and Bot Framework, two concepts come into play:

Activities
Fixed JSON format. This fixed JSON format includes a JSON property called channelData, which channel developers and integrators can use to encode channel-specific data in JSON format and its corresponding semantics.

In this framework, we introduce the so-called Aura Request - Response Semantic Model. This semantic model makes it possible for external software components, application and services to speak a language Aura can understand using the DirectLine API.

Currently, this semantic model is used both for the communication of channels with aura-groot and for the communication of aura-groot with skills in two Aura communication protocols:

Within the Aura Request - Response Semantic Model, two key properties will be fully described throughout these documents:

The general-purpose channel-specific channelData property is the one used by Aura to define the model. When a message is sent from a channel, part of that message is specific to the channel/service and it allows aura-groot and skills to communicate correctly with it.
The payload property can be used by an external component to facilitate the sharing of information between the channels, aura-groot and the skills in order add extra information associated with the channel and, consequently, enrich the request.

Versioning

Aura supports different versions of the Aura Request - Response Semantic Model.

channelData v1

In older Aura Platform releases, the default version was version 1, so some channels did not send the version of the channelData and aura-bot handled their requests as version 1.0.0.

There is an exception: in release Aerosmith, aura-bot included Living Apps handling as a platform feature.

➡️ Access to channelData v1 documentation

channelData v2

Afterwards, version 2 was created to extend the channelData property with new fields, such as actions and status, that with the regular communications with the channels are not needed or, in the case of the actions, are solved by other ad-hoc means.

In Greenday (6.0.1) release, Aura included a WhatsApp connector, the so-named aura-bridge and also channelData version 2 was used to access and to use the extra features of actions and status.

In these Aura Platform releases until Iron Maiden, aura-bot included a lazy definition of the channelData model, so, in fact, it handled almost a different version per channel.

➡️ Access to channelData v2 documentation

channelData v3

From Iron Maiden (7.2.0) onwards, we introduce the so-called channelData normalized version (version 3), an enriched version of channelData with new fields that provide a complete schema definition. This will enforce all channels to use the very same schema to access aura-groot.

➡️ Access to channelData v3 documentation

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Authentication API

aura-authentication-api is the component in charge of handling users in Aura.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

aura-authentication-api is the component in charge of the management of the users’ authentication in Aura. It is a web server with several endpoints dedicated to handle users in Aura or to allow the access of the users to aura-bot.

The web server is built on Typescript 4.3 using Nodejs as engine. It is api-first designed, using Open API v3 to provide the API definition.

The authorization used in the server is based on an APIKey, which is ciphered for the environment and generated to access to a group of endpoints or only to a given path or a specific consumer.

Find detailed information regarding aura-authentication-api:

Communication protocol

aura-authentication-api communication protocol is completely synchronous, this means that the answer of a request is included in the HTTP response related to the incoming request.

Aura authentication API components

Server

The web server is implemented using express, that is the main web framework for Nodejs. It uses oas-tools on top of Express, to allow handling the Open API v3 file.

It is in charge of setting up all the rest of the components that are needed during a request processing, as well as reading the before mentioned swagger file and setting up all the routes defined in it.

Middlewares

Each route published in the API definition file is handled by a controller, but before a request lands on its controller, it goes through a series of middlewares, that provide some common steps needed by all the controllers of the server such as: request authorization, request validation, common parameters extraction, logging, metrics initialization, etc.

Controllers & Services

Then, the request lands on the controller. Each controller processes the request through a service in charge of implementing the logic. Once the request has been processed, the controller prepares and sends the response.

Database access

Some of the services of the aura-authentication-api access aura-users database to validate, get or update users’ information.

This database is a MongoDB one, with a collection that holds all the existing and valid Aura users. To access this collection, the server implements a data access object with the queries needed by the services. This object is a class called UserDao that provides a single access to the database, isolating the services from the real database schema and internal implementation.

The entity relationship diagram of this database is:

erDiagram
    users {
        string auraId
        string id
        string auraGlobalId
        string channelId
        string userId
        string authorizationId
        string authenticationType
        string authenticationIdentifier
        string idTokenHint
        date created
        date lastAccess
        date expiresAt
    }
    aura-version-control {
        string id
        string name
        string history
        date timestamp
        string version
    }

Collections description

users:
- _id: Internal MongoDB identifier. Not used by the service.
- auraId: Identifier of the user in Aura.
- auraGlobalId: Cross-channel identifier of the same authentication (type and identifier) of a user.
- channelId: UUID that univocally identifies the channel that lead the authentication session in Aura.
- userId: Kernel user identifier
- authorizationId: Identifier of the authentication session in Kernel. UNIQUE.
- authenticationType: Type of authentication used by the user. Values: email, uid, network, phone_number.
- authenticationIdentifier: Identifier used by the user during the authentication.
- idTokenHint: Token generated during the authentication of internal channels.
- created: Date when the user was created.
- lastAccess: Last date access of the user.
- expiresAt: Time when the user authentication should be discarded.
aura-version-control:
- _id: Internal MongoDB identifier.
- name: Name of the index being handled.
- history: Array of objects with all the versions already applied.
- timestamp: Date when the version was applied.
- version: Version of the database-index definition file applied.

Users database cache

Users collection data stored in MongoDB are also cached in Redis to increase request’s speed. The time to live of these documents in cache is configurable with the aura-authentication-api environment variable AURA_REDIS_CACHE_TTL.

Anonymous users

aura-authentication-api can return auto-generated anonymous users if needed for anonymous KPIs instead of returning 404 if the user is anonymous.

To enable this feature, you should include two new headers when the call to the endpoint /aura-services/v1/users/{auraId} is done:

return-anonymous: Boolean header to enable this feature.
x-4p-channel-id: UUID which identifies the Aura channel. This is needed to generate a consistent user data for this anonymous user.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Configuration API server

aura-configuration-api is the component of Aura in charge of handling and centralizing configuration tasks.
Find in the current documents the description of this component, its architecture, components and processes.

Shared component between Aura Virtual Assistant and ATRIA

Introduction

aura-configuration-api is the component in charge of the management of all the configuration tasks in Aura. It is mainly a web server built on Typescript 4.3 using nodejs as engine. It is api-first designed, using OpenAPI v3 to provide the API definition.

aura-configuration-api server is divided into several modules and plugins, each of them managing different tasks, which are:

Modules

Administration: Administrative management module for general tasks such as exporting and importing configuration.
Channel: Module in charge of managing the configuration of each Aura channel as well as the libraries associated with each channel (replacement for the bot-response.json configuration file).
Skill: Module in charge of managing the configuration of each skill.
Application: Module in charge of managing the configuration of each application.
Component: Module in charge of managing the configuration of each Aura component.

Plugins

Preset: Module in charge of managing the configuration of each preset.
Tv section: Module in charge of managing the configuration of each tv section.
Suggestion: Module in charge of managing the configuration of each suggestions.
Agent: Module in charge of managing the configuration of each agents.
Agent base: Module in charge of managing the configuration of each agents base.
Agent deployment: Module in charge of managing the configuration of each agents deployment.
Routing filter: Module in charge of managing the configuration of each routing filters.
Filter engine: Module in charge of managing filters engine.

Detailed information regarding aura-configuration-api is found in the following documents:

Aura configuration API architecture and components

The following figure shows the main components of the aura-configuration-api.

Server

The web server is implemented using express, which is the main web framework for NodeJS. It uses openapi-backend on top of express, to allow handling the OpenAPI v3 file.

It is in charge of setting up all the rest of the components that are needed during a request processing, as well as reading the aforementioned swagger file and setting up all the routes defined in it.

Middlewares

Each route published in the API definition file is handled by a controller, but before a request lands on its controller, it goes through a series of middlewares that provide some common steps needed by all the controllers of the server.

The following middlewares are executed at each request to the server:

Normal flow

helmet: Secure express app by setting various HTTP headers.
body-parser (urlencoded): It parses URL-encoded bodies using querystring parsing (qs).
body-parser (json): Maximum size of JSON body (current 50 MB).
set-correlator: It sets the correlator to response header and response locals.
log-http-request-response: Log HTTP request and response information.
express-prom-bundle: It saves metrics information using Prometheus.
oas_tools (swaggerMetadata): It interprets swagger resources and attaches metadata to request.
oas_tools (swaggerValidator): It validates requests using swagger definition.
oas_tools (swaggerRouter): It routes validated requests to appropriate controller.
oas_tools (swaggerUi): It serves the swagger documents and Swagger UI.
not-found-handler: It handles requests that have not been routed (Not Found).

Error flow

error-handler: Global error handler.

Controllers & Services

When the previous step is finished, the request lands on the controller. Each controller processes the request through a service, that is in charge of implementing the logic. Once the request has been processed, the controller prepares and sends the response.

There is a controller to manage requests associated with the server itself (generic controller) and a controller for each module, in charge of handling the requests of that module.

The event synchronization service can be enabled by setting the AURA_CONFIGURATION_EVENTS_ENABLED configuration variable to true.

Generic controller

As indicated above, the generic controller handles requests to provide information from the state of the server itself.

healthz: It is used to check if the server is online returning a simple status response: {"status": "ok"}.
metrics: It gets Prometheus metrics information.
heapStatistics: It gets memory usage information.
heapSnapshot: It gets memory head snapshot from v8 engine.
shutdown: Service to be invoked mainly by Kubernetes and allows to do a server graceful shutdown.

Modules controller

The controllers and services information of each module is described in the Modules section.

aura-configuration-api server is divided into several modules and plugins, each of them managing different tasks, which are:

Modules

Administration: Administrative management module for general tasks such as exporting and importing configuration.
Channel: Module in charge of managing the configuration of each Aura channel as well as the libraries associated with each channel (replacement for the bot-response.json configuration file).
Skill: Module in charge of managing the configuration of each skill.
Application: Module in charge of managing the configuration of each application.
Component: Module in charge of managing the configuration of each Aura component.

Access here detailed information of aura-configuration-api modules

Plugins

Preset: Plugin in charge of managing the configuration of each preset.
TV section: Plugin responsible for the management of TV sections.
Suggestion: Plugin responsible for the management of suggestions.
Agent: Plugin in charge of managing the configuration of each agent.
Agent base: Plugin in charge of managing the configuration of each agent base.
Agent Deployment: Plugin responsible for managing the deployment of agents.
Routing filter: Plugin responsible for managing the routing filters.
Engine filter: Plugin responsible for managing the engine filter.

Access here detailed information of aura-configuration-api plugins

Database access

Most aura-configuration-api services access aura-configuration database to get or set the information of each module.

This database is a MongoDB one, with a collection for each module that stores information related to that module. To access this collection, the server implements a data access object with the queries needed by the services. This object follows the DAO (Data Access Object) pattern that provides a single access to the database, isolating the services from the real database schema and internals implementation.

Communication protocol

aura-configuration-api communication protocol is completely synchronous, this means that the answer of a request is included in the HTTP response related to the request.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Context

Aura Context, Aura’s short-term memory, is a key component that allows the enrichment of the responses provided to the users with information from their previous experiences.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction to Aura Context

In previous Aura Platform releases, Aura experiences had a lack of memory, as every time the user interacted with Aura, the system took it as a first interaction and, once it had finished, Aura forgot it.

In this framework, Aura Global Team has introduced the concept of context, defined as information surrounding a user interaction. Aura Context aims to enhance the Aura-user interaction by providing the system with short-term memory. This capability enables Aura to understand, act and learn based on recent interactions with the user.

Through the storage of key information regarding the user’s previous experiences and environment in a temporary, high-available, low-latency database during a configurable period of time and its further integration in the conversational flow due to the real-time recovery of data with no lateness, Aura is able to better understand the user. This, in turn, allows it to provide more comprehensive responses and adjust its actions based on this information. The stored data is specific to one user and one interaction of that user.

Aura Context is not a critical module for use cases development but can be used optionally for the enrichment of the overall experience.

Aura Context stakeholders

In Aura Context framework there are two main stakeholders, with differentiated roles:

Aura Global Team
- Management of the Aura Context data model
- Edition of the Aura Context global data model
- Feeding of the database with data from different Aura components
- Use of Aura Context by global developers
Aura Local Team (OBs)
- Mainly consumers of Aura Context data (although they can also carry out certain modifications over the global data model)
- Storage of data in the Aura Context database
- Making queries to the database

ℹ️ The current documents are focused on the use of Aura Context for Aura Local Teams

What is included in Aura Context v1?

From the general definition and scope of Aura Context included in the previous section, the current version of this component, Aura Context v1 constitutes the first approach of the end-to-end product.

Currently, the context generated by Aura is able to retrieve the following information:

What has Aura understood?
What is the current status of the channel/device from which the user is interacting?

For this purpose, Aura Context v1 includes the following basic functionalities:

Each OB can create its own agent to start working with Aura context in the aura-bot dialogs.
Through this agent, the OB can perform the following actions:
- Queries to the context database in order to retrieve key user’s information.
- Creation of custom context fields, if it is needed to store and query information that is not included in the context data model.
- Storage of data in context, for further use in use case development.

Every information included in the context module is stored in Kernel, so datasets are available for analytical purposes or to create intelligent models upon it, as long as you have the right consent to do so.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura entities

Description of entities in Aura and components in charge of managing these entities

Aura entities belongs to both Aura Virtual Assistant and ATRIA

Introduction

Aura entities are files stored by different Aura components that contain relevant information related to key system processes or actions: user messages, Aura components interactions, message handling, applications, etc.

They are useful to measure and evaluate the performance of the system against defined objectives through the generation of KPIs, processes tracking, identification of issues and decision-making.

Each Aura component generates a series of entities which are uploaded into Azure Storage for different purposes.

aura-kpis-uploader is the component in charge of the management of entities for every Aura component. On an hourly basis, it uploads all the generated files to Kernel datasets. Once there, specific algorithms are executed to calculate the KPIs of each instance or for other purposes.

Detailed information regarding Aura entities is found in the following documents:

Types of logs in Aura: Operational logs and KPIs entities logs.
Description of Aura KPIs blob container.
Aura Databricks Jobs: Component that imports Avro-formatted files into Kernel datasets.
Aura KPIs uploaderComponent in charge of handling entities and dimensions.
KPI entity handler: aura-bot module in charge of managing Aura entities.
Status codes stored in KPIs entities: List of response codes stored in KPIs by aura-bot.
Aura entities definition: Database that includes the different entities currently used in Aura.

Types of logs in Aura

aura-bot writes two different types of logs:

Operational logs

Operational logs are written using AuraLogger that writes each row, by default, in JSON format in the standard output of the POD running each instance of aura-bot.

These logs are used to monitor or debug aura-bot. The standard output of each POD is aggregated to be stored in an ElasticSearch cluster, to make it available in Kibana.
Entities logs

An entity is a specific definition of one of the actors involved in the processing of each activity in aura-bot. So, the bot is in charge of writing the corresponding rows for each activity on each entity.

This section aims to describe how the entities are handled by aura-bot and how the rows are written and included in the entity files.

Logs are written in a blob file: a blob container in Azure Storage using an internal library aura-kpi-handler, that provides classes and utilities to decouple Aura components: how the rows are written from how the needed information is gathered.

The database Aura entities definition includes the different entities currently used in Aura. Entities are generated by different Aura’s modules, each of them in charge of performing a different task: aura-bot, aura-groot, aura-services, etc.

Currently, these logs are used to generate KPIs for the measurement of Aura performance, tracking processes or identifying issues.

Aura KPIs blob container

The following figures show the Aura KPIs blob container and an example of content of the blob container for aura-bot.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura file manager

aura-file-manager is the microservice in charge of uploading a file to a temporary repository, validating it and returning a link as validation result.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

aura-file-manager is a microservice based on a REST API with the capability of uploading a file to a temporary repository, validating it, and returning a direct download link as validation result.

aura-file-manager uses an utility, aura-file-validator for the validation of files.

Find detailed information regarding Aura file manager in the following documents:
. Architecture and main components
. Communication protocol
. Executed process flowchart and API functionalities
. Environment variables
. API definition

⚠️ In order to use aura-file-manager, it is necessary to activate it previously. Learn how in Aura File Manager activation.

Aura file manager architecture

In the following diagram, the architecture and main components of aura-file-manager are shown:

Internal components

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a configuration file or environment variables.

FileService

FileService is where the main functionality of the microservice resides. It is where file upload and validation is done.

HTTP server

Microservice is implemented as an HTTP server (FileManagerServer) that exposes an API to receive the files to be validated.

Middlewares

The route published in the API definition file is handled by a controller but, before a request lands on its controller, it goes through a series of middlewares that provides some common steps needed by all the controllers of the server such as: request validation, common parameters extraction, logging, metrics initialization, etc.

Queue messages processing

Queuing system that allows tolerance to server failures, storing the received files in a DB and processing them later.

Communication protocol

aura-file-manager communication protocol is completely asynchronous, this means that the answer of a request is not included in the HTTP response related to the request, but it is sent as a new HTTP request to the callback aura-bridge.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Complex Logic Framework

Aura Complex Logic Framework (CLF) is the component in charge of the resolution of complex use cases in Aura.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

What’s Aura Complex Logic Framework?

The resolution of complex experiences in Aura requires that, once the user’s intent has been recognized by Aura NLP (Natural Language Processing), the dialog triggered by this intent summons the Complex Logic Framework (CLF) that allows developers to improve use cases, creating and integrating plugins in Aura Platform to be consumed by the aura-bot dialogs and extend their logic for the resolution of complex users’ requests.

Therefore, it will be a key Aura component for the resolution of use cases, for example, for TV platformization.

Throughout the CLF documentation, you will discover:

CLF architecture and components: Description of CLF architecture and main components.
CLF configuration: General configuration and configuration of plugins.
CLF files: Description of Aura CLF files and libraries.
CLF global plugins: CLF global plugins developed by Aura Global Team.
CLF API definition: The Complex Logic Framework is based on Python plugins and is able to access Kernel APIs and other internal APIs exposed by other internal Aura components. Check here the main swagger for all the CLF plugins.

Interaction of CLF with Aura

The interaction of the Complex Logic Framework with Aura is shown in the following figure and explained below:

An Aura user asks for a service/request (utterance) through a specific channel.
aura-bot receives the user´s request. Firstly, aura-bot recognizers come into play.
The corresponding recognizer summons Aura NLP for the recognition of the user’s utterance (in certain situations corresponding to auraCommands, the user’s request is understood directly by the bot).
NLP returns the intent and associated entities in the user’s utterance.
The recognized intent triggers a specific aura-bot dialog, in charge of the logic for the resolution of the use case.
The dialog recognizes the complexity of the use case and, if required, summons the Complex Logic Framework.
Depending on the type of plugins, the bot will call a different API.
After that, the Complex Logic Framework selects the appropriate plugin among the ones existing on this specific type, based on the identified intent (mapping the intent with the plugin).
The plugin executes the programmed tasks and calls different modules (Aura’s internal data, Kernel data, external databases, etc.) in order to fetch the information.
The Complex Logic Framework receives the required data.
The information is sent back to aura-bot.
aura-bot composes the response to the user’s request.
The response is provided to the user.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Redis Mongo Synchronizer (ARMS)

aura-redis-mongo-synch (ARMS) is a service that allows the use of Redis and MongoDB to create a Bot Framework Context management as a two-level cache.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

To implement a two-level cache strategy:

Redis will be used as the first-level cache, with a short expiration time to speed up writing operations.
MongoDB will be used as the second-level cache, with a high expiration time.

aura-redis-mongo-synchronizer (ARMS) will be responsible for transferring expired data from Redis to MongoDB.

The service is implemented as a web server just to make it easier to handle it and to connect with Aura monitoring systems, such as Prometheus. It is in charge of managing the expired data in Redis using the Pub/Sub system provided by Redis.

Aura Redis Mongo Synchronizer components

The following figure shows the main components of the aura-redis-mongo-sync.

Server

The web server is implemented using fastify, a very fast web framework for NodeJS.

The web server works only as the service controller, while all the synchronization tasks are performed by a module outside the service called aura-redis-mongo-sync or ARMS.

Modules

Redis Mongo Sync

It is responsible for synchronizing data between Redis and MongoDB.

It uses a publish/subscribe system provided by Redis to notify the module when to synchronize data from Redis to MongoDB, based on events generated by Redis regarding the changes in the status of the documents.

Controllers & Services

When the previous step is finished, the request lands on the controller.

Each controller processes the request through a service in charge of implementing the logic. Once the request has been processed, the controller prepares and sends the response.

There is a controller to manage requests associated with the server itself (generic controller) and a controller for each module, in charge of handling the requests of that specific module.

Generic controller

As indicated above, the generic controller handles requests to retrieve information of the status of the server itself.

healthz: It is used to check if the server is online returning a simple status response: {"status": "ok"}.
lastevent: It returns the date of the last synchronization between Redis and MongoDB: { status: 'ok', lastEvent: DATE };
shutdown: Service to be invoked mainly by Kubernetes cluster to perform a server graceful shutdown.

Data Storage

A new class has been created that implements the Bot Framework storage interface called RedisMongoDbStorage.

This class writes the context data to Redis and firstly, it reads the data from Redis. If this data does not exist in Redis, it fetches it from MongoDB.

This class is included in the aura-common-utilities repository. Check the Github repository for further details.

Data structure in Redis

Redis stores its data in key:value structures. This value can support different data structures.

In the following sections, the ones used to store the bot context are explained.

These structures support an expiration time so that Redis deletes them when the expiration time is reached, so the content has expired. The expiration of a record in Redis emits an specific event, of expired type, that can be handled by an external module.

Context (Bot Framework Context)

The bot context is stored in a key:string structure:

Key: context:[MONGO DATABASE NAME]:[MONGO COLLECTION NAME]:[MONGO CONTEXT INDEX KEY]
Value: The content of the context is stored in a string obtained by converting the Object context in JSON to string.
Expiration: [EXPIRATION]

Where:

MONGO DATABASE NAME: Name of the database to use when ARMS writes this data in MongoDB.
MONGO COLLECTION NAME: Name of the collection to use when ARMS writes this data in MongoDB.
MONGO CONTEXT INDEX KEY: Key to use when ARMS writes this data in MongoDB.

When a key:value data expires in Redis, an event is emitted indicating which key contained the expired record (event => expired). The data can no longer be retrieved because Redis deletes value data before emitting the event and if this event is not processed, it will not recur, forcing us to use some auxiliary data structures in order to recover the expired data.

Shadow Keys

A Shadow Key is a key:void structure used to emit the event. As the expiration event returns a key, that will contain the current key of the context containing the data.

To improve efficiency, this shadow-key contains the key SHARD, which allows ARMS to subscribe only to certain shards to improve performance.

The number of shards is defined in the variable AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT, which indicates that the algorithm to calculate the shard based on the key will generate numbers between 1 and the value of the variable.

It is important to notice that the number of shards configured in AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT must be the same as the number of pods configured in the aura-redis-mongo-synchronizer.

Key: shadow-key:[SHARD]:context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY]
Value: empty
Expiration:  [REDIS CONTEXT DATABASE EXPIRATION]

When the service receives the expiration of this key, we can extract the shard and the real value from the context.

Active Context List

To ensure all data is synchronized, we need to keep in a list all the active context keys. By doing that, if an error occurs in the service and the expiration notifications are lost, we can recover data by periodically checking this list.

The data structure to be used is key: score-List. The score list contains a list of values with a string field and a numeric field by which the data can be sorted. The string field is used to store the key and the numeric field to store the expiration date.

In this way, we can retrieve those expired records that have not been processed, as items are removed from the list once they are synchronized.

Key: active-context:[SHARD_VALUE]
            context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY 1]     DATE_1
            context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY 2]     DATE_2
            ...
            context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY N]     DATE_N

To store these elements, we will use partitioned lists in SHARD mode, that is, they will be included in the list that corresponds to them according to their KEY. The SHARD_VALUE is a number that is calculated from the key and the module obtained by dividing this number by the number of partitions that are configured.

For example, there is a series of lists to avoid collisions between PODS of the service (ARMS). That is, if the KEY1 key returns 301 (this data is obtained by performing an operation on the content of the key) this is divided by the number of partitions, 5 for example, and the module +1 is obtained, to avoid the 0. This indicates that the list where it will be stored will be active-context:2.

The execution of this process is based on the current system load. The variables associated to the execution period are:

AURA_REDIS_CONTEXT_CACHE_CACHE_CHECK_UNPROCESSED_INTERVAL_MIN, which indicates in seconds the minimum time that will be set.
AURA_REDIS_CONTEXT_CACHE_CACHE_CHECK_UNPROCESSED_INTERVAL_MAX, which indicates in seconds the maximum time for the interval.

The system will recalculate the next run based on the load. If the number of elements to be processed is very high, the interval will be reduced until it reaches the minimum, while if the load is low, the interval will be increased until it reaches the maximum.

SHARD variables

To manage the SHARD and the list to be selected by the ARMS, two variables come into play in Redis stored in the key:string structure.

AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT : Number of lists to generate (partitions).
AURA_REDIS_CONTEXT_LAST_SHARD_PROCESSED: Partition to manage at that moment, that is, when a service requests this data, one is added to it so that the next service that requests it can access the next partition, and so on.
AURA_REDIS_CONTEXT_LAST_INDEX_SHARD_PROCESSED: This variable is used to assign event subscriptions in an orderly fashion. For example, if we have two ARMS services and the AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT is set to four, then the first service will subscribe to 1 and 2 and the second service to 3 and 4. This order of assignment is done by incrementing this environment variable.

Synchronization Flows

Read Data

The Bot Framework components will read data in the following order:

They will try to read data from Redis.
If Redis does not return data, they will try it in MongoDB.

ARMS will read data in two situations:

When an expiration event is issued, ARMS reads the corresponding data, obtaining the context key contained in the expired KEY.
When the active context control service runs (it does it periodically) and it has some expired context that must be synchronized.

Write Data

Bot Framework components will write all their data in Redis.
ARMS will write the synchronized records in MongoDB.

NOTE: ARMS also writes data in Redis but only to control its core and internal behavior, incrementing SHARD, updating active context lists, etc.

Distribution of the event subscription

As previously mentioned, every time a Shadow-Key expires in Redis, it emits an event that will be received by the ARMS services that are subscribed to that event.

Now, as the Shadow-Key contains the SHARD of the real key of the data, we can distribute the events among different ARMS to improve performance and avoid network overload.

For the assignment of the events to subscribe, there is a method that is executed every certain time. The value is in the variable AURA_REDIS_CONTEXT_CACHE_CHECK_INDEX_SHARD_INTERVAL and is measured in seconds.

Steps to generate event subscription

Save the current state of the service: This saves, in a Redis structure, the identifier of the current ARMS and the current time.
Consult the number of ARMS services available: This is done by consulting the previous structure, obtaining those services that have been recently updated.
Calculate the number of events to subscribe: This is done by dividing AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT, which contains the number of partitions, by the number of services. There is a variable that can limit this assignment, AURA_REDIS_CONTEXT_CACHE_MIN_SHARD_BY_NODE: if this variable is greater than the previous result, the value of the variable is used. For example, if the result is 1, but this variable has the value 2, the service must subscribe to 2 events.
Assign the events: Once the number of events to subscribe to is determined, the AURA_REDIS_CONTEXT_LAST_INDEX_SHARD_PROCESSED variable is consulted and incremented. The value to assign is the module between the returned value and the value of AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT. In this way, it will assign consecutive values between 1 and the AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT.
Subscription to Redis: Once the SHARDs to be subscribed have been assigned, we only need to disconnect the events that do not match the current selection and connect the ones we do not have connected.

To obtain an optimal performance, ideally the number of AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT should at least match the number of ARMS services.

Example of distribution for AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT = 5, and adding ARMS services until reaching 5.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura microfrontend base

Description of aura-microfrontend-base, foundational component for the development of user interfaces in Aura ecosystem

Aura Virtual Assistant component

Introduction

aura-microfrontend-base is a foundational component that provides a common structure and libraries for the development of user interfaces in Aura ecosystem (Aura Virtual Assistant and ATRIA).

Its microfrontend component is a small, self-contained element that is part of a larger microfrontend architecture. It follows the same principles as microservices, but for the frontend, allowing independent development, deployment and scaling of different parts of a web application.

As defined before, this is a base component over which certain interfaces will be developed, but it will not be directly productized.

Technical foundations

This is a Next.js project bootstrapped with create-next-app.

The project leverages the following technologies and tools:

React: A JavaScript library for building user interfaces, used as the core framework for component development.
@telefonica/mistica: A design system and component library that provides a set of reusable UI components and styles, ensuring consistency and best practices in UI development.
Tailwind CSS: A utility-first CSS framework for styling components and layouts.
TypeScript: A strongly typed programming language that builds on JavaScript, ensuring type safety and better developer experience.
Microfrontend Architecture: Enables independent development, deployment, and scaling of frontend components.
Dynamic Imports: Used to load microfrontend components at runtime, improving modularity and performance.
Environment Variables: Configuration is managed through environment variables like AURA_SITE_CONFIG_PATH to ensure flexibility and adaptability across environments.

These foundations ensure scalability, maintainability, and a seamless developer experience.

Architecture and components

aura-microfrontend-base is composed of two main components:

Core Framework: The foundational Next.js structure that handles routing, rendering, and component lifecycle.
Dynamic Component Loader: The system responsible for loading and rendering components based on the site configuration.

Basic architecture diagram

sequenceDiagram
    participant Project as user request
    participant Config as site-config.json
    participant Pages as Page Manager
    participant Layout as Layout Engine
    participant ComponentLoader as Component Loader
    participant DOM as Browser DOM

    Project->>Config: read site configuration
    Config-->>Project: return site structure
    
    Note over Project: site path or default page
    Project->>Pages: setup page routes from config
    
    Pages->>Layout: process page layout configuration
    
    Layout->>Layout: setup container with containerClass
    Layout->>Layout: create grid with gridClass
    
    Note over Layout: process components by position
    
    loop for each component
        Layout->>ComponentLoader: request component by importType
        
        alt importType is "internal"
            ComponentLoader->>ComponentLoader: load from local components folder
        else importType is "external"
            ComponentLoader->>ComponentLoader: load from library dependency
        else importType is "microfrontend"
            ComponentLoader->>ComponentLoader: load dynamically from remote source
        end
        
        ComponentLoader-->>Layout: Return component
        Layout->>Layout: Apply className and position
        Layout->>Layout: Inject attributes and props
        Layout->>DOM: Render component in grid position
        DOM-->>Layout: Component rendered
    end
    
    Layout-->>Pages: Page layout rendered

    
    Pages-->>Project: page configured and route ready
    Project->>Project: start listening for navigation

Server features

The server component of aura-microfrontend-base provides:

Dynamic Routing: Routes are generated based on the configuration file and mapped to corresponding components.
Server-Side Rendering (SSR): Components can be rendered on the server for improved performance and SEO.
API Integration: Built-in capabilities for connecting to backend services and APIs.
Environment Variable Management: Configuration is handled through environment variables for different deployment environments.
Development Server: Local development environment with hot-reloading for a smooth development experience.

Configuration

The AURA_SITE_CONFIG_PATH environment variable must be set to point to your project’s configuration file, as in the example below:

AURA_SITE_CONFIG_PATH=site-config.json npm run dev

aura-microfrontend-base configuration file must be a valid JSON that defines the structure of the site, including the layout of the pages and the components to render.

The configuration file must follow this structure:

{
    "mainPage": "home",
    "pages": [
        {
            "id": "home",
            "name": "Home",
            "path": "/home",
            "layout": {
                "containerClass": "min-h-screen flex flex-col container mx-auto",
                "gridClass": "grid grid-cols-1 grid-rows-[200px_1fr] flex-1",
                "components": [
                    {
                        "library": "@telefonica/example-library",
                        "componentName": "Header",
                        "position": 1,
                        "importType": "external",
                        "className": "col-span-2 row-span-1"
                    },
                    {
                        "componentName": "TestComponent",
                        "position": 2,
                        "importType": "internal",
                        "className": "h-full col-span-1 row-span-2"
                    },
                    {
                        "componentName": "TestPlugin",
                        "position": 3,
                        "importType": "microfrontend",
                        "className": "h-full col-span-1 row-span-2",
                        "attributes": {
                            "API_ENDPOINT": "${AURA_MICROFRONTENDS_SERVER_HOST}/api/plugin-test",
                            "REFRESH_INTERVAL": "1000"
                        }
                    }
                ]
            }
        }
    ]
}

Configuration fields

mainPage: Default page ID to load when accessing the root path
pages: Array of page configurations
- id: Unique identifier for the page
- name: Display name of the page
- path: URL path for the page
- layout: Container configuration - containerClass: Tailwind classes for the main container - gridClass: Tailwind classes for the grid layout - components: Array of component configurations - library: (Optional) External library name - componentName: Name of the component to render - position: Order in the layout - importType: Type of component (internal, external, or microfrontend) - internal: Components that are part of the current project and are located in the components directory - external: Components from external libraries (like @telefonica packages) that are installed as dependencies - microfrontend: Components that are loaded dynamically from a remote server, allowing for runtime integration of independent applications - className: Tailwind classes for the component - attributes: (Optional) Additional component properties

Technical guidelines

Guides to manage the aura-microfrontend-base component:

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

This section contains the technical documentation of Aura widget components.

⚠️ This component is not available in the current Aura Platform release

The current documentation about Aura widget component includes:

Aura channels factory in charge of the integration of aura-bot in web and mobile environments (iOS and Android).

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

APIs documentation

Index of Aura Virtual Assistant and ATRIA APIs definition documents

APIs belong to both Aura Virtual Assistant and ATRIA

Introduction

An application programming interface (API) is a software interface that allows two applications to interact with each other without any user intervention. The API is composed of a collection of software functions and procedures for creation of applications to access features or data of a system, application or other service.

This section contains all the information related to Aura APIs, both the ones focused on internal use and others used by Aura’s channels and users to handle their conversations.

Most of our services lay on OpenAPI definitions v3.

APIs documentation is provided within the documentation for each specific component. Select the component of your interest and find the available APIs.

Select the specific API in the left menu and access to the corresponding section to find the API definition document.

Available APIs

Aura Virtual Assistant components	APIs
aura-bridge	aura-bridge APIs definition
Complex Logic Framework	Complex Logic Framework API definition
aura-authentication-api	aura-authentication-api API definition
aura-file-manager	aura-file-manager API definition

ATRIA components	APIs
aura-gateway-api	aura-gateway-api API definition
atria-model-gateway	atria-model-gateway API definition
Advanced Analytics Models (AAMs)	Advanced Analytics Models (AAMs) API definition

Cross components	APIs
Aura NLP	Aura NLP API definition
aura-configuration-api	aura-configuration-api API definition
Release Train Manager	Release Train Manager API definition