Aura – architecture

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Bot architecture

Description of Aura Bot architecture and starting components

Introduction to Bot Framework service

A bot is an application, particularly an HTTP server, that allows interaction with users in a conversational way. Different means of interaction are available, such as text, cards with images, video or audio and speech.

Users interact with the bot through a given channel, that is the final application used directly by the users. For instance, the bot can be accessible from Facebook, WhatsApp or from any private application, such as My O2 or Mi Movistar. Every message exchanged between the user and the bot is called an activity.

Microsoft Bot Framework Service, that is the main component of Azure Bot Service, is responsible for sending the user’s request to the particular bot instance, in this case, to the corresponding instance of aura-bot. Channels communicating with a bot should include additional information required by this specific bot to handle the message in the body of the request, i.e., in the activity. aura-bot provides a definition of the channelData field of the activity, so channels are able to send and receive this information needed to handle Aura use cases.

aura-bot, as a bot based on Microsoft Bot Framework, understands different types of activities. The most basic types of activities used by aura-bot are:

Message: Means to transmit the user’s needs to the bot.
Event: Events are in charge of the end-to-end communication between aura-bot and channels. They are managed by the incoming-event-middleware.
ConversationUpdate: Activity that describes a change in conversation’s members, description, existence, etc.

Communication between BotFramework Service and aura-bot is based on HTTP. BotFramework Service sends an HTTP POST request to the bot, with the incoming activity from the user. The bot should respond with a 200 HTTP status code to this request. In aura-bot, this is executed before going on with the processing of the activity, so the echo message of the user is returned before the activity generated as an answer. These activities are sent to BotFramework Service as HTTP POST request, that is responded with 200 HTTP status code, too.

Once an activity arrives at the bot, the HTTP server should pass it to the BotFramework Adapter, that starts the activity processing.

At this moment, a turn starts, that refers to all the steps done from the moment that an activity enters the bot until all the answer activities are returned to the user. Each turn lays on a TurnContext, that holds all the information about the current activity. Bot Framework automatically includes in the TurnContext object some information, but aura-bot extends this object with extra information that makes it easier to provide a response to the user.

Please, check detailed information about Microsoft Bot Framework Service and the related SDK.

Aura Bot core architecture

The following diagram shows all the components running in any aura-bot instance. They are shown as they are loaded during bot start-up.

Legend
Blue boxes: singleton components Yellow boxes within them: components that are handled the same way Middlewares: - M: mandatory ones - O: optional, executed only if they are properly configured Execution mode: - F: execution flow stops when middleware fails - T: execution continues when middleware fails

Aura Bot messages flowchart

As explained before, messages are a type of activities that manage requests from the users: aura-bot receives a request from a user and provides an answer back.

The following diagram include the messages flow through aura-bot.

Aura Bot starting components and modules: Aura orchestrator

Currently, aura-bot includes an orchestrator that controls how the system is loaded and indicates which components and modules are required for the bot start-up.

The orchestrator handles a sorted array of components that must implement a series of methods that may receive the bot configuration if necessary. Modules are then initialized asynchronously, although they were synchronous.

This behavior simplifies both the definition of the singleton modules and how the orchestrator initializes them, with a minimum loss of performance during start-up.

        const orderedModules = [ModuleA, ModuleB, ModuleC];
        // Instantiate the Service App.
        const appOrchestrator = new Orchestrator();
        // Add configuration, that will be required by almost all other modules
        appOrchestrator.setConfigurationManager(Configuration);
        // Add the dependent modules in order.
        appOrchestrator.addModules(sortedModules);
        // Initiate the App.
        await appOrchestrator.init();

The orchestrator module is created during the server start-up with the following parameters:

ConfigurationManager:

Class that handles aura-bot configuration, that is the component in charge of loading the environment variables.

Singleton Modules:

They are: MiniBotServiceMock, AuraChannelsConfiguration, AuraMongoDb.Connector, KpiHandler, HttpMonkeyPatcher, PluginManager, LocaleManager, AuraCacheManagement, AuraBotServer.

These modules are loaded in the given order. If any of these modules does not start correctly, aura-bot does not start. Otherwise, AuraBotServer is up and running at the end of the process. All these modules are available all along the bot.

MiniBotServiceMock is only started if the bot is running locally through Aura minibot.

How to define singleton modules in Aura

In order to develop a new singleton module that can be used by aura-bot both internally and during the dialog execution, developers should keep in mind the following issues:

All modules added to the orchestrator need and publish a static variable called instance, that holds the singleton instance of that module.
Modules must have a public static method: init, that could be async or not depending on the needs of the module start-up. This method creates the singleton instance (and sets it to then instance variable). This method should only be called once by module or must throw an exception. It is only called by the orchestrator.
It is also recommended to provide an empty private constructor that avoids the creation of new instances of the module.

An example of implementation is shown below:

export class Class1 {
    public static instance: Class1;
    private constructor() {
        // Stuff here...
    }
    public static async init(config: Configuration): Promise<Class1 > {
        if (!Class1 .instance) {
            Class1 .instance = new Class1 ();
            // More stuff here

            return Class1 .instance;
        } else {
            throw new Error('An instance of Class1  already exists');
        }
    }
}

To use any of these modules:

Import its dependency.
Access the module instance property and call the required method.

The following example shows how to get a text translated by the LocaleManager.

import { LocaleManager } from '@telefonica/aura-utilities/lib/aura-locale-manager';
…
LocaleManager.instance.getText('errors:error.invalidScopes', auraUser, corr);

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura bridge architecture

Description of Aura bridge general architecture at the level of components

Introduction

aura-bridge architecture is designed based on three fundamental components:

aura-bridge: It is identified as the core component of the aura-bridge. It is in charge of starting the basic functionality modules, loading the plugins and starting the expressjs server.
aura-bridge-common: It can be defined as an SDK (Software Development Kit) to be able to develop a plugin.
aura-bridge-plugin. Component that provides functionality to aura-bridge. This component works the same way as a service in an architecture oriented to microservices (isolated, small and self-contained).

Aura bridge

As mentioned above, it is the core of the system and can work without the need for any plugin.

Source code structure

src
├── cache               # Two level cache module
├── config              # Configuration manager module 
├── controllers         # Generic and metric controllers (oastool)
├── make                # Aura bridge make up
├── message-sync        # Queue management system
├── middlewares         # Express middlewares
├── modules             # Bridge modules: behavior-manager, plugin-manager
└── utils               # Bridge utilities: prometheus-utils

Aura bridge common

aura-bridge-common contains all the necessary utilities in order to develop a new aura-bridge plugin and the models used in these utilities.

Aura bridge plugin

Base class to develop an aura-bridge client.

To implement a new client, it is only necessary to extend the class AuraBridgeClient and implement the sendMessage method.

export class TestClient extends AuraBridgeClient {

    public async sendMessage(message: TestMessage, options: SendMessageOptions): Promise<any> {
      ...
    }
}

If the new aura-bridge client needs to use the OAuth protocol, it is possible to extend the class AuraBridgeClientOAuthTokens and indicate the configuration of OAuth clients, passing the parameter authClients in constructor. In this way, the tokens will be updated automatically.

export class TestOauthClient extends AuraBridgeClientOAuthTokens {
  ...
}

aura-bridge-flow

AuraBridgeFlow allows the generation of a “processor” type plugin in a quickly and standardized way: control of errors, logs, metrics, client retry policies, etc.

As a rule, a “processor” type plugin receives, converts and sends a message to a destination. In case of error, it can send messages to another list of destinations.

sequenceDiagram
    Origin->>+Processor Plugin: Message
    Processor Plugin->>+Processor Plugin: Convert message

    alt ok
        Processor Plugin->>+Destination: Converted message
    end
    opt Extra response
        Processor Plugin->>+Error destination (1): Error message (1)
        Processor Plugin->>+Error destination (2): Error message (2)
    end

aura-bridge-utils

AuraBridgeUtils contains utilities for request information, get generic errors, etc.

aura-bridge plugins

A plugin is a module/component that provides aura-bridge with new functionality and works independently without affecting other existing functionality in the system.

📃 Find here detailed descriptive documentation about aura-bridge plugins.

📃 Find here the guidelines for the generation of a plugin and activation in aura-bridge.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

ATRIA Model Gateway architecture and components

Development architecture and technical components of the atria-model-gateway

Technical foundations

atria-model-gateway is responsible for managing the communication with different AI models. This component receives a request from aura-gateway-api, together with other input data, and makes a call the corresponding AI models.

If the selected AI model is RAG, then atria-model-gateway calls the atria-rag-server, which is in charge of executing the RAG chain and making the corresponding calls to the LLM models and databases.

Functional components

The functional components of atria-model-gateway are described in the document LLM/LMM Experiences Builder

Architecture overview

The following diagram schematically shows the main technical components integrated into atria-model-gateway.

A brief description of these components is included below:

Access module

Module for the management of different profiles to access atria-model-gateway.

Context module

Module in charge of the storage of a conversation history in a cache (currently, Redis is used) over a period of time, grouped by session ID. These conversations are taken into account when calling the generative LLM models.

Model manager

Module that includes the available models and presets. It is in charge of receiving the info from aura-gateway-api and calling the corresponding model.

Models

Available AI models integrated into the atria-model-gateway.

Presets

Presets are configurable entities to define the specific model to work with and certain parameters associated to it: model Id, name, description, model parameters, etc.

Constructors can use the default presets or build new ones: Go to document ATRIA configuration.

When configuring an application, all the presets that can be used for this application must be previously defined.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

ATRIA RAG Generate DB architecture and components

Development architecture and technical components of the atria-rag-generate-db

Architecture overview

The following diagram schematically shows the main technical components integrated into atria-rag-generate-db.

A brief description of the technical components is included below:

Data sources

A project contains information required for the execution of the generation of the databases: specific path of documents to feed the databases, allowed file extensions, etc. It can read from different sources, this source type is defined in the extensions field.

Before the information from the documents is stored in the corresponding database, the documents are processed, e.g., they are cut up and cleaned.

Retrievers

The retrievers are in charge of reading the information from the documents and feeding the databases.

The retrievers are defined in the retrievers field of the project. Each retriever is associated with a database in order to feed or retrieve information from it.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

ATRIA RAG Server architecture and components

Development architecture and technical components of the atria-rag-server

Architecture overview

The following diagram schematically shows the main technical components integrated into atria-rag-server.

A brief description of the technical components is included below:

Project

A project contains information required for the execution of the RAG pipeline: specific models for semantic search and lexical search; path where the documents to feed the LLMs are located; allowed file extensions, etc.

Semantic search (embeddings)

Qdrant database that stores the embeddings generated through semantic search (OpenAI embeddings) technology.

Lexical search (LLMs)

Database that stores the required documentation for making lexical searching, based on keywords.

Configuration

atria-rag-server includes a default configuration. Constructors can use it as is or they can modify it to be adapted to their requirements or business models: Go to document ATRIA configuration.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Complex Logic Framework architecture and components

Architecture and main components that you should handle when working over Aura Complex Logic Framework

CLF architecture

Aura Complex Logic Framework is a docker container, meaning that it is configured as a lightweight, stand-alone, executable package of software that includes all the required elements to run an application.

The Complex Logic Framework architecture is schematically shown in the following figure, containing different elements described in the succeeding sections.

CLF main components

Sandbox

A sandbox is an isolated testing environment that enables users to run programs or execute files without affecting the application, system or platform on which they run and to test new programming code.

Plugins

Plugins in Aura Complex Logic Framework are components that add a specific feature to an existing computer program, enabling its customization.

The role of plugins within the Complex Logic Framework is to receive data, make the required computations and return an adequate response to the handler.

CLF manages different types of plugins, which are encapsulated in the sandbox. Moreover, it contains several global plugins, but it is possible for developers to add custom ones.

Some key features of CLF plugins are defined below:

Architecture allows each package of plugins to be deployed in its own environment: it is possible to have an isolated container deployed in an independent way containing all the plugins of a specific package (for instance, global plugins).
Each package is isolated by being run as a different process by a unique user with randomized name. The rationale behind this is to provide an isolated sandbox to make sure no package of plugins is a security risk. Thereby, each user has limited privileges and computational resources (memory, file descriptors and CPU time) and only runs one process for package.
The running packages can only receive data from a socket, and the appropriate plugin is then executed depending on both the plugin type and the intent contained in the data. The API handler is in charge of forwarding the data to the package, acting as a proxy, but it also checks the requests and transforms the responses back.

To ease working with this architecture, packages, users and sockets have the same random name. This name also appears in the information and error log files.

OBs can develop their own custom plugins in the Aura Complex Logic Global repository, with each plugin implemented as a class that inherits from PluginBase and overwrites run method. This method receives the request data and contains the logic of the plugin. It must also generate a response that will be returned to the API handler. When a request is performed, depending on the intent, the request data will be forwarded to the corresponding plugin automatically. Note that two intents can trigger the same plugin but it is not possible to trigger two plugins with the same intent.
Each package of plugins is watched by a supervisor. If the package process crashes, it will be restarted automatically by supervisor. The causes of abrupt exit could be excessive consumption of resources (CPU time, memory or file descriptors) or plugin internal malfunction.

Stub

A stub, in the context of distributed computing, is a piece of code that is used to convert parameters during a remote procedure call (RPC). An RPC allows a client computer to remotely call procedures on a server computer.

The parameters used in a function call have to be converted because the client and server computers use different address spaces. Stubs perform this conversion so that the remote server computer perceives the RPC as a local function call.

API server

API server is the process that listens for requests. This API server is based on the cognitive API package and, in this case, is configured to run the Complex Logic module.

HTTP requests will be passed to the package API handlers according to the CLF configuration.

It is also important to mention that API server process is watched by supervisor (meaning that it is fault-tolerant).

API handler

In Aura Complex Logic Framework, API handlers are classes that inherit from BaseComplexLogicAPIHandler. They are responsible for dealing with HTTP requests through the following methods:

Checking the request is valid.
Checking that the response is in the correct format.
Making a new request to the corresponding package process, which contains a group of plugins.

The first two items are implemented by the declaration of schemas that define the query request and the response. Plugin API handlers are only responsible for defining which request schema is accepted and which response is valid (also by using a schema).

The request to a specific package is made automatically by the system as long as the specific intent that summons the trigger is recognized. The configuration is described in plugins configuration.

API plugin

API plugin is an internal component based on the cognitive API package. It exposes the plugins contained in a package.

Kernel Platform connectors

Kernel is the database where Telefónica stores all the data of its customers in a safe and reliable way, allowing users to control their data at the same time.

These data can be consulted through a Kernel API with the necessary permissions. From the Complex Logic Framework, the only way to access Kernel is through our connectors. The connectors are used, for example, to obtain customer invoices or offer the catalog of devices and services.

These connectors provide total security in order to avoid problems of wrong, malicious or not careful use by third party developers. For this purpose, prior to any request to Kernel, a check of the required permits is made.

CLF Sequence diagram

The following sequence diagram shows the functional flow of the Complex Logic Framework, starting with the request from aura-bot to the Complex Logic module.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura infrastructure

Scope: Aura Virtual Assistant and ATRIA infrastructure documentation for DevOps Teams that describes architecture, networks and platform services

Introduction

The current section includes a view of Aura architecture focused on the Aura deployment process.

Aura architecture flowchart for DevOps Teams
Nodepool common: set of nodes with a common configuration and specification
Kubernetes cluster
Aura Platform services

Aura architecture flowchart

Nodepool common

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

DAPR analysis

DAPR (Distributed Application Runtime) is a framework designed to simplify the development of microservices-based applications.

It provides a set of building blocks that help developers address common challenges in distributed applications, such as service communication, state management, event publishing and subscription, and more.

Change motivation

The motivation behind adopting DAPR in our microservices platform is to enhance the development and operational efficiency of distributed applications.

DAPR provides a standardized way to handle common microservices challenges, such as service discovery, state management, and event-driven communication, which can significantly reduce the complexity of building and maintaining microservices architectures.

Summary

Currently, our platform relies on custom implementations for these functionalities, leading to increased development time and potential inconsistencies. By integrating DAPR, we aim to streamline our microservices development process, improve scalability, and enhance the overall reliability of our applications.

Moreover, DAPR would minimize the complexity of new services deployment leaning on features such as automatic service discovery and built-in support for various state stores and message brokers. This will allow our development teams to focus more on business logic rather than infrastructure concerns, ultimately accelerating our time to market and improving the maintainability of our codebase.

Date	Proposal leadership team
June 2025	Aura Engineering team

Advantages of using DAPR in a microservices platform

Complexity abstraction: DAPR abstracts the complexity of communication between microservices, allowing developers to focus on business logic.
Portability: Being language and platform agnostic, DAPR can run in any environment, whether in the cloud or on-premises servers.
Scalability: It facilitates horizontal scalability of services, enabling applications to grow as needed.
Interoperability: DAPR supports multiple programming languages and protocols, making it easier to integrate heterogeneous services.
Security: It provides built-in mechanisms for authentication and authorization between services.
Rich ecosystem: Includes building blocks such as service invocation, state management, event publishing/subscription, and more.

DAPR architecture

DAPR’s architecture is designed to provide a flexible and modular framework for building distributed applications.

It consists of a control plane and a data plane, where the control plane manages configurations and policies, while the data plane handles the actual service interactions.

Architecture diagram

Below is a visual representation of DAPR’s architecture:

graph TD
    classDef graphTitle font-size: 20px, font-weight: bold, fill:#f0f0f0, stroke:#333, stroke-width:2px;
    subgraph Control Plane
        A[Configuration Management]
        B[Policy Enforcement]
        C[Monitoring and Observability]
        D[Service Discovery]
        E[Scaling and Resilience]
    end
        F[Sidecars]
    subgraph Data Plane
        G[Service Invocation]
        H[State Management]
        I[Publish and Subscribe]
        J[Bindings]
    end

    A --> F
    B --> F
    C --> F
    D --> F
    E --> F
    F --> G
    F --> H
    F --> I
    F --> J

This diagram illustrates how the control plane and data plane interact to enable DAPR’s functionalities, ensuring seamless communication and management of distributed applications.

Components of DAPR Architecture

DAPR’s architecture is composed of several key components that work together to enable microservices communication and management:

Sidecars: These are lightweight containers that run alongside each microservice, providing DAPR’s functionalities such as service invocation, state management, and event handling.
Building blocks: DAPR offers modular building blocks for common tasks, including:
- Service invocation: Enables direct communication between services.
- State management: Provides distributed state storage.
- Publish and subscribe: Facilitates event-driven communication.
- Bindings: Connects services to external systems like databases and message queues.
Control plane: Manages the configuration, security, and monitoring of DAPR instances across the platform.
Data plane: Handles the actual runtime operations, including service-to-service communication and data processing.
APIs: DAPR exposes APIs that developers can use to interact with its features programmatically.

These components collectively simplify the development and operation of distributed applications, ensuring scalability, portability, and ease of integration.

Sidecars in DAPR

In the context of DAPR, a “sidecar” is a container that runs alongside each microservice on the same machine or pod. This container acts as an intermediary that provides DAPR capabilities to the microservice. Sidecars enable:

Service communication: Facilitate service invocation and message transmission.
State management: Provide distributed state storage.
Event publishing and subscription: Allow services to communicate through events.
Integration with external resources: Simplify connections to databases, message queues, and other external services.

Sequence diagram: communication flows in DAPR

Below is a simplified sequence diagram illustrating how communication works in DAPR:

sequenceDiagram
    participant ServiceA
    participant DAPR-SidecarA
    participant DAPR-SidecarB
    participant ServiceB

    ServiceA->>DAPR-SidecarA: Invoke ServiceB
    DAPR-SidecarA->>DAPR-SidecarB: Forward request
    DAPR-SidecarB->>ServiceB: Deliver request
    ServiceB->>DAPR-SidecarB: Respond
    DAPR-SidecarB->>DAPR-SidecarA: Forward response
    DAPR-SidecarA->>ServiceA: Deliver response

This diagram demonstrates how DAPR sidecars facilitate communication between services, ensuring seamless interaction and abstraction of complexity.

Events in DAPR

DAPR uses a publish-and-subscribe model to manage events. This model allows services to communicate asynchronously and decoupled. Events in DAPR have the following characteristics:

Decoupling: Event producers and consumers do not need to know each other.
Scalability: The system can handle large volumes of events without affecting performance.
Compatibility with multiple brokers: DAPR supports various messaging systems such as Kafka, RabbitMQ, Azure Service Bus, among others.
Ease of use: Developers can define their subscriptions and publications easily through configurations.

State management with Redis

DAPR supports state management by providing a distributed state store that can be backed by various storage solutions, including Redis. Redis is a popular in-memory data structure store that is often used for caching, session management, and real-time analytics.

How DAPR uses Redis for state management

Configuration: Developers can configure DAPR to use Redis as the state store by specifying Redis connection details in the DAPR configuration file.
Key-value storage: Redis allows DAPR to store and retrieve state in a key-value format, making it simple and efficient.
Persistence: While Redis is primarily an in-memory store, it can persist data to disk for durability.
Scalability: Redis can be scaled horizontally to handle large amounts of state data across distributed systems.

Example use case

Imagine a shopping cart application where user session data needs to be stored temporarily. DAPR can use Redis to store the cart state, ensuring fast access and updates:

Write state: When a user adds an item to the cart, DAPR writes the state to Redis.
Read state: When the user views the cart, DAPR retrieves the state from Redis.
Delete state: When the user checks out, DAPR deletes the state from Redis.

By leveraging Redis, DAPR ensures efficient and reliable state management for distributed applications.

Control plane in DAPR

The control plane in DAPR is responsible for managing the overall configuration, security, and monitoring of the runtime environment. It ensures that the distributed application operates smoothly and adheres to defined policies.

Key responsibilities of the control plane

Configuration management: The control plane handles the configuration of DAPR components, including sidecars, building blocks, and integrations with external systems.
Policy enforcement: It ensures that security policies, resource limits, and other operational rules are applied consistently across the platform.
Monitoring and observability: The control plane provides tools and interfaces for monitoring the health and performance of DAPR instances and microservices.
Service discovery: It facilitates the discovery of services within the distributed application, enabling seamless communication between microservices.
Scaling and resilience: The control plane supports scaling operations and ensures resilience by managing failover and recovery mechanisms.

By centralizing these responsibilities, the control plane simplifies the management of distributed applications and enhances their reliability and security.

Cost implications of DAPR

When adopting DAPR, it is important to consider the cost implications of its components:

Sidecars

Lightweight and efficient: DAPR sidecars are designed to be lightweight, consuming minimal resources. They run alongside microservices and provide essential functionalities without adding significant overhead.
Almost free: The resource usage of sidecars is negligible in most scenarios, making them an economical choice for enabling DAPR features.

Control plane

Resource intensive: Unlike sidecars, the control plane requires dedicated resources to manage configurations, policies, monitoring, and scaling operations.
Operational costs: The control plane introduces additional costs related to infrastructure, maintenance, and monitoring tools.
Scalability considerations: As the platform scales, the control plane may need to be expanded to handle increased workloads, further adding to the costs. This would mean deploying new nodes in the environment, that would also mean extra charges.

The overall cost of the control plane is based on the number of nodes needed to support it. As an estimation, it would need adding 1 extra node to the current environment in the case of the production environment in Spain and 2 extra nodes in the case of the production environment in Brazil. Assuming the cost of each node of the family D4as v4 is around 150 € per month (this is the raw price shown publicly in Microsoft pricing page, without any discounts and for west europe region), the monthly cost for the control plane would be approximately:

Spain: 1 nodes * 150 € = 150 € per month
Brazil: 2 nodes * 150 € = 300 € per month

Prices in June 2025 ¹

Conclusion

DAPR is a powerful tool for building distributed applications based on microservices. Its focus on simplicity, portability, and scalability makes it an ideal choice for developers looking to optimize their microservices platforms. Sidecars and the event model are key components that facilitate communication and integration between services, making applications more robust and easier to maintain.

https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/?cdn=disable#pricing ↩︎

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura development architecture

Description of the Aura Virtual Assistant development architecture, its operation and main components

Introduction

In Aura distributed architecture, the user interacts directly with the root bot, here named aura-groot, and the root bot delegates some of its conversational logic to the skill aura-bot.

Skills and root bot communicate over HTTP using the Bot Framework protocol.

Both aura-bot and aura-groot are separate bots and are published independently.

Main technical components

The key Aura technical components that executes all the functionalities of Aura Virtual Assistant are defined below.

Take into account the mapping between the functional architecture with development architecture:

aura-groot
aura-groot is the main component in charge of handling the assistant. It is responsible for the communication between each channel and its corresponding skill, keeping track of all the communication process, but introducing a minimal interference on it.

Access detailed technical documentation regarding aura-groot component
aura-bot
aura-bot is a skill based on Microsoft Bot Framework that constitutes Aura’s neuronal network. It handles the requests received from aura-groot and interacts with every Aura system or component in order to provide Aura users with the intended answer or requested service.

Access detailed technical documentation regarding aura-bot component

Operational flow

The main stages in tasks in Aura Virtual Assistant conversational process are described below and shown in the following flowchart.

aura-groot adapter receives activities from the user and forwards them to the aura-groot activity handler. (Activities from the user are received at the aura-groot messaging endpoint.)
The aura-groot uses a skill HTTP client to send an activity to the skill. The client gets the consumer-skill conversation information from a skill definition and a skill conversation ID factory. This includes the service URL that the skill will use to reply to the activity.
The aura-bot adapter receives activities from aura-groot and forwards them to the skill’s activity handler. (Activities from the consumer are received at the skill bot’s messaging endpoint).
When aura-bot responds, the aura-groots skill handler receives the activity. It gets the root-user conversation information from the skill conversation ID factory. It then forwards the activity to the aura-groot’s adapter. (Activities from the skill are received at the aura-groot’s skill host endpoint).
The aura-groot adapter sends messages from the skill to the user.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Analytics 2.0.0. architecture

Technical architecture of Aura Analytics 2.0.0 and description of main processes and components

Architecture overview

Aura Analytics 2.0.0 contains two different environments:

OB local environment: Processes in this side are managed by the OB, that should install and execute certain processes related to the PPD-Creator for the creation of raw datasets.
Global environment: Processes here are managed by Aura Global Team for data recovery, processing and generation of dashboards and statistics. The output includes data and metrics to be consumed by Aura Global Team for decision-making.

Aura Analytics 2.0.0 architecture flowchart

The following diagram shows an overview of Aura Analytics 2.0.0 architecture, including the environments involved and the main components and processes, which are fully described in succeeding sections.

Figure 1. Aura Analytics 2.0.0 Architecture flowchart

Aura Analytics 2.0.0 processes

PPD-Creator process

The PPD-Creator is a Python module for the creation of PPD-Raw datasets.

It is the only component that belongs to the OB environment. The OB should install it and is responsible for its execution. The PPD-Raw datasets will be stored in the destination blob PPD-RAW.

This process reads the files included in OB MANAGED INSTANCES columns of the tables in Annex: Dataset fields. The result of this process is the PPD RAW columns of the tables in the above-mentioned annex.

The main tasks executed by the PPD-Creator are summarized below:

Reads the Aura log files in a Kernel Blob.
Anonymize the sensible fields (AuraID, AuraGlobalID, and personal information of user sentence such as DNI, phone numbers, etc).
Save the anonymized files to another directory of blob (PPD-Raw).

Figure 2. PPD-Creator process

The PPD-Creator anonymizes the following data, in the different OBs:

ES	UK
dni	creditcard
nie	insurance
phone	postcode
email	imei
	phone
	imsi
	email
	twitter
	passport

Manage PPD-Raw process

The Manage PPD-Raw process inserts the PPD-Raw path files (output from PPD-Creator) to PostgreSQL table for files management data centric:

It reads the output of PPD-Creator JSON file
Afterwards, it saves the paths to PostgreSQL server

Figure 3. Manage PPD-Raw process

PPD-Clean process

The PPD-Clean is a Python package used to clean PPD-Raw datasets.

Firstly, this process locates the directory where the PPD-Raw files are located, reads the corresponding files and processes them.

Once the process is finished, it writes to the files_processed table in the database and saves them in the PPD-Clean directory.

The main tasks executed by the PPD-Clean are summarized below:

Apply transformations to columns
Extract the explicit frustration
Calculates the Nones n-grams
Save the data in Directory or blob, PostgreSQL server and ElasticSearch for visualization

Figure 4. PPD-Clean process

User Dynamics process

User dynamics is a script used to measure the user’s behavior through metrics. It extracts statistics on the recurrence of users in Aura in a monthly basis.

The processes executed are summarized below:

User dynamics reads the file_processed table of the database and the all PPD-Clean files stored for 1 month.
It extracts metrics regarding new users, recurrent users, lost users and recovered users.
Afterwards, it saves these metrics in the User_dynamics schema, in a PostgreSQL database, within the tables connections, daycount, user and channel.
Data is also saved in ElasticSearch.

Figure 5. User Dynamics process

Components

Active Listening Database

The Active Listening Database is a PostgreSQL database that stores the processed and to-process files centrally in one place. It is used by the PPD-Clean and User Dynamics processes to store the processed data and metrics.

Schema files management

Currently, in the Active Listening project, we have input and output files for each of the processes and files that are processed. With the proposed database solution through the files management database, a more efficient management of raw files is achieved:

The PPD-Creator process transfers files from the OB to a shared blob.
The transferred files are written to a file in that blob called aura-sync-cache-dst.json.
The manage_ppd_raw process will read the aura-sync-cache-dst.json file from the PPD-Raw folder and ingest the records into the FILE_PPD_RAW table of the database.
It will also insert into the EMPTY_DATA_FILES table the days that are not found in the JSON file. This table is necessary for logging metrics in Prometheus. This process will run daily.

Figure 6. Files management database

Schema user dynamics

The User Dynamics process generates the statistics of Aura users, number of daily active users and types of users, with 4 categories: new, recurring, lost and recovered.

The Channel table contains all the channels in Aura that have been processed by the User Dynamics process.
The User table contains the unique Aura users in each environment and country.
The Daycount table contains the number of total users for each day, indicating how many of them are new, recurring, recovered or lost users, the number of weekly unique recurring users and the number of monthly unique recurring users.
The Connection table has the status of the user for each day (whether it is new, recurring, lost or recovered).

Figure 7. User dynamics database

Aura Analytics Dashboard

Aura Analytics 2.0.0 produces as a result, among other elements, an analytics component named Aura Analytics Dashboard that is the one used by Aura Global Team to gather statistics on the production system and to analyze user’s behavior.

This Analytics Dashboard is based on the ELK stack that contains:

ElasticSearch: distributed search and analytics engine at the heart of the Elastic Stack. It allows the storage of data and its subsequent indexing, search and analysis.
Kibana: provides a visualization tool that includes dashboards and panels created over the ElasticSearch data. Users interactively explore, visualize and share insights into data and manage and monitor the stack.

Once installed:

An ElasticSearch index is created. It is called aura-ppd-ENTITY-COUNTRY-YEAR, and its index schema contains a cleaned version of the AURA MESSAGE, RECOGNIZER or API tables (which registers input and output messages).
A Kibana index pattern is created, matching the uploaded ElasticSearch index.
A pre-defined set of visualizations are installed in Kibana over that index pattern, as a means to get a default peek on the index data. See the section pre-installed analytics dashboard.
The system automatically ingests any new clean PPD being produced in the ElasticSearch database, so that the index and dashboards remain up to date.

In principle, the PPD creation process specifies daily production, since Aura logs are sent to Kernel once a day. This means that information about Aura behavior and user actions on one given day will be available in the dashboards of the following day.

As mentioned above, the Aura Analytics Dashboard is conceived to be used by Aura Global Team. However, OBs can install locally the ELK stack or any other visualization tool for data consumption. Access to the document Local data visualization for further details.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Analytics 1.1. architecture

Technical architecture of Aura Analytics 1.1.

Architecture description

The following figure shows a full overview of Aura Analytics Dashboard architecture and operation, which is also described below:

 Aura logs generated in local instance are converted to datasets and transferred to local Kernel via the standard procedure and with the established frequency (typically, daily).
 Once there, the “Active listening” process flow fires up daily. Through a specialized process that runs on an Aura local instance and with access to the stored datasets in the Kernel local storage space:
- PII (Personally Identifiable Information) is removed or encrypted.
- The result is transferred to a bucket/blob set up for this task and managed by Global Aura team.
- Here, the PPDs (Privacy-Preserving Datasets) are created. Currently, only MESSAGE, RECOGNIZER and API datasets are involved in this process.
In order to convert PII data to PPD, every field in these datasets can be:
- a. Not transferred.
- b. Pseudo-anonymized. In this situation, the field is transformed through a cryptographic hashing process using a secret set up by the OB.
- c. Anonymized fragments of the field (e.g., credit card number, email, etc.). The field is processed to detect specific patterns and replaces them with a specific tag (idemail, idpassport, etc.). The list of anonymization strings is agreed with each OB.
- d. Transferred as is.
 After that, the Raw PPD Datasets stored in bucket/blog managed by the Global Team are processed generating clean PPD Datasets in order to adapt them to the analytics tools.
 From that space, the clean PPD Datasets can be:

Accessed by the Aura Global Team that use them for several tasks, with the purpose of evaluating Aura quality and taking the best decisions regarding to product evolution:
- Perform analytics on Aura behavior and prototype Analytics Dashboard features
- Improve Aura Platform capabilities (e.g., adapting machine learning models)
Accessed by a Local Aura Team, ingesting the data to a dedicated server managed by the OB with analytics and data visualization capabilities. In order to do that, the Aura Global Team provides a component with the ELK (elasticsearch, logstash & kibana) preconfigured with a set of dashboards that can be deployed and adapted by the OB team.

All the code involved in this process can be found in Github. Particularly:

PPD RAW creation package
Conversion from PPD RAW to PPD Clean
Pseudo-anonymization function for identifiers
Utterance anonymization (agreed individually for ES and UK)

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Agents Manager architecture and components

Development architecture and technical components of the agents-manager

Technical foundations

agents-manager is mainly a web server built on Typescript using nodejs as engine. It is api-first designed, using OpenAPI v3 to provide the API definition and openapi-backend to handle swagger specification.

agents-manager server is composed by several plugins, which provide different functionalities to this component.

A channel, service, or skill uses an application to connect with agents-manager following this communication protocol.

Architecture overview

The following figure shows the main technical components of the agents-manager, which are described below.

agents-manager components

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a configuration file or environment variables.

RedisConnector

RedisConnector is a handler connection to Redis.

HTTP server

Microservice is implemented as an HTTP server (AuraServer) that exposes an API to receive the request to be validated.

Middlewares

The route published in the API definition file is handled by a controller but, before a request lands on its controller, it goes through a series of middlewares that provides some common steps needed by all the controllers of the server such as: request validation, common parameters extraction, logging, metrics initialization, etc.

Controllers and Services

agents-manager is composed of plugins, which provide functionality and uses specific modules from aura-configuration-api.

Check the available plugins together with detailed information in: agents-manager plugins.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Agents Server architecture and components

Development architecture and technical components of the agents-server

Technical foundations

The agents-server component will start a server that listens for incoming requests and executes the agent’s tasks based on the received input.

The server can be configured to use an agent by specifying the agent package name.

This server creates a REST API that can be used to interact with different agents. The API allows sending requests to the agent and receiving responses.

Within the agents-server, the agents package component enables information processing tasks to be performed and acted upon to achieve specific objectives. This information can come from a database and respond to the user’s request based on that acquired information.

Architecture overview

The following figure shows the main technical components of the agents-server, which are described below.

agents-server components

HTTP server

Microservice is implemented as an HTTP server (fastapi) that exposes an API to receive the request to be validated.

Config Api Handler

The agents-server uses the aura-configuration-api to get the configuration of the agent. This handler is responsible for retrieving the configuration of the agent from the aura-configuration-api and providing it to the server.

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a configuration file or environment variables.

Agents Package

The server is configured to use the agent of a specific agent package name. This package contains the agent’s code and its dependencies. The server will load the agent package and use it to execute the agent’s tasks.

Event Subscribers

Event subscribers are implementations for managing event registration and handling in the agents-server. They allow the server to listen for events and trigger actions based on those events.

Redis Async Subscriber

RedisAsyncSubscriber is a handler for subscribing to Redis channels asynchronously. It allows the server to receive messages from Redis and process them in real time.

DAPR Async Subscriber

DaprAsyncSubscriber is a handler for subscribing to DAPR topics asynchronously. It allows the server to receive messages from DAPR and process them in real time.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Gateway API architecture and components

Development architecture and technical components of Aura Gateway API

Technical foundations

aura-gateway-api is mainly a web server built on Typescript 4.3 using nodejs as engine. It is api-first designed, using OpenAPI v3 to provide the API definition and openapi-backend to handle swagger specification.

aura-gateway-api server is composed by several plugins, which provide different functionalities to this component. From the different capabilities that aura-gateway-api can manage, certain plugins are used by all of them and others are specific of one capability.

A channel, service, or skill uses an application to connect with aura-gateway-api following this communication protocol.

Architecture overview

The following figure shows the main technical components of the aura-gateway-api.

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a configuration file or environment variables.

RedisConnector

RedisConnector is a handler connection to redis.

Kpis-handler

KpisHandler is the module responsible for writing the KPIs entities.

HTTP server

Microservice is implemented as an HTTP server (AuraGatewayServer) that exposes an API to receive the request to be validated.

Middlewares

Plugins

Different plugins provide functionality to aura-gateway-api.

Check the available ones together with detailed information in: aura-gateway-api plugins.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Context architecture and operation

Discover Aura Context functional architecture and its operational process

Functional architecture

Aura Context v1 functional architecture contains the following components:

Context broker & server API
Server API in Aura Context for uploading, updating and recovering context data by different Aura sub-systems.
Aura Context Service
Core module of Aura Context in charge of gathering data from the context API, uploading it into de context database and data recovery.
Aura Context cache/database
Temporary high-performance database placed inside the Aura instance that is able to storage user’s data corresponding to the user’s previous experiences or environment. Data is available in the context database for a limited period of time and it cannot be recovered once this period expires.
The key features of Aura Context database are shown below:
- 360 approach: Aura Context aims to gather all type of information from an Aura interaction, including data coming from channels.
- High-available database: real-time recovery during the user’s interaction, with no lateness.
- Temporary data storage: data available only for an established period of time.
- Data persistence in Kernel for historical data exploitation purposes.
Context Kernel persistence mechanism
First basic version of the Kernel persistence mechanism. Within a one-direction flow, context data stored in the database is sent to Kernel for an unlimited period of time. Data on Kernel can be checked or processed following the same procedure as for Aura data.

Aura Context performance

The following steps schematically shows the process carried out by Aura Context:

Context data is sent to Kernel where it is stored for an unlimited period of time.
This information can be retrieved by an external stakeholder just by calling the corresponding API.
Developers can consume data in the Aura Context database when developing a use case. For this purpose, the aura-bot dialog must call the corresponding API in order to retrieve context data and include it in the response to the user.

Aura Context operational flowchart

Before facing this section, it is recommended to read the document Aura Context components.

Considering the Aura Context components, the following flowchart shows the operational process carried out by the context:

An external component makes a request to Aura Context API (POST HTTP request) for carrying out a specific operation:
- Create: creation of a data entry in Aura Context
- Upsert: modification of a data entry in Aura Context
- Fetch: data recovery from Aura Context
The context API sends the request to Aura Context service.
The request transform converts input data into data suitable with the Aura Context global data model.
The specific operation is performed: Create / Upsert / Fetch.
Output data from the operation is converted into data compatible with the user’s data model through the response transform.
The response is sent back to the context API and provided to the external component.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Groot

aura-groot is the router bot that manages the behavior of Aura Root ecosystem functionalities acting as a router for the request to a specific skill.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

aura-groot is the main component in charge of handling the distributed architecture of Aura. It is responsible for the communication between each channel and its corresponding skill, keeping track of all the communication process, but introducing a minimal interference on it.

aura-groot main role includes:

It is the component in charge of making the decision about what is the best bot to answer the user’s request and routing the request to the corresponding skill (bot).
It is able to map configurable input parameters such as the channel that sent the request, user type, previous requests stored in the context, utterance, etc. with a specific domain. The OB will be able to configure the parameters used for the routing process.

Currently, routing is done only by channels or group of channels (no configuration options are available).

Aura Groot components

aura-groot contains two main components:

Middlewares: software components in charge of the message flow.
Recognizers: specific type of middlewares that are executed in a certain stage of the message flow.

Access to the corresponding sections to find detailed information regarding these components.

Aura Groot architecture

The first diagram shows all the components running in aura-groot.

The following diagram includes the messages flow through aura-groot.

Aura Groot configuration and operation

Find out the environment variables included in the aura-groot component here.

The operation of aura-groot is described in the document Aura Groot operational flows.

ConversationId handling

aura-groot uses groot-channel conversation identifier to track the conversation with the user and the use case with the skill. It is a string generated by the channel and is unique for each conversation. The groot-channel is sent in the incoming request to aura-groot in the conversationId field of the message object.

Additionally, it adds conversationReference inside. In this way, aura-groot only has to manage a single conversation and avoids synchronizing two different conversations. Doing so, accesses from aura-groot to the database are reduced.

This change implies a modification in the process the BotFramework works with a conversation, in two different documents which are updated separately. But, on the contrary, the advantage is a most efficient and secure way of working.

sequenceDiagram
title: BEFORE-DIRE-STRAITS

    actor Channel
    participant AuraGroot #ebdff7
    participant AuraBot #ebdff7

    Channel ->> AuraGroot: request message1
    AuraGroot ->> AuraGroot: create targetSkill in groot-channel
    AuraGroot ->> AuraGroot: save conversationReference in groot-channel
    AuraGroot ->> AuraBot: request message1
    AuraBot -->> AuraGroot: response message1
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot -->> Channel: response message1
    AuraBot -->> AuraGroot: response endOfConversation
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot ->> AuraGroot: delete targetSkill

    Channel ->> AuraGroot: request message2
    AuraGroot ->> AuraGroot: create targetSkill in groot-channel
    AuraGroot ->> AuraGroot: save conversationReference in groot-channel
    AuraGroot ->> AuraBot: request message2
    AuraBot -->> AuraGroot: response message2
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot -->> Channel: response message2
    AuraBot -->> AuraGroot: response endOfConversation
    AuraGroot ->> AuraGroot: load conversationReference in groot-channel
    AuraGroot ->> AuraGroot: delete targetSkill

Aura Groot endpoints

Path	Method	Description
/api/messages/	POST	Incoming requests of channels or aura-bridge
/refresh/	POST	Update skills and channels. Data: `{"changes": ["skills", "channels"]}`. You can use “skills” and/or “channels” or neither of them. In the last case, all will be updated.
/api/skills/*	POST	Endpoints of Direct Line API for skills can exchange messages with aura-groot. The path where the skills will return the answers is configured in the environment variable `AURA_SKILLS_RESPONSE_PATH`, by default, `/api/skills`.
/healthz	GET	Monitoring endpoint. Healthcheck endpoint to validate if the server is up and running, only available from local network
/shutdown	GET	Monitoring endpoint. Wait until shutdown (Kubernetes lifecycle).
/heapSnapshot	GET	Endpoint that must be called internally directly accessing to the aura-groot pod. It stores in Aura Azure Storage Account a copy of the heap memory of the pod. It is useful to debug memory leaks.
/heapStatistics	GET	Endpoint that must be called internally directly accessing to the aura-groot pod. It returns the heap memory statistics of the pod.

These endpoints do not need to be defined in the plugin swagger, since they are supplied by the aura-groot core. Specific routes are defined in https://github.com/Telefonica/aura-distributed-bot/blob/main/master/packages/aura-groot/src/modules/server/routes.ts.

Postman collection

Aura Platform Team has generated a Postman collection for simulations over Aura distributed architecture: Aura distributed Postman collection.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes cluster

Description of the Kubernetes cluster: components, connection, automation, storage and more

Introduction

The Kubernetes cluster is created on the cloud using aks-engine (Azure), the tool provided by Microsoft to generate the Azure Resource Manager templates that deploy Kubernetes cluster. This tool simplifies the cluster installation, management and upgrades.

Kubernetes are internally used by the Aura Platform installer but they are not intended to be directly used by the Operations and Support Teams. The reason is that changes made to a Kubernetes cluster that are not reflected in the Aura Platform deployment profile could be lost in a platform upgrade. For example, if you changed the instance type in a specific agent pool with Kops, you would lose the changes in a new deployment.

Cluster metadata

The cluster metadata is saved in the object storage (Azure Blob Storage).

⚠️ Remember that if you lose this bucket, you need to rebuild the cluster manually (Azure). It also contains critical information to access the cluster (certificates, private keys), so it must not be shared.

Connecting to Kubernetes

The kubectl command-line tool is the basic tool to operate the Kubernetes cluster. It uses kubeconfig files that contain all the required information (endpoints, certificates, etc) to connect with the Kubernetes API and manage the cluster in a secure way.

A default (root) kubeconfig file with full access to the cluster is created the first time you install the Aura Platform and is stored in a bucket as part of the cluster metadata.

⚠️Do NOT use the default kubeconfig file to manage the cluster. Use it to create new users with limited permissions instead.

By default, kubectl looks for a file named config in the $HOME/.kube directory. However, you can specify other kubeconfig files by setting the KUBECONFIG environment variable or passing the flag --kubeconfig to kubectl.

export KUBECONFIG=/path/to/kubeconfig.json  # Azure

⚠️ For security reasons, kubeconfig files are personal and must not be shared. Each action a user executes on the cluster is logged. If your kubeconfig file is compromised, you must report it.

📄 You can get more information about kubeconfig files at the official Kubernetes documentation.

⚠️ Kubernetes-dashboard is not deployed in Aura cluster as we promote the kubectl use, but if you still want to use it, you can run the dashboard in your machine and connect to the cluster with your kubeconfig as follows:

docker run -p 9090:9090 -v ${PATH_TO_YOUR_KUBECONFIG}/kubeconfig.json:/opt/kubeconfig.json -v /tmp:/tmp kubernetesui/dashboard:v2.2.0 --kubeconfig /opt/kubeconfig.json

Kubernetes automation with the Cloud

Kubernetes automates certain tasks such as creating a Load Balancer for a service or mounting a disk on a node as a persistent volume for a pod.

Authentication against the cloud provider APIs is done using credentials configured automatically:

Azure: by using a Service Principal with contributor role and scope for the infrastructure Resource Group.

⚠️ Please do not change the password, delete the Service Principal or remove its Contributor role If doing so, credentials will not be automatically updated due to a known limitation of aks-engine.
See the issue Azure/aks-engine/724 in the aks-engine repository for more information.

The “Security” section describes an unofficial procedure to change the Service Principal credentials in Azure that involves a period of service disruption.

Kubernetes Namespaces

Kubernates Namespaces are a way to divide and organize cluster resources.

You can list the existing namespaces running kubectl get namespaces. In Aura Platform, you will find the following ones:

Namespaces used by the Aura Platform services:
- aura-$ENV: Aura Platform core services.
- aura-system: Aura Platform system services (prometheus, alertmanager, node-exporter, fluentd, elasticsearch, kube-static-metrics).
Namespaces used by Kubernetes:
- kube-system: for objects created by the Kubernetes system. Aura Platform also deploys some objects into this namespace that are very tied to the infrastructure.
- kube-public: readable by all users. It should be empty.
Default namespace (default): for objects with no other namespace.

In most situations, you will need to use aura-$ENV. You can use the --namespace flag in kubectl to specify which namespace you are referring to. For example, to get the pods in the Aura Platform core:

$ kubectl get pods --namespace aura-$ENV

Remember that some low-level resources, such as nodes and persistent volumes, are not in a namespace.

Kubernetes objects

Working with pods

You can list all pods in a given namespace along with additional metadata (the node where the pod is allocated, its age, etc.):

$ kubectl get pods -n aura-$ENV -o wide
NAME                                 READY   STATUS      RESTARTS   AGE
aog-bridge-744bbb9595-94g7n          1/1     Running     0          4h15m
aog-bridge-744bbb9595-pzg2l          1/1     Running     0          4h15m
api-gw-5c584b4c8d-hdk25              1/1     Running     0          4h15m
api-gw-5c584b4c8d-knm27              1/1     Running     0          4h15m
aura-bot-84bd44dc6d-5jdzk            1/1     Running     0          4h16m
aura-bot-84bd44dc6d-ktz74            1/1     Running     0          4h16m
aura-bot-makeup-4qrbl                0/1     Completed   0          4h16m
authentication-api-b849b6ff9-5t96c   1/1     Running     0          4h16m
authentication-api-b849b6ff9-rwcvm   1/1     Running     0          4h16m
nginx-5fd94584d8-tcpwd               2/2     Running     0          4h15m
nginx-5fd94584d8-z72tf               2/2     Running     0          4h15m
nlp-85b4b446cc-df2zw                 1/1     Running     0          4h16m
nlp-85b4b446cc-s6z5k                 1/1     Running     0          4h16m
nlp-provisioning-zxkk5               0/1     Completed   0          4h16m
user-helper-67b75cb8fc-6vkt9         1/1     Running     0          4h15m
user-helper-67b75cb8fc-td42l         1/1     Running     0          4h15m
web-sdk-5f7654b797-9npzn             1/1     Running     0          4h16m
web-sdk-5f7654b797-mlcrj             1/1     Running     0          4h16m

Pods can have different statuses:

Running
Completed: some pods (e.g., jobs) have a reduced lifespan. They change to completed status when they finish. Kubernetes eventually removes them from the list of pods.
Others

📄 You can get more information about working with pods in the Kubernetes documentation.

Each pod is configured using environment variables. They are a OS-agnostic standard that allows to change the configuration between deployments without changing any code in a very easy way. Sensitive information (e.g., passwords) is configured using Kubernetes secrets.

Working with deployments

A deployment controller provides declarative updates for Pods and ReplicaSets, according to the desired state described in a deployment object.

$ kubectl get deployments

NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
aog-bridge           2/2     2            2           4h16m
api-gw               2/2     2            2           4h16m
aura-bot             2/2     2            2           4h17m
authentication-api   2/2     2            2           4h16m
nginx                2/2     2            2           4h16m
nlp                  2/2     2            2           4h16m
user-helper          2/2     2            2           4h16m
web-sdk              2/2     2            2           4h16m

ℹ️ You can get more information about working with deployments in the Kubernetes documentation.

Working with nodes

Nodes are the virtual machines that run Aura Platform.

There are two types of nodes in any Kubernetes cluster:

Master nodes: they host the control plane aspects of the cluster. Typically, these nodes are not used to schedule application workloads.
Compute nodes: nodes which are responsible for executing workloads for the platform services.

If there is an issue in the cluster, the first thing you should review is the status of the nodes.

$ kubectl get nodes

NAME                                 STATUS   ROLES    AGE     VERSION
k8s-common-26582301-vmss000000       Ready    agent    7h33m   v1.13.4
k8s-common-26582301-vmss000001       Ready    agent    7h33m   v1.13.4
k8s-database-26582301-vmss000000     Ready    agent    7h33m   v1.13.4
k8s-database-26582301-vmss000001     Ready    agent    7h33m   v1.13.4
k8s-database-26582301-vmss000002     Ready    agent    7h33m   v1.13.4
k8s-management-26582301-vmss000000   Ready    agent    7h33m   v1.13.4
k8s-management-26582301-vmss000001   Ready    agent    7h33m   v1.13.4
k8s-management-26582301-vmss000002   Ready    agent    7h33m   v1.13.4
k8s-master-26582301-0                Ready    master   7h33m   v1.13.4
k8s-master-26582301-1                Ready    master   7h33m   v1.13.4
k8s-master-26582301-2                Ready    master   7h32m   v1.13.4

You should see three nodes with the role “master”, that is the default recommended value. An odd number of master nodes is mandatory to guarantee that etcd, the service that stores the cluster status, reaches its consensus (see “Why an odd number of cluster members” in etcd frequently asked questions).

The normal status for a node is “Ready”, that means that the node is up and a healthy member of the Kubernetes cluster.

Nodes can become “NotReady” for different reasons when something is not right. In this case, the first step is to describe the affected node and check the “Conditions” and “Events” to determine what could be wrong:

$ kubectl describe node k8s-common-26582301-vmss000000

...
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 04 Jul 2019 07:08:50 +0200   Thu, 04 Jul 2019 07:08:50 +0200   RouteCreated                 RouteController created a route
  MemoryPressure       False   Thu, 04 Jul 2019 14:41:10 +0200   Thu, 04 Jul 2019 07:07:06 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 04 Jul 2019 14:41:10 +0200   Thu, 04 Jul 2019 07:07:06 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 04 Jul 2019 14:41:10 +0200   Thu, 04 Jul 2019 07:07:06 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 04 Jul 2019 14:41:10 +0200   Thu, 04 Jul 2019 07:07:20 +0200   KubeletReady                 kubelet is posting ready status. AppArmor enabled
...

With kubectl top nodes, you can get a quick overview of each node CPU and memory usage. If it is not enough and you need more details about the usage of resources, go to Grafana dashboards section in Monitor Aura documentation.

If a node is misbehaving, the recommended steps are:

Drain the Kubernetes node. This way, Kubernetes give the pods running on that node a chance to stop in an orderly way and stops scheduling new pods on it. This step is not mandatory, but recommended.
Terminate the node. It is always safe to terminate one node at a time, waiting until it joins the cluster. Terminating many nodes at a time can affect the quorum of services that need to form a cluster, so it is not recommended if the node you want to terminate contains these kind of pods.

Nodes that are cordoned appear with the status “SchedulingDisabled”:

k8s-common-26582301-vmss000000      Ready,SchedulingDisabled   agent    ...

Filtering nodes with kubectl is very handy. For example, you can filter nodes to get those in a specific agent pool:

$ kubectl get nodes -l 'agentpool in (common)'
NAME                             STATUS   ROLES   AGE     VERSION
k8s-common-26582301-vmss000000   Ready    agent   7h36m   v1.13.4
k8s-common-26582301-vmss000001   Ready    agent   7h36m   v1.13.4

Or to get only those in a specific availability zone:

$ kubectl get nodes -l 'failure-domain.beta.kubernetes.io/zone in (0)'
NAME                                 STATUS   ROLES    AGE     VERSION
k8s-common-26582301-vmss000000       Ready    agent    4h14m   v1.13.4
k8s-database-26582301-vmss000000     Ready    agent    4h14m   v1.13.4
k8s-management-26582301-vmss000000   Ready    agent    4h14m   v1.13.4
k8s-master-26582301-0                Ready    master   4h14m   v1.13.4
k8s-master-26582301-1                Ready    master   4h14m   v1.13.4

You can find information about the nodes usage in Grafana. Also, describing the nodes gives you information about how Kubernetes allocated resources on it:

$ kubectl describe node k8s-common-26582301-vmss000001

Non-terminated Pods:         (13 in total)
  Namespace                  Name                                     CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                                     ------------  ----------  ---------------  -------------  ---
  aura-es-test               aog-bridge-744bbb9595-ffbzz              100m (5%)     1 (51%)     256Mi (4%)       512Mi (9%)     35m
  aura-es-test               api-gw-5c584b4c8d-qg7bz                  100m (5%)     1 (51%)     256Mi (4%)       512Mi (9%)     35m
  aura-es-test               aura-bot-84bd44dc6d-7cc46                100m (5%)     1 (51%)     256Mi (4%)       512Mi (9%)     36m
  aura-es-test               authentication-api-b849b6ff9-pjdk6       100m (5%)     1 (51%)     256Mi (4%)       512Mi (9%)     35m
  aura-es-test               nginx-5fd94584d8-fwmcd                   600m (31%)    2 (103%)    768Mi (14%)      1Gi (19%)      34m
  aura-es-test               nlp-85b4b446cc-2dtj8                     100m (5%)     1 (51%)     256Mi (4%)       512Mi (9%)     35m
  aura-es-test               user-helper-67b75cb8fc-szmv8             100m (5%)     1 (51%)     256Mi (4%)       512Mi (9%)     35m
  aura-es-test               web-sdk-5f7654b797-jbrfg                 100m (5%)     1 (51%)     256Mi (4%)       512Mi (9%)     35m
  aura-system                fluentd-st7jb                            50m (2%)      100m (5%)   256Mi (4%)       512Mi (9%)     51m
  aura-system                node-exporter-4mlqs                      10m (0%)      50m (2%)    24Mi (0%)        32Mi (0%)      52m
  kube-system                kube-proxy-fnb2d                         100m (5%)     0 (0%)      0 (0%)           0 (0%)         4h14m
  kube-system                kubernetes-dashboard-7947fffdf5-pdrf2    300m (15%)    300m (15%)  150Mi (2%)       150Mi (2%)     4h14m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests      Limits
  --------                       --------      ------
  cpu                            1860m (96%)   10450m (541%)
  memory                         3246Mi (62%)  5814Mi (111%)
  ephemeral-storage              0 (0%)        0 (0%)
  attachable-volumes-azure-disk  0             0

The CPU and memory limits can be over 100%. But the CPU and memory requests cannot. This means that the resources requested by the pods also establish a limit even if they do not use the requested resources.
If all nodes in an agent pool are full, new pods will wait in a “pending” status until the cluster autoscaler adds a new node to the agent pool.

Autoscaling groups

Compute nodes In Azure, we use virtual machine scale set, this means that, when a node is terminated for any reason, another one will be automatically created and each agent pool corresponds to one VMSS in Azure.

Master nodes Master nodes in Azure are individual nodes that do not belong to any VMSS. This means that, if you remove a master node in Azure, you need to run the installer again to recreate it. For this reason, you can restart the node first and wait some minutes to verify if is able to rejoin the Kubernetes cluster. In this case you do not need to terminate it. Remember to uncordon the node if you had cordoned it previously.

Horizontal scaling of a component

Scaling deployments is easy using the kubectl scale command. It enables you to scale one or more replicated services either up or down to the desired number of replicas.

For example, you might want to scale the number of replicas of the apigw deployment to 6:

$ kubectl scale deployment aura-bot --replicas=6 -n aura-$ENV

Deployments contain stateless loads, so they can safely scale up and down. The only restrictions are:

To have only one insights-loader in order to avoid race conditions trying to load insight files from the object storage.
To have only one kube-state-metrics to avoid duplicated metrics in Prometheus.

The platform supports Horizontal Pod Autoscalers.

They have been included in some relevant services to autoscale the number of replicas based the CPU usage of a pod.

$ kubectl get hpa -n aura-$ENV

NAME                 REFERENCE                       TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
aog-bridge           Deployment/aog-bridge           0%/150%   2         4         2          65m
api-gw               Deployment/api-gw               0%/150%   2         4         2          65m
aura-bot             Deployment/aura-bot             0%/150%   2         4         2          66m
authentication-api   Deployment/authentication-api   0%/150%   2         4         2          66m
nginx                Deployment/nginx                0%/150%   2         4         2          65m
nlp                  Deployment/nlp                  0%/150%   2         4         2          65m
user-helper          Deployment/user-helper          0%/150%   2         4         2          65m
web-sdk              Deployment/web-sdk              0%/150%   2         4         2          66m

MINPODS values are set according to your deployment profile using the service_replicas value for each service.
MAXPODS values are stablished as three times the value of MINPODS.
TARGET is defined as a CPU utilization threshold fixed to 80% for all platform services. According to these policies, each service will scale up/down (when needed) based on their own CPU usage metrics. Support for additional custom metrics (e.g., latencies) will be added in future releases.

This feature is very powerful in combination with the cluster autoscaler, because once the pods created by the HPA do not fit in the available compute nodes, the cluster autoscaler will automatically add new nodes to the cluster.

Regarding statefulsets, not all of them scale nicely, so it is really important to understand them well to be able to scale them safely.

$ kubectl get statefulsets --namespace aura-system

NAME                    READY   AGE
alertmanager            2/2     82m
elasticsearch           3/3     83m
fluent-bit-aggregator   3/3     83m
mongodb                 3/3     84m
prometheus              3/3     83m

ℹ️ You have to keep in mind that:

elasticsearch scales well for adding or removing nodes.
fluentd-aggregator scales well for adding or removing nodes.
prometheus stores the same information in all the available replicas, so it is recommended to keep the number of replicas in, at least, 2 for high availability reasons.

Once you are sure about it, use kubectl to scale the statefulset, for example:

$ kubectl scale statefulsets elasticsearch --replicas=7 -n aura-$ENV

Jobs

There are some scheduled jobs that run in Aura Platform. You can check them with kubectl get jobs:

$ kubectl get jobs --namespace aura-es-test

NAME                      COMPLETIONS   DURATION   AGE
aura-bot-makeup           1/1           92s        69m
nlp-provisioning          1/1           10m        68m

Most of them are provisioning jobs that create all the required entities during the installation (applications, APIs, etc).

Horizontal scaling the infrastructure

It is possible to add and remove nodes to the different agent pools in the Kubernetes cluster.

Adding nodes to a running cluster is a safe operation. However, bear in mind that removing nodes can result in statefulsets not working properly. The reason is that some agent pools are dedicated to stateful services that need to form a cluster.

That is the case of the following agent pools:

master: Kubernetes master nodes use a quorum protocol that needs an odd number of nodes. Three nodes is the minimum number to have HA (high availability), so 3 nodes in preproduction and 5 in production environments is a safe choice. The number of master nodes in an existing environment cannot be modified for now.
database: It must be 3, for a PostgreSQL cluster with one master and two followers.

Cluster autoscaler

The Aura Platform Kubernetes cluster deploys the official Kubernetes cluster-autoscaler. It is a deployment with one pod that runs in one of the master nodes.

$ kubectl get po -l app=cluster-autoscaler -n kube-system

NAME                                  READY   STATUS    RESTARTS   AGE
cluster-autoscaler-6fb6b8dcdc-xr59x   1/1     Running   1          4h48m

This feature is intended to automatically adjust the Kubernetes cluster size when one of these conditions are met:

Scale up: there are pending pods that do not fit in the cluster due to insufficient available resources, but could fit if new compute nodes are added.
Scale down: there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

The cluster autoscaler is able to scale the agent pools down to zero if needed. This is the reason why it is not needed to configure the number of nodes in your deployment profile. The cluster autoscaler will take care of everything to cut your cloud costs to the minimum.

ℹ️ You can find more information about the cluster autoscaler in the official GitHub repository.

Vertical scaling the infrastructure

The process is similar to the horizontal scaling. You need to tune the type properties in the infrastructure section of your profile configuration file.

# Infrastructure
infrastructure:
  region: "westeurope"
  compute:
    masters:
      size: 3
      type: "Standard_DS2_v2"
    common_nodes:
      min_size: 2
      max_size: 8
      type: "Standard_DS2_v2"
    database_nodes:
      min_size: 3
      max_size: 6
      type: "Standard_DS2_v2"
    management_nodes:
      min_size: 3
      max_size: 6
      type: "Standard_DS3_v2"

Afterwards, run the installer to apply the changes:

$ ./aura deploy_infra --cfg /PATH/TO/config.yml -c /PATH/TO/credentials.k8s.json -v "VAULT_PASSWD"
$ ./aura deploy_system --cfg /PATH/TO/config.yml -c /PATH/TO/credentials.k8s.json -v "VAULT_PASSWD"
$ ./aura deploy_core --cfg /PATH/TO/config.yml -c /PATH/TO/credentials.k8s.json -v "VAULT_PASSWD"

The operation has to terminate and recreate every node. It has to be done as a rolling update to avoid service disruption, so it can take a lot of time to complete (around 5-10 minutes per node) in a big Kubernetes cluster.

In Azure, it is not possible to change the instance types with the Aura Platform installer yet. This means that changing the instance types in the deployment profile has no effects on redeployments.

⚠️ Do not use the Azure Portal to modify the cluster nodes. It is an error-prone and unsupported way to scale the cluster that could impact the Aura Platform stability. Kubernetes must be aware of the changes done to the cluster and all changes must be kept in sync with the deployment profile.

Kubernetes storage

Kubernetes uses persistent volumes. They are backed by Managed Disks in Azure.

Services logs with kubectl

The best way to access to the logs of one service is using Kibana platform. Find more information in Manage Aura logs.

However, you can access the same way to:


$ kubectl logs -f -l app=aura-bot -n aura-$ENV

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Virtual Assistant development architecture and operation

Documents from this point forward contain low-level technical information regarding Aura Virtual Assistant, aimed at technical profiles.

We highly recommend reading these basic Aura Virtual Assistant documents first to have a clear overview of its foundation, benefits and functional behavior:

Introduction

Aura Virtual Assistant architecture is conceived as a multi-bot system based on Microsoft skills architecture.

In this framework, Aura is divided into isolated modules (skill-based bots) that, currently, are independent domain bots able to provide a specific function or capability in a coordinated and efficient manner.

These self-supported skills work with an orchestrator at the top, responsible for connecting channels with bots and managing the conversational flow with the customer.

Mapping functional architecture with development architecture

As explained before, Aura Virtual Assistant documentation contains both:

Low-level technical documents, aimed at not-technical profiles, that describe the functional behavior of our assistant
High-level technical documents, for highly-skilled technical profiles, that include detailed technical information, system architecture and technical processes

In order to ease comprehension for readers, take as a basis the Aura Virtual Assistant functional flowchart and check the mapping between functional and technical components.

Functional component	Technical component
Channel	Channel
Aura Root	Routing module	aura-groot
Channel adapter	aura-bridge
Skill	Skill

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura microfrontend base

Description of aura-microfrontend-base, foundational component for the development of user interfaces in Aura ecosystem

Aura Virtual Assistant component

Introduction

aura-microfrontend-base is a foundational component that provides a common structure and libraries for the development of user interfaces in Aura ecosystem (Aura Virtual Assistant and ATRIA).

Its microfrontend component is a small, self-contained element that is part of a larger microfrontend architecture. It follows the same principles as microservices, but for the frontend, allowing independent development, deployment and scaling of different parts of a web application.

As defined before, this is a base component over which certain interfaces will be developed, but it will not be directly productized.

Technical foundations

This is a Next.js project bootstrapped with create-next-app.

The project leverages the following technologies and tools:

React: A JavaScript library for building user interfaces, used as the core framework for component development.
@telefonica/mistica: A design system and component library that provides a set of reusable UI components and styles, ensuring consistency and best practices in UI development.
Tailwind CSS: A utility-first CSS framework for styling components and layouts.
TypeScript: A strongly typed programming language that builds on JavaScript, ensuring type safety and better developer experience.
Microfrontend Architecture: Enables independent development, deployment, and scaling of frontend components.
Dynamic Imports: Used to load microfrontend components at runtime, improving modularity and performance.
Environment Variables: Configuration is managed through environment variables like AURA_SITE_CONFIG_PATH to ensure flexibility and adaptability across environments.

These foundations ensure scalability, maintainability, and a seamless developer experience.

Architecture and components

aura-microfrontend-base is composed of two main components:

Core Framework: The foundational Next.js structure that handles routing, rendering, and component lifecycle.
Dynamic Component Loader: The system responsible for loading and rendering components based on the site configuration.

Basic architecture diagram

sequenceDiagram
    participant Project as user request
    participant Config as site-config.json
    participant Pages as Page Manager
    participant Layout as Layout Engine
    participant ComponentLoader as Component Loader
    participant DOM as Browser DOM

    Project->>Config: read site configuration
    Config-->>Project: return site structure
    
    Note over Project: site path or default page
    Project->>Pages: setup page routes from config
    
    Pages->>Layout: process page layout configuration
    
    Layout->>Layout: setup container with containerClass
    Layout->>Layout: create grid with gridClass
    
    Note over Layout: process components by position
    
    loop for each component
        Layout->>ComponentLoader: request component by importType
        
        alt importType is "internal"
            ComponentLoader->>ComponentLoader: load from local components folder
        else importType is "external"
            ComponentLoader->>ComponentLoader: load from library dependency
        else importType is "microfrontend"
            ComponentLoader->>ComponentLoader: load dynamically from remote source
        end
        
        ComponentLoader-->>Layout: Return component
        Layout->>Layout: Apply className and position
        Layout->>Layout: Inject attributes and props
        Layout->>DOM: Render component in grid position
        DOM-->>Layout: Component rendered
    end
    
    Layout-->>Pages: Page layout rendered

    
    Pages-->>Project: page configured and route ready
    Project->>Project: start listening for navigation

Server features

The server component of aura-microfrontend-base provides:

Dynamic Routing: Routes are generated based on the configuration file and mapped to corresponding components.
Server-Side Rendering (SSR): Components can be rendered on the server for improved performance and SEO.
API Integration: Built-in capabilities for connecting to backend services and APIs.
Environment Variable Management: Configuration is handled through environment variables for different deployment environments.
Development Server: Local development environment with hot-reloading for a smooth development experience.

Configuration

The AURA_SITE_CONFIG_PATH environment variable must be set to point to your project’s configuration file, as in the example below:

AURA_SITE_CONFIG_PATH=site-config.json npm run dev

aura-microfrontend-base configuration file must be a valid JSON that defines the structure of the site, including the layout of the pages and the components to render.

The configuration file must follow this structure:

{
    "mainPage": "home",
    "pages": [
        {
            "id": "home",
            "name": "Home",
            "path": "/home",
            "layout": {
                "containerClass": "min-h-screen flex flex-col container mx-auto",
                "gridClass": "grid grid-cols-1 grid-rows-[200px_1fr] flex-1",
                "components": [
                    {
                        "library": "@telefonica/example-library",
                        "componentName": "Header",
                        "position": 1,
                        "importType": "external",
                        "className": "col-span-2 row-span-1"
                    },
                    {
                        "componentName": "TestComponent",
                        "position": 2,
                        "importType": "internal",
                        "className": "h-full col-span-1 row-span-2"
                    },
                    {
                        "componentName": "TestPlugin",
                        "position": 3,
                        "importType": "microfrontend",
                        "className": "h-full col-span-1 row-span-2",
                        "attributes": {
                            "API_ENDPOINT": "${AURA_MICROFRONTENDS_SERVER_HOST}/api/plugin-test",
                            "REFRESH_INTERVAL": "1000"
                        }
                    }
                ]
            }
        }
    ]
}

Configuration fields

mainPage: Default page ID to load when accessing the root path
pages: Array of page configurations
- id: Unique identifier for the page
- name: Display name of the page
- path: URL path for the page
- layout: Container configuration - containerClass: Tailwind classes for the main container - gridClass: Tailwind classes for the grid layout - components: Array of component configurations - library: (Optional) External library name - componentName: Name of the component to render - position: Order in the layout - importType: Type of component (internal, external, or microfrontend) - internal: Components that are part of the current project and are located in the components directory - external: Components from external libraries (like @telefonica packages) that are installed as dependencies - microfrontend: Components that are loaded dynamically from a remote server, allowing for runtime integration of independent applications - className: Tailwind classes for the component - attributes: (Optional) Additional component properties

Technical guidelines

Guides to manage the aura-microfrontend-base component: