Guidelines for developers willing to develop features over Aura Platform
Introduction
Aura is designed as a configurable Platform in which developers can build specific features both for customizing their Aura system or in order to contribute to Aura global code.
For this purpose, Aura Platform put at the OBs’ disposal tools, guidelines and rules that should be followed by the developers:
Use Aura utilities: specific procedures based on Microsoft Bot Framework 4 functionalities that ease the execution of different tasks in Aura.
Develop an Aura bridge plugin: guidelines for the generation of a new plugin and its activation/deactivation in aura-bridge
aura-bot-library-util utility contains different utility files for aura-bot libraries
Introduction
aura-bot-library-util utility is an NPM library with utility functions provided by the Aura Bot Global Team to make it easier the development of dialogs in aura-bot.
This library only contains utilities used in the dialogs, not needed by the bot itself.
To use it, just define the dependency with @telefonica/aura-bot-library-util in your package.json file. It is published as a private library in NPM, so request a valid NPM token to access it.
The utility files included in aura-bot-library-util utility are described in the following sections.
CurrencyUtils
aura-bot-library-util/currency-util.ts
getLocalizedCurrency
It returns money amounts formatted with country specific settings.
It returns converted internet data amount and unit from bytes quantity, following the country specific settings.
param
type
description
mandatory
content
TurnContext
Step context
yes
configuration
Configuration
Dialog config variables
yes
data
number
Data amount
yes
forcePoint
boolean
It indicates if the decimals will be separated by points
no
const[dataQt,dataUnit]=getDataAndUnit(context,config,1536)const[dataQt,dataUnit]=getDataAndUnit(context,config,1536,true)/*
* Result depending on AURA_DEFAULT_LOCALE env var
* pt-br - ['1,5', 'GB']
* en-gb - ['1.50', 'GB']
*/
LocaleUtils
aura-bot-library-util/locale-util.ts
getLiteral
This intermediate method generates a getText function ready to receive resources keys and sequential params.
param
type
description
mandatory
context
TurnContext
step context
yes
library
string
libraryName
no
/* If we need to convert only one text in function scope, we can use it as one line function */consttext: string=getLiteral(context,'services')('services.find.moreinfo',param1,param2);consttext: string=getLiteral(context)('services:services.find.moreinfo',param1,param2);/* When we need to convert several texts in function scope for the same library,
we can preassign result function to variable and proceed all along dialog step */constgetServicesText: Function=getLiteral(context,'services');consttext: string=getServicesText('services.find.moreinfo',param1,param2);/* if we need to convert several texts in function scope for several libraries */constgetText: Function=getLiteral(context);consttext: string=getText('services:services.find.moreinfo',param1,param2);consterrorText: string=getText('errors:error.notFound');
getTextByKeys
Factory function used mainly in graphs to retrieve converted text objects, avoiding redundancy.
Given an intent, it replaces the current dialog by the intent that matches with it. This method triggers the main dialog and keeps the context-filter functionality in the replaced one.
aura-minibot-service simplifies how aura-mini-bot and aura-mini-groot are generated.
Introduction
The aura-minibot-service utility contains the common elements needed by aura-bot and aura-groot to generate their mini versions: scripts, mock classes and templates, etc.
aura-movistar-libraries utility contains utilities to use with dialogs in Movistar channels
Introduction
aura-movistar-libraries utility is an NPM library that contains utility functions provided by Aura Global Team to be used with dialogs in Movistar channels:
To use it, just define the dependency with @telefonica/aura-movistar-libraries-utilities in your package.json. It is published as a private library in npm, so request a valid NPM token to access it.
models/intent-models
It contains all the necessary classes and interfaces for Movistar use cases. Some of these classes and interfaces are:
MovistarPlusResolution, MovistarStatus, SuggestionAction, etc.
utils/movistar-payload-utils
It contains the functions executed to fill the payload of the message depending on the settings/payloadType in the intent configuration.
The principal functions are: getPayloadDefault, getPayloadCommunications, getPayloadTV and getPayloadWifi.
utils/movistar-resolution-utils
It contains the function to make a query to the Movistar plus resolution API. This query returns a MovistarPlusResolution object that contains suggestions, actions, sound, payload, etc.
Object example returned by getMovistarPlusResolution with intent intent.common.greetings in the STB channel:
{"intent":"intent.common.greetings","entities":[],"resources":[{"type":"title","name":"tv.notUnderstand","params":{}}],"status":"ok","payload":{"type":"","data":{},"data_additional":{},"action":"NONE","status":{"code":"470","message":"Intent not Supported Error - General Code","info":{}}},"user_action":{},"suggestions":[{"intent":"intent.tv.display","entities":[{"entity":"Canal Cocina","type":"ent.audiovisual_channel","score":1,"canon":"Canal Cocina","label":""}],"resources":[{"type":"title","name":"tv.switchToChannel.suggestion.title","params":{"channels":"Canal Cocina"}},{"type":"button","name":"tv.switchToChannel.suggestion.button","params":{"channels":"Canal Cocina"}}]}/// More suggestions were returned but are ommited in this example object
],"sound":"none"}
utils/movistar-utils
A compendium of utilities which formats activity’s channelData to be compatible with Movistar channels.
The principal function of this file is getMovistarMessage. It returns a properly formed channelData message for the Movistar channels, depending on the input parameters and the settings field in the configuration of the intent. All other functions are auxiliary of this principal.
Example of object returned by getMovistarMessage with input params:
{"text":"Hola, buenas","channelData":{"intent":{"id":"intent.common.greetings","name":"intent.common.greetings","entities":[]},"content":{"text":"Hola, buenas","speak":"Hola, buenas","sound":"positive","actions":[]},"dialogContext":[{"text":"Quiero ver Canal Cocina","value":{"intent":"intent.tv.display","entities":[{"entity":"Canal Cocina","type":"ent.audiovisual_channel","score":1,"canon":"Canal Cocina","label":""}]}}/// More suggestions were returned but are ommited in this example object
],"customData":{"type":"common","action":"COMMON.GREETINGS","data":{},"data_additional":{},"status":{"code":"200","message":"Success - General Code","info":{}}},"fullScreen":false,"version":"1.0.0"},"inputHint":"acceptingInput","attachments":[{"contentType":"application/vnd.microsoft.card.hero","content":{"buttons":[{"type":"postBack","title":"Quiero ver Canal Cocina","value":{"name":"Quiero ver Canal Cocina","text":"Ver Canal Cocina","type":"suggestion","intent":"intent.tv.display","entities":[{"entity":"Canal Cocina","type":"ent.audiovisual_channel","score":1,"canon":"Canal Cocina","label":""}],"inputIntent":"intent.common.greetings","inputEntities":[]}}/// More suggestions were returned but are ommited in this example object
]},"name":"SUGGESTIONS"}]}
suggestionType
Formerly, there was a differentiation between channels based on their prefix. This implementation was quite strict and to parameterize this behavior a new configuration variable has been developed.
SuggestionType must be configured for channels Movistar-Plus, Set-on-Box and Set-on-Box-Haac and its value can be either actions or attachments describing where suggestions must be placed in activity’s channelData.
‘actions’: Movistar-Plus
‘attachment’: Set-on-Box, Set-on-Box-Haac
1.1.5 - Aura Bot Common library
Aura Bot Common library
Aura Bot common is a core library that contains utilities for the integration of different components with Aura
Introduction
Aura Bot common is a helper library published in NPM that includes useful utilities to handle conversations both in aura-bot and in the dialogs.
The Aura Bot common library holds models and utility code shared between aura-bot and the libraries. It contains two different types of methods: external and internal, which are described in the following sections.
Aura utils: generic tools to manage dialogs and prompts.
It gets a validator function that will be called each time the user responds to the prompt. In this function, it controls the number of retries to show the prompt.
If attempt is allowed, it is the recognizer that will let an attempt (when there are not results) or the dialog the will manage the result found.
If the retries exceed to maxRetries parameter, the control will be returned to dialog, managing results or not.
If an attempt is shown, the configured retryPrompt (or the same prompt as default) will be shown.
param
type
description
mandatory
maxRetries
number
Number of retries to show the prompt
yes
// Create a ChoicePrompt without retries
constmyPrompt=newChoicePrompt(ID_PROMPT,PromptUtils.getRetriesControl(0));
getRetriesValidatorAndOverwriteRecognizerResult
This validator function is similar to getRetriesValidator but also overwrites the recognizer result obtained by the prompt recognizer with a previous stored value.
This recognizer result value can be set with the function ContextUtils.setPromptRecognizedResult. By default, the result is: prompt-check-recognizer-middleware.
param
type
description
mandatory
maxRetries
number
Number of retries to show the prompt
yes
// Create a ChoicePrompt with 3 retries and use of ordinals control overwriting promptRecognizedResult with the value of aura-bot.
constmyPrompt=newChoicePrompt(ID_PROMPT,PromptUtils.getRetriesValidatorAndOverwriteRecognizerResult(3));
It contains different methods, described in the following sections.
getAsyncCallbackUrl
This method is used for use cases that use asynchronous APIs and have to send a callback URL. It builds the URL with the parameters expected by the end point of aura-bridge, which will be the one that receives the event.
This returns the URL to send as callback parameter.
It gets AppContext from client and returns an structure with certain information about the client such as: device, application, location, channel, etc.
It sets AppContext in channelData. In certain scenarios, it will be necessary to overwrite it to keep the track of changes that have occurred (e.g., in the applied contextFilters).
⚠️ In some cases, the recognition process has been altered to avoid the execution of other recognizers and the result might not be as expected.
param
type
description
mandatory
context
TurnContext
Context where the channel data will be taken from
yes
exportinterfaceRecognizerResult{/**
* Utterance sent to recognizer
*/readonlytext: string;/**
* If original text is changed by things like spelling, the altered version.
*/readonlyalteredText?: string;/**
* Intents recognized for the utterance.
*
* @remarks
* A map of intent names to an object with score is returned.
*/readonlyintents:{[name: string]:{score: number;};};/**
* (Optional) entities recognized.
*/readonlyentities?: any;/**
* (Optional) other properties
*/[propName: string]:any;}
It gets the recognizerResult from turnState to retrieve the value in the validator function.
This function is currently used in getRetriesValidatorAndOverwriteRecognizerResult function, used for overwrite the default recognizerResult with the result got in the prompt recognizer.
It saves the recognizerResult in the turnState to retrieve the value later in the validator function.
This function is currently used in the prompt recognizer where a custom recognizeChoices is performed (including number and ordinal recognition).
param
type
description
mandatory
context
TurnContext
Context where the channel data will be taken from
yes
promptRecognizedResult
PromptRecognizerResult
Result of prompt recognition
yes
Example of use:
constresult: PromptRecognizerResult<FoundChoice>={succeeded: false};if(matched.length>0){result.succeeded=true;result.value=matched[0].resolution;}// Save personalized recognizer result
ContextUtils.setPromptRecognizedResult(context,result);
existError
It checks if an error exist and returns a boolean. (Mainly, it is designed for middlewares)
param
type
description
mandatory
context
TurnContext
Context where the channel data will be taken from
yes
Example of use:
if(!ContextUtils.existError(context)){...}
setError
It sets an error without warning the user at that time, allowing the execution of remaining middlewares. (Mainly designed for middlewares)
2021-3-15 15:55:02 User: Que ponen en la cuatro
2021-3-15 15:55:04 AURA: Ok, estarei aqui sempre que você precisar.
2021-3-15 15:55:08 User: Hola?
2021-3-15 15:55:09 AURA: Ok, estarei aqui sempre que você precisar.
2021-3-15 15:55:15 User: Bye
2021-3-15 15:55:16 AURA: Ok, estarei aqui sempre que você precisar.
It saves the cached state object if it has been changed. If the force flag is passed in the cached state, the object will be saved regardless of whether it has been changed or not. If no object has been cached, an empty object will be created and then saved.
param
type
description
mandatory
context
TurnContext
Context for current turn of conversation with the user
yes
force
boolean
If true, the state will always be written, out regardless of its change state. By default, false.
It reads and caches the backing state object for a turn. Subsequent readings will return the cached object, unless the force flag is passed, in which the state object to be re-read will be forced.
param
type
description
mandatory
context
TurnContext
Context for current turn of conversation with the user
yes
force
boolean
If true, the cache will be bypassed and the state will always be read directly from storage. By default, false.
AI Service is an utility found in Aura Bot common library in charge of managing calls to external AI services.
Introduction
AI Service allows you to manage calls to external AI services through generative APIs. The service is responsible for abstracting the details of the API calls and providing a simple interface for recognizing intents using different recognizers.
Creating an instance of the service
To create an instance of the service, you need to provide a AiRecognizerServiceConfiguration object with the following parameters:
The recognize method allows you to recognize intents from a TurnContext using a specific intent configuration. The method takes the following parameters:
This method uses the generative API to send the message and receive the response. It also manages the session ID by storing it in the TurnContext for future requests.
Recognizing intents using triage. The recognizeUsingTriage method
The recognizeUsingTriage method allows you to recognize intents from a TurnContext using the triage recognizer. The method takes the following parameter:
The aura-channel-data-handler utility contains the necessary utilities to manage the content of []channelData](/docs/components/request-response-model/) for the input/output response to aura-bot.
These guidelines will allow you to get a copy of the project running on your local machine for development and testing purposes.
Run tests
This project uses Mocha for BBDD testing.
Unit tests
$ npm run test
Style tests
These tests perform the validation coding rules defined in Aura using the eslint tool.
You can validate the code using:
$ npm run lint
Coverage tests
You can run the coverage tests using:
$ npm run test
The threshold established to validate the coverage is as follows:
lines: 90%
functions: 90%
statements: 90%
branches: 70%
Use aura-channel-data-handler utility
Transform channelData from an old version to the latest version (normalized)
The mapper will try to obtain the correct mapper based on the version indicated in version field. If the request does not contain a version, it will try to map using version 1.0.0:
Continuing with the previous example, the channelData obtained from the normalization can also be used as a helper to perform predefined operations:
// You can get the original request (example v1)
...import{ChannelDataRequestHelper}from'@telefonica/telefonica/aura-channel-data-handler';...consthelper: ChannelDataRequestHelper=channelDataasunknownasChannelDataRequestHelper;constchannelDataOriginal=helper.getOriginalRequest();
For this component, SemVer is used for versioning.
Tests running
This project uses tslint to validate the coding style.
Unit tests
To execute the tests, use the following command:
$npmruntest
Coding style tests
These tests perform the validation coding rules defined in Aura using the tslint tool.
You can validate the code using:
$npmrunlint
Developer notes
The folder structure used for development is:
.├──__test__// Folder for test in jest
└──src// Shared code between APIs and modules.
├──handlers// Expressjs handlers folder
├──middlewares// Expressjs middlewares folder
├──models// Models
├──modules// Modules to be loaded by orchestrator
├──routes// Expressjs routes folder
└──utils// Common utilities
Handlers
validate-auth-apiKey: To get or create the correlator and validate the APIKey of the request.
validate-google-auth: To validate the Google x-goog-signature of the request.
Middlewares
error-handler: Global error handler.
incoming-request: To register metrics for incoming requests.
set-correlator: To get the correlator from several places.
set-not-cache: To set Cache-Control and ETag headers to prevent caching.
validate-apiKey: To get or create the correlator and validate the APIKey of the request.
validate-token: To validate the token.
Modules
async-hooks-blocking: To manage async hooks and capture stack traces.
configuration-manager: To manage config and load environment variables.
oas-manager: To register security handlers, for example openAPIBackend.
plugin-manager: To manage plugins to add different services.
prometheus-manager: To manage Prometheus metrics.
Routes
generic: To manage generic routes such us: healthz, favicon and shutdown.
metrics: To manage metrics routes.
Utils
aura-error-factory: Collection of functions to manage conversion errors.
oas-utils: Collection of functions to manage token, validation failures and error responses.
time-utils: Collection of functions to manage time.
token-validations: Collection of functions to validate tokens.
Plugins
dapr-pubsub-service: plugin which allows the interaction with the Dapr PubSub Service
1.2.2.1 - dapr-pubsub-service plugin
dapr-pubsub-service plugin
Description of the dapr-pubsub-service plugin
Introduction
The dapr-pubsub-service allows the interaction with the Dapr PubSub Service.
aura-configuration utility allows loading Aura configuration and using models related to channels
Introduction
aura-configuration utility is a library designed to load Aura configuration (channels, etc.) and use models (classes and interfaces) related to channels.
AuraCurrentChannelsConfiguration is a singleton class that contains some utils to obtain information about the channels.
This module is initialized at server start-up like other singleton modules. To use it, you should use the instance already created:
Contains the URL to access aura-configuration-api service
AURA_AUTHORIZATION_HEADER
Complete authorization header to be sent to aura-authentication-api, with the following format: APIKEY xxxxxx.
Aura Skill Configuration
AuraSkillConfiguration is a singleton class that contains some utils to obtain information about the skills.
This module is initialized at server start-up like other singleton modules. To use it, you should use the instance already created:
AuraApplicationConfiguration is a singleton class that contains some utils to obtain information about the applications.
This module is initialized at server start-up like other singleton modules. To use it, you should use the instance already created:
aura-crypto-adapter utility provide methods for encrypting and decrypting data.
Introduction
aura-crypto-adapter is a utility that provides methods for encrypting and decrypting data using a specified encryption algorithm and managing vectors associated with the encryption process. It also includes a method for generating checksums using different algorithms.
To encrypt information, call the encrypt method of the cryptoAdapter instance, passing the information and an optional key (if it is needed to use a different one than the given on configuration).
Checksum Verification: If you need to verify the integrity of the data, you can use the checksum method of the CryptoAdapter class. Store or transmit the checksumValue along with the encrypted data. When decrypting, calculate the checksum again and compare it with the stored value to ensure data integrity.
constchecksumValue=CryptoAdapter.checksum(data);
Run tests
You can validate run jest testing tools with the script:
$ npm run test
Linter
You can validate the code using the eslint tool with the script:
aura-locale-manager utility allows Aura Bot to manage text resources
Introduction
aura-locale-manager is a utility that allows aura-bot to manage text resources, that means, to use custom i18n per environment and channel.
This library provides two functionalities:
LocaleManager class to handle i18n texts resolution.
LocaleRemoteLoader class that runs before LocaleManager starts in order to get a fresh version of the locale files from the configured Azure Storage blob container. If an error occurs when uploading the remote files, the former set of locale files will be used.
ℹ️ This step is not required when developing a use case, as the locale manager instance is provided by aura-bot. Take it only for descriptive purposes.
A new instance can be created by calling getInstance() method, passing config object as parameter (mandatory in the first call). Subsequent calls to getInstance will return the same instance (singleton).
When creating the instance, the folder specified in the config var AURA_LOCALE_FOLDER is read and looks for JSON files. All those files are loaded once, during the start-up stage.
The environment can be specified in AURA_SERVICE_ENVIRONMENT config variable.
Use of aura-locale-manager utility
After getting the instance, translations can be got by using getText method (using the default locale specified in AURA_DEFAULT_LOCALE) or by using getTextByLocale (specifying the desired locale).
This method gathers the translation for the given term key, locale and user’s configuration. It allows getting the texts without a full AuraUser instance.
aura-orchestrator utility allows Aura Bot to manage text resources by environment and channel
Introduction
aura-orchestrator utility contains the orchestrator used in aura-bot and aura-bridge based on the strategy design pattern. It is based on @telefonica/event-bus dependency.
The orchestrator handles an array of classes (with certain required elements), that aura-bot may receive configuration if necessary. Modules are then initialized asynchronously. For example:
constorderedModules=[ModuleA,ModuleB,ModuleC];// Instantiate the Service App.
constappOrchestrator=newOrchestrator();// Add configuration, that will be required by almost all other modules
appOrchestrator.setConfigurationManager(Configuration);// Add the dependent modules in order.
appOrchestrator.addModules(sortedModules);// Initiate the App.
awaitappOrchestrator.init();
Creation of singleton modules
In order to develop a new singleton module that will be used by aura-bot both itself and during the plugins execution, developers should keep in mind the following issues:
All modules added to the orchestrator need and publish a static variable called instance, that will hold the singleton instance of that module.
Modules must have a public and static method init, that could be async or not, and that will create the singleton instance (and setting it to the instance variable). This method should only be called once by module or must throw an exception. It is only called by the orchestrator.
An example implementation is shown below:
exportclassClass1{publicstaticinstance: Class1;privateconstructor(){// Stuff here...
}publicstaticasyncinit(config: Configuration):Promise<Class1>{if(!Class1.instance){Class1.instance=newClass1();// More stuff here
returnClass1.instance;}else{thrownewError('An instance of Class1 already exists');}}}
1.2.9 - aura-storage-file-manager
aura-storage-file-manager utility
aura-storage-file-manager utility is in charge of uploading / downloading remote files
Introduction
aura-storage-file-manager is an utility to manage local and remote resources. At the moment, remote files are arranged using Azure Storage.
aura-validate-apikey utility is a singleton module prepared to be used with orchestrator.
(async()=>{try{// Sorted modules initialization
constsortedModules=[AuthenticationApiKey,...];// Instantiate the Service App.
constappOrchestrator=newOrchestrator();// Add configuration, that will be required by almost all other modules
appOrchestrator.setConfigurationManager(ConfigurationManager);// Add the dependent modules in order.
appOrchestrator.addModules(sortedModules);// Initiate the App.
awaitappOrchestrator.init();}catch(error){logger.error({error: error.message,msg:'Server cannot start',stck: error,corr: CorrelatorUtil.auraSystem});}})();
Use aura-validate-apikey utility
After initialising the AuthenticationApiKey instance (calling static init method once), the APIKey could be validated by different methods:
Middleware use
APIKey Authorization for the swagger tools middleware use. This function lets us send a callback when the validation has finished. (Not used yet).
Each externally available API endpoint in Aura is authenticated by an APIKey that should have been generated independently for each environment.
These APIKeys contain an encrypted data model that allows both checking that the key was encrypted with the environment ENCRYPTION_KEY and that it granted the access to the given API endpoint and method.
The APIKey model is described below:
Field
Type
Description
i
string
id autogenerated unique identifier (UUID) of the APIKey. Added to be able to invalidate individually one APIKey. ⚠️ Future use.
s
string
scope that will be accessible with this APIKey. Currently, it contains part of the path of the endpoint. - To access all endpoints of Aura Services, it contains aura-services. - To access only one of the endpoints, for instance, to access only /token in authentication-api, it should contain aura-services:token. To access /token and /ping, its content should be aura-services:token,ping. It is used in all modules.
a
string
authorized. It should contain the name of the client that is authorized by this APIKey: Kernel, Novum, etc. Currently, it is not checked, meaning that this field is not taken into account for accessing or not to an endpoint.
v
string
version. Version of Aura where it has been created. ⚠️ Future use, it will be used in case of changing the internal model.
e
string
environment where this APIKey applies. ⚠️ Future use. Currently, it is checked by having a different ENCRYPTION_KEY per environment.
m
string
mode. API access mode granted by this APIKey: r (read), w (write), rw (read and write).
t
Date
date of creation of the APIKey. Used to invalidate already created APIKeys.
c
string
checksum of the APIKey to validate that the APIKey is encrypted with the defined ENCRYPTION_KEY. It is used in all modules.
API Key validation is applied in:
aura-bot to validate requests coming from aura-bridge
aura-authentication-api
aura-bridge
channel-communications-manager
1.2.11 - aura-file-validator
aura-file-validator utility
aura-file-validator is an utility to validate files
Introduction
aura-file-validator is a small utility capable of validating files type and size.
aura-file-validator uses mmmagic library to determine the real mime type and extension of a file, reading its bytes. Once determined the type/mime type, it will be compared against a list of valid types.
File size validation
aura-file-validator reads file data length and check it. Three types of size validations are defined:
File is bigger than a given size.
File is smaller than a given size.
File size is between a minimum and maximum size.
File type and size validation
aura-file-validator checks file type.
If it passes the validation, then a size validation is done.
If the type is not supported, then size validation is not executed.
Complete file validation flowchart
graph LR
A(start) --> B[Validate type]
B --> C{Valid type?}
C --> |Yes| D[Validate max size]
C --> |No| G(Validation result)
D --> E{Valid max size?}
E --> |Yes| F(Validate min size)
E --> |No| G
F --> G
1.2.12 - aura-redis-handler
aura-redis-handler utility
aura-redis-handler utility suitable for the connection and use of Redis
Introduction
aura-redis-handler is a utility to connect and use methods from Redis.
constconfig={AURA_REDIS_MODE:'SINGLE',// SENTINEL, CLUSTER or SINGLE
AURA_REDIS_HOSTS:'localhost:6379,127.0.0.1:6379',// {host}:{port},{host}:{port} ... n
AURA_REDIS_PASSWORD:'',AURA_REDIS_DATABASE:'',AURA_REDIS_MAX_RECONNECT_RETRIES: 25,AURA_REDIS_MAX_RECONNECT_INTERVAL: 5000,// ms
AURA_REDIS_SENTINEL_INSTANCE_NAME:'',AURA_REDIS_USE_CONNECTION_POOL: true,AURA_REDIS_CONNECTION_POOL_MAX: 2,AURA_REDIS_CONNECTION_POOL_MIN: 100,};this.client=awaitRedisConnector.init(config);
Models
ZRangeWithScoresOptions
Property
Type
Description
BY
string
Option key
LIMIT
Object
Object with properties offset and count
Methods
zRangeWithScores
This method returns all the elements in the sorted set at key with a score between min and max. The elements are considered to be ordered from low to high scores.
This method returns all the elements in the sorted set at key with a score between min and max. Unlike the default ordering of sorted sets, for this command the elements are considered to be ordered from high to low scores.
This method inserts all the specified values at the head of the list stored at key. If key does not exist, it is created as an empty list before performing the push operations. When key holds a value that is not a list, an error is returned.
This method returns the specified elements of the list stored at key. The offsets from and to are zero-based indexes, with 0 being the first element of the list, 1 being the next element and so on.
This method adds the specified members to the set stored at key. Specified members that are already a member of this set are ignored. If key does not exist, a new set is created before adding the specified members.
This method returns the specified members from the set stored at key. Specified members that are not a member of this set are ignored. If key does not exist, it is treated as an empty set and this command returns 0.
When loading the modules in the app orchestrator (or any other method), the methods startMetricsTimer and kpiApiRequest could be overriden.
(both are optional).
// Initiate the App.
awaitappOrchestrator.init();// Sets required methods in HttpMonkeyPatcher
HttpMonkeyPatcher.instance.startMetricsTimer=startPrometheusTimer;HttpMonkeyPatcher.instance.kpiApiRequest=KpiHandler.instance.apiRequest.bind(KpiHandler.instance);
1.6 - aura-json-schema-generator
aura-json-schema-generator utility
aura-json-schema-generator utility is a tool to generate Typescript classes
Introduction
aura-json-schema-generator utility is a tool to generate Typescript classes from JSON schema file(s). It provides a command that generates Typescript interface files from one or more schema files.
The input file(s) is mandatory. The output directory is optional (./types by default). In it, a directory is created for every schema including the schema version, and inside that directory, another one is created with the schema name that will contain the typescript interfaces.
Installation
Although it is possible to install the package locally, it is recommended to install it globally, in order to allow the execution of the generator binary anywhere.
Once installed globally, a new command aura-json-schema-generator will be available in the PATH, so we can generate Typescript interface files from one or more schema files.
Run aura-json-schema-generator utility
aura-json-schema-generator
Two other names for this command are available: aura-jsg and aujsg
Arguments
There are two arguments that can be passed in the command line when running:
The first argument (mandatory) is the path of a JSON Schema file or a directory that contains JSON Schema files.
The second argument (optional) is the path where the Typescript files will be created. types is the default value. This information can be found when running the command without arguments:
This command will create the Typescript interfaces from a JSON Schema file named simple.json in the default output directory types within current directory:
$ aura-json-schema-generator ./simple.schema.json
DEBUG Reading json schema ./simple.schema.json
INFO Output path will be './types'INFO Writing ./types/1.0.0/simple/simple.d.ts
This command will create all the Typescript interfaces from channelData schemas in the directory resources/channel-data/3.0.0 in a specified output directory src/channel-data/:
$ aura-json-schema-generator resources/channel-data/3.0.0/ src/channel-data/
DEBUG Reading json schema ./resources/channel-data/3.0.0/request/channeldata-3.0.0-request.schema.json
DEBUG Reading json schema ./resources/channel-data/3.0.0/response/channeldata-3.0.0-response.schema.json
INFO Output path will be './src/channel-data'INFO Writing ./src/channel-data/3.0.0/request/request-channel-data.d.ts
INFO Writing ./src/channel-data/3.0.0/response/response-channel-data.d.ts
The following example shows the execution of the aura-json-schema-generator in the happening of an error in any of the provided schemas. As can be seen, an error is shown indicating both the file that contains the error and thus cannot be processed and the specific error or errors in the schema.
In this case, the error is in the file request/channeldata-3.0.0-request-wrong.schema.json and the problems are that neither ApplicationWrong nor AuraCommandWrong cannot be found, so the schema for this file cannot be generated. The rest of the files in the folder resources/channel-data/3.0.0/ will be processed and their schemas will be generated correctly.
$ aura-json-schema-generator resources/channel-data/3.0.0/ src/channel-data/
DEBUG Reading json schema ./resources/channel-data/3.0.0/request/channeldata-3.0.0-request-wrong.schema.json
ERROR Object ApplicationWrong not found
ERROR Object AuraCommandWrong not found
ERROR Error reading file ./channel-data/3.0.0/request/channeldata-3.0.0-request-wrong.schema.json
DEBUG Reading json schema ./channel-data/3.0.0/response/channeldata-3.0.0-response.schema.json
INFO Output path will be './src/channel-data'INFO Writing ./src/channel-data/3.0.0/response/response-channel-data.d.ts
aura-types-from-swagger-generator
Two other names for this command are available: aura-tfsg and autfsg
Arguments
There are three arguments that can be passed in the command line when running:
The first argument (mandatory) is the path of a JSON/YAML Schema file or a directory that contain JSON/YAML Schema files.
The second argument (optional) is the path where the Typescript files will be created. types is the default value.
The third argument (optional) is an array of strings with options:
–unique-file: Create all the models data in a single file by swagger.
–unify-enumerated: Unify the generated enums.
–particularize-enumerated: Particularize the generated enums.
This command will create the typescript interfaces, enums and types from a Swagger file named swagger.json in the default output directory types within current directory:
{..."Payload":{"description":"Channel data payload information","title":"Payload","type":"object","additionalProperties":false,"properties":{"bridge":{"$ref":"#/definitions/channel-data/Bridge","description":"Information sent from the bridge","title":"bridge"},"handover":{"$ref":"#/definitions/channel-data/Handover","description":"Information sent from Handover","title":"handover"},"event":{"$ref":"#/definitions/channel-data/PayloadEvent","description":"Information sent as an event","title":"event"}},"required":["bridge"]},...}
The outcome of the execution of the previous command will generate:
The input JSON schema (in the example, aura-models/resources/channel-data/payload.schema.json)
A markdown output file, in the example: payload-types.md:
### Payload
Channel data payload information
| Property | Type | Description |
| --------------- | ------------ | -------------------------------------------------- |
| **bridge** | Bridge | Information sent from the bridge |
| *handover* | Handover | Information sent from Handover |
| *event* | PayloadEvent | Information sent as an event |
Components
Enums
Particularize Enumerated
For this flow, enums are generated as follows:
Enums are generated from the properties of models that have the isEnumerated and enums flags.
Enums are generated from types, checking if it is a type with an array, a type with different types, etc.
The enums that are not being used are filtered out to avoid generating them.
oneOf
oneOf is a property used to indicate that a component can have one of the types it receives as options. For example:
file: Entity files will be stored locally on the current server instance. For development purposes.
blob: Entity files will be stored remotely on an Azure Blob Container with aura-storage-file-manageruploadLogs.
Configuration
These configuration variables MUST be set in the server using the library.
KPIs environment variables:
AURA_VERSION: mandatory, Aura release.
AURA_DEFAULT_LOCALE: mandatory, string with the default locale to be used in the server. It uses two letter codes both for country and culture, for example: es-es, en-gb.
AURA_DEFAULT_TIME_ZONE: mandatory, timezone where the service is running.
AURA_KPI_TO_DSV_CACHE_TTL: optional, number of milliseconds to cache existing requests to calculate their duration. By default, 1800.
AURA_KPI_TO_DSV_DELIMITER: optional, string with the delimiter to be used in KPIs entities files. By default, |.
AURA_KPI_TO_DSV_EXTENSION: optional, string with the extension to be used in KPIs entities files. By default, txt.
AURA_KPI_FILE_PREFIX: mandatory, string with the prefix used in the KPIs entities files of this service. Usually its format is component/prefix. For CSV files it is used as is, so files are stored in Azure Storage in that path, meaning a virtual folder named component and that the files will start by prefix. But for Avro files, the value of the variable is split into 2 by / and only the prefix part is used to name the files, that are store together with files generated by all components in the same virtual folder.
AURA_KPI_STORE_MODE: optional, string to indicate what is the destination of the KPIs entities files. By default, blob.
If file, they will be stored locally on the instance, in the folder shown in KPI_TO_DSV_LOCAL_FILES_DIRECTORY. For development purposes.
If blob, they will be stored remotely on the Azure blob container shown in KPIS_STORE_CONTAINER. Mandatory in environments running on k8s.
AURA_KPI_TO_DSV_LOCAL_FILES_DIRECTORY: optional, string with the local directory to store KPIs entities files. By default, ./kpis-dsv. It MUST be the same than the one configured in KPIS_UPLOADER module. Only needed if AURA_KPI_STORE_MODE==file.
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY: optional, string with the Azure Storage Access Key to be able to write files in KPIS_STORE_CONTAINER. Only needed if AURA_KPI_STORE_MODE==blob.
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT: optional, string with the Azure Storage Account name to be able to write files in KPIS_STORE_CONTAINER. Only needed if AURA_KPI_STORE_MODE==blob.
AURA_KPIS_STORE_CONTAINER: optional, string with the name of the Azure Blob container to store KPIs entities files. By default, aura-kpis. It MUST be the same than the one configured in KPIS_UPLOADER module. Only needed if AURA_KPI_STORE_MODE==blob.
AURA_KPIS_BLOB_STORE_INTERVAL: optional, number with the time interval in milliseconds to upload asynchronously logs to the KPIS_STORE_CONTAINER. By default, 60000. Only needed if AURA_KPI_STORE_MODE==blob.
AURA_SOURCE_PATH_AVRO_ADAPTERS: optional, string containing the adapters to transform data, '/schemas/aura-csv-adapter.json' for CSV transform and '/schemas/aura-avro-adapter.json' to transform in CSV and AVRO. By default: '/schemas/aura-csv-adapter.json'.
Use aura-kpis utility
aura-kpis utility needs to add a dedicated events emitter that handles KPI events.
Initialize the specific KPI writer, based on your service configuration:
If the name of the environment variables of your service do not match with the library names, the configuration can be also be passed as an object of type KPIWriterConfiguration.
Depending on the value of AURA_KPI_STORE_MODE a different set of properties is validated.
In those meaningful steps, emit the corresponding event. The basic approach is to emit one event (KPIEmittersEvents.REQUEST_STARTED) when the request lands on the server and another when the response is returned (for instance, KPIEmittersEvents.KPIS_USER_REQUEST_FINISHED, in the case of a user request):
API entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_API_REQUEST_STARTED: event that shows that a new request to one external API is started. It adds to a cache a time mark, so the duration of the request can be calculated.
KPIS_API_REQUEST_FINISHED: last event of a new request to one external API has finished, emitted when the response is returned.
MESSAGE entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_MESSAGE_REQUEST_STARTED: event that shows that a new message goes in aura-bot. It adds a time mark to a cache, so the duration of the request can be calculated.
KPIS_MESSAGE_REQUEST_FINISHED: event to write the KPI of the given message being handled by aura-bot. It is emitted when a new message comes in and when its response is returned.
EXTENDED_MESSAGE entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_EXTENDED_MESSAGE_REQUEST_STARTED: event that shows that a new message enters aura-bot. It adds a time mark to a cache, so the duration of the request can be calculated. It is an extended logic made from MESSAGE entity.
KPIS_EXTENDED_MESSAGE_REQUEST_FINISHED: event to write the KPI of the given message being handled by aura-bot. It is emitted when a new message comes in and when its response is returned. It is an extended logic made from MESSAGE entity.
NOTIFICATION entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_NOTIFICATION_REQUEST_STARTED: event that shows that a new notification request has arrived. It adds a time mark to a cache, so the duration of the request can be calculated.
KPIS_NOTIFICATION_REQUEST_FINISHED: last event of a notification request, emitted when the response is returned in the corresponding controller.
RECOGNIZER entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_RECOGNIZER_REQUEST_STARTED: event that shows that a new recognizer request is started. It adds a time mark to a cache, so the duration of the request can be calculated.
KPIS_RECOGNIZER_REQUEST_FINISHED: last event of a recognizer request, emitted when the response is returned.
SUGGESTION entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_SUGGESTION_REQUEST_STARTED: event that shows that a suggestion is shown to the user or clicked by her. It adds a time mark to a cache, so the duration of the request can be calculated.
KPIS_SUGGESTION_REQUEST_FINISHED: last event of a suggestion shown to the user or clicked by her, emitted when the response is returned.
USER entity filling: This case needs partial events to assure that no value of the USER entity is missed, both in successful and error cases.
KPIS_REQUEST_STARTED: event that shows that a new request has arrived. It adds to a cache a time mark, so the duration of the request can be calculated.
KPIS_USER_REQUEST_FINISHED: last event of a user request, emitted when the response is returned in the corresponding controller.
KPIS_USER_CREATE_DB: partial user event, emitted when requesting the DB.
KPIS_USER_DELETE_DB: partial user event, emitted when the user is deleted from the DB.
KPIS_USER_FIND_DB: partial user event, emitted when the user is found in the DB.
KPIS_USER_CREATE_SERVICE: partial user event, emitted when returning from the user creation service.
KPIS_USER_LOGIN_SERVICE: partial user event, emitted when returning from the user login service.
KPIS_USER_DELETE_SERVICE: partial user event, emitted when returning from the user deletion service.
GATEWAY entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_GATEWAY_MESSAGE_REQUEST_STARTED event that shows that a new request enters aura-gateway-api. It adds a time mark to a cache, so the duration of the request can be calculated.
KPIS_GATEWAY_MESSAGE_REQUEST_FINISHED: event to write the KPI of the given request being handled by aura-gateway-api. It is emitted when its response is returned.
GROOT entity filling: this case just needs a couple of events to handle incoming and outgoing steps.
KPIS_GROOT_MESSAGE_REQUEST_STARTED: event that shows that a new message enters aura-groot. It adds a time mark to a cache, so the duration of the request can be calculated.
KPIS_GROOT_MESSAGE_REQUEST_FINISHED: event to write the KPI of the given message being handled by aura-groot. It is emitted when a new message comes in and when its response is returned.
1.8 - aura-behavior-manager
aura-behavior-manager utility
Guidelines for using aura-behavior-manager utility that allows modifying the behavior of Aura Bot or Aura Bridge in development environments without restarting them
Introduction
The main feature of the aura-behavior-manager is to be able to change the application (aura-bot or aura-bridge) behavior in development environments. All these changes can be made to the fly, without the need to restart the bot/bridge instance.
For this purpose, the aura-behavior-manager makes use of a series of commands linked to each of the previously configured behavior that can be sent to application in a specific format.
⚠️ WARNING
aura-behavior-manager must be exclusively used in development environments.
Do not use in production environments, as it disrupts Aura behavior and can lead to a faulty operation of the system.
As an example, aura-behavior-manager could change the value of an environment variable AURA_TERMS_AND_CONDITIONS_EXPIRATION:
To implement the aura-behavior-manager, you should make use of @telefonica/aura-behavior-manager library. This contains the following components:
BehaviorManager: Base class from which you must extend a specific BehaviorManager (Example: BotBehaviorManager, BridgeBehaviorManager).
BehaviorCommandControl: Responsible for the tasks associated with user commands.
BehaviorStubsHandler: It allows to register/modify/restore the modified methods in the code.
BehaviorCoreCommands: It contains the basic commands of the behavior manager.
RemotelyConfigurable: Utility that allows any class to read configuration from Azure using cron.
@startuml BehaviorManager class diagram
class RemotelyConfigurable {
storageCredentials
storageManager
containerName
sourceFilePath
cronExpression
cronTask
settings
status
lastUpdated
+start()
+stop()
+read()
}
class BehaviorManager {
name
commandPattern
behaviors
commandControl
stubsHandler
}
class BehaviorStubsHandler {
stubs
+start()
+restore()
+getStubs()
}
class BehaviorCommandControl {
component
commandPattern
+isCommand()
+getCommandList()
+execute()
}
class BehaviorProperty {
+setProperty()
+unsetProperty()
+getProperty()
+getDefaultProperty()
+getBehavior()
}
RemotelyConfigurable <|--down BehaviorManager
BehaviorManager *---right BehaviorCommandControl
BehaviorManager *---right BehaviorStubsHandler
BehaviorManager <|--down BehaviorProperty
BehaviorProperty <|--down BotBehaviorManager
BehaviorProperty <|--down BridgeBehaviorManager
@enduml
How that works?
In order to use the aura-behavior-manager, it must be loaded at the component start (aura-bot/aura-bridge).
Once it has started, it will load the list of behaviors (can be identified as plugins) that will allow certain actions to modify the behavior of the component. Examples of these behaviors can be: Property, Profile, etc. In the section Description of a Behavior, a behavior is described in detail.
When a message is received, the BehaviorCommandControl is responsible for detecting if it is a command. If so, it tries to execute the command associated with the behavior that defines it:
@startuml Behavior manager command control flow
start
#palegreen:message;
if (text message is command?) then (true)
#cyan:Execute command using "commandControl";
#cyan:Reply with command response;
end
endif
#palegreen:Process the message normally;
stop
@enduml
Description of a behavior
A behavior is the minimum unit that allows defining a change in the behavior of the system (aura-bot, aura-bridge).
Every behavior to be incorporated to aura-behavior-manager must implement the following interface:
/**
* @interface BehaviorComponent
*/exportinterfaceBehaviorComponent{getBehavior:()=>BehaviorComponentInformation;}/**
* @interface BehaviorComponentInformation
* Behavior information
*/exportinterfaceBehaviorComponentInformation{/**
* Behavior name.
*/name: string;/**
* Behavior description.
*/description?: string;/**
* Information with the piece of code used as a substitute for other functionality.
*/stub?: BehaviorStubInformation;/**
* Command information associated with this behavior.
*/commands: BehaviorCommand[];}
Therefore, a behavior has the following properties:
name (*): Behavior name.
description: Behavior description.
stub: As it is done in unit tests, it allows defining a code that will replace the original functionality in the code.
commands: List of commands to invoke the different functionalities that the behavior defines.
As an example, we will create a new HelloBehavior, which will respond with a simple ‘Hello’ in the command response:
exportclassHelloBehaviorimplementsBehaviorComponent{/**
* Get behavior (hello).
* @returns {BehaviorComponentInformation} Get behavior information
*/publicgetBehavior():BehaviorComponentInformation{return{name:'Hello',description:'Hello example behavior',// stub -> We don't need to "stub" any code for this example
commands:[{name:'get:hello',description:'Say hello',execute: this.getHello.bind(this)}]};}/**
* Say hello.
* @param {CommandControlOptions} options Options
*/privateasyncgetHello(options: CommandControlOptions){return'Hello';}}
In order to activate aura-behavior-manager in aura-bot, two environment variables must be configured with the following values:
AURA_SERVICE_ENVIRONMENT, with the value "DEV" and
DEV_AURA_BEHAVIOR_MANAGER_ACTIVE, with the value true.
With these two variables correctly configured, the BotBehaviorManager module will be added to the Orchestrator module list and a message will be shown in the startup process: ‘Behavior manager loaded’.
Defined behaviors
Below are the different behaviors defined in the Bot and the commands to be able to interact with them.
Those behaviors marked with ‘(*)’ are necessary for the BotBehaviorManager to work properly.
CommandBehavior (*)
It is the necessary behavior to be able to interact with the Bot’s behavior system, so without this behavior it will not be possible to send commands.
Stub:
Name
Stub destination
Description
Command control
beginDialogStep
Modify “beginDialogStep” function to detect and execute commands
Property behavior
It allows you to modify a list of environment variables without the need to restart the system. Currently, only the following list of variables have been enabled to be modified by the Behavior Manager:
AURA_ACCESS_TOKEN_EXPIRATION_MARGIN
AURA_MONGODB_BOT_COLLECTION_CONTEXT_INDEX_TTL
If you need to add new variables, you should use the ConfigurationManager.instance.get method on all sites where you can use that variable.
Stub:
Name
Stub destination
Description
Property
ConfigurationManager.get
Modify get function to change the value of a variable
Commands:
Name
Description
set:property
Set property
get:property
Get property
get:properties
Get properties. All properties configured by the user (conversationId)
get:defaultProperty
Get default environment property. The value of the current environment variable
get:defaultProperties
Get default environment properties. All current environment variables
Examples:
/bot:set:property AURA_MONGODB_BOT_COLLECTION_CONTEXT_INDEX_TTL 10# Set property value for conversationId/bot:get:property AURA_MONGODB_BOT_COLLECTION_CONTEXT_INDEX_TTL # Get property value for conversationId/bot:get:properties # All properties configured by the user/bot:get:defaultProperty AURA_MONGODB_BOT_COLLECTION_CONTEXT_INDEX_TTL # The value of the current environment variable/bot:get:defaultProperties # All current environment variables
Aura Bridge Behavior Manager
How to activate Aura Bridge Behavior Manager ?
In order to activate aura-behavior-manager in aura-bridge, two environment variables must be configured with the following values:
AURA_SERVICE_ENVIRONMENT, with the value "DEV" and
DEV_AURA_BEHAVIOR_MANAGER_ACTIVE, with the value true.
With these two variables correctly configured, the BridgeBehaviorManager module will be added to the Orchestrator module list and a message will be shown in the startup process: ‘Behavior manager loaded’.
Settings
Environment variables that can be configured to adapt the Behavior Manager:
Profile configuration file location for the behavior-manager
DEV_AURA_BEHAVIOR_SETTINGS_FILE_CRON_PATTERN
CRON expression associated with the reload time of the configuration file
Using the default value for DEV_AURA_BEHAVIOR_COMMAND_PATTERN allows you to run commands of the type:
/bridge:get:property DEV_AURA_BEHAVIOR_COMMAND_PATTERN
/bridge get property DEV_AURA_BEHAVIOR_COMMAND_PATTERN
/bridge unset proxies
The aura-behavior-manager allows to use a configuration file to manage profiles (currently disabled). This file is read at start-up and refreshed according to the variable settings DEV_AURA_BEHAVIOR_SETTINGS_FILE_CRON_PATTERN. The configuration file is stored in Azure Blob Storage. The path to this configuration file is built as follows:
A behavior allows defining a change in the behavior of the aura-bridge.
In order to allow an aura-bridge plugin to manage BridgeBehaviorManager commands, the interface BehaviorComponentInformation has been extended as follows:
This new pluginControllerStub field defines the decorator function for plugin controllers. The original function (endpoint controller defined in the swagger) will be passed as the last argument (the original parameter).
Enable a plugin to manage behavior commands
Only the Api or Processor (extends API type) plugins can manage Behavior Manager commands.
The only action by a plugin to enable Behavior Manager command controller is to complete the information from the behaviorController field defined in the PluginApi interface:
exportinterfacePluginApiextendsPlugin{/**
* If defined, the plugin can manage commands for the behavior manager.
* This only takes effect in development environments.
*/behaviorController?: BehaviorController;}exportinterfaceBehaviorController{/**
* Get request information.
*
* @param {express.Request} req Express request
* @returns {AuraBridgeRequestInfo} Request information
*/getRequestInformation(req: Request):AuraBridgeRequestInfo;/**
* Get text from input message.
*
* @param {express.Request} req Express request
* @returns {string} Text message
*/getTextFromMessage(req: Request):string;/**
* Send text message using client.
*
* @param {string} message Text message
* @param {express.Request} req Express request
* @param {AuraBridgeRequestInfo} requestInfo Request information
*/sendTextMessage?(message: string,req: Request,requestInfo: AuraBridgeRequestInfo):Promise<void>;}
The getRequestInformation and getTextFromMessage functions are required and necessary to be able to recognize and execute commands. The sendTextMessage function is optional and allows the user to receive feedback from the command execution.
Defined behaviors
Below are the different behaviors defined in aura-bridge and the commands to be able to interact with them.
Those behaviors marked with ‘(*)’ are necessary for the BridgeBehaviorManager to work properly.
CommandBehavior (*)
It is the necessary behavior to be able to interact with the aura-bridge behavior system, so without this behavior it will not be possible to send commands.
Plugin controller stub:
Name
Stub destination
Description
Command control
plugin controller
Modify plugin controller to detect and execute commands
This behavior will affect those plugins that define the behaviorController field in the API type plugin definition.
CommonBehavior (*)
It defines commands that are associated with BridgeBehaviorManager itself.
Commands:
Name
Description
get:conversationId
Get conversation id
get:all
Get all configuration associated with the current conversation
unset:all
Unset all cache information associated with the current conversation
PropertyBehavior
It manages information from the aura-bridge environment variables.
Currently, only the following list of variables have been enabled to be modified by the BridgeBehaviorManager:
AURA_BOT_APIKEY
AURA_BOT_ENDPOINT
AURA_BRIDGE_ENDPOINT
AURA_FP_WHATSAPP_ENDPOINT
AURA_BRIDGE_REQUEST_VERSION
If you need to add new variables, you should use the ConfigurationManager.instance.get method on all sites where you can use that variable.
Stub:
Name
Stub destination
Description
Property
ConfigurationManager.get
Modify get function to change the value of a variable
Commands:
Name
Description
set:property
Set property
get:property
Get property
get:properties
Get all the properties configured
get:defaultProperty
Get default environment property
unset:property
Unset property (If property name is not present, all will be deleted)
unset:allProperties
Unset all properties
How to redirect requests to your local Aura Bot
Set the property AURA_BOT_ENDPOINT with the next aura-bridge command and all requests in the conversation will be redirected to the local aura-bot. For example, at a local address using ngrok (https://1111-11-11-111-11.ngrok.io):
/bridge set property AURA_BOT_ENDPOINT "https://1111-11-11-111-11.ngrok.io/api/messages"
To see the answer, you need to modify the service URL of the activity with the aura-bridge enpoint of the environment.
ProxyBehavior
It allows redirecting requests made to aura-bridge to an alternative URL.
This redirection is based on the configuration of two parameters:
urlPath: String to match with the value of req.path in the original request.
endpointDestination: URL base where the petition will be redirected.
Plugin controller stub:
Name
Stub destination
Description
Bridge proxy
plugin controller
Modify plugin controller to perform a redirection of incoming request
This behavior will affect those plugins that define the behaviorController field in the API type plugin definition.
Commands:
Name
Description
set:proxy
Set new proxy. Params
get:proxy
Get proxy. Params [urlPath]
get:proxies
Get all proxies
unset:proxy
Remove proxy. Params
unset:proxies
Remove all proxies
PluginBehavior
It allows you to display information regarding the aura-bridge plugins.
Commands:
Name
Description
get:plugins
Get all loaded bridge plugins
get:stubPlugins
Get plugins with behavior manager active (can manage commands)
Use cases
How to redirect requests from WhatsApp to your local Aura bridge
Set proxy with the following aura-bridge command and all requests from WhatsApp (“whatsapp/messages”) in the conversation will be redirected to the local aura-bridge. For example, at a local address using ngrok (https://1111-11-11-111-11.ngrok.io):
/bridge set proxy "whatsapp/messages""https://1111-11-11-111-11.ngrok.io"
1.9 - aura-logging
aura-logging utility
aura-logging utility is used for controlling login in Aura
Introduction
aura-logging utility is a custom login tool across every Aura apps intended to prevent possible errors in execution time or unspecified login format.
The current logging format, including different fields and logging levels to be used across Aura components, is included in the Aura logging common format section.
emit function has been deprecated, please use logging method in future implementations:
{"time":"2019-04-05T11:51:09.325Z","lvl":"INFO","corr":"9c83c5d8-f3b5-4074-b475-28e2f528dd50","method":"GET","domain":"api.global-int.baikalplatform.com","path":"/userprofile/v3/users/123456677890","version ":"8.2.1","module ":"subscriptions-service","msg ":"User profile cannot be retrieved","origin":"10.0.0.1","drt ":45,"status":403,"error":"{\"code\":\"PERMISSION_DENIED\",\"message\":\"Client does not have sufficient permission\"}"}
Aura Bot
In aura-bot, the outgoing response message of the server should include these fields:
Field
Type
Format
Description
conversationId
string
string
Identifier of the conversation in DL
fromId
string
string
Identifier of the user in the channel
replyToId
string
string
Identifier of the message being answered
type
string
string
Type of the activity, in this version only message
locale
string
2 letters: lang-country
ISO culture code: en-gb, es-es, etc.
Aura NLP
In Aura NLP, the outgoing response message of the server should include these fields:
Field
Type
Format
Description
intent
string
intent.{name} or None
Intent resolved by Aura NLP server
domain
string
string
Domain to which the detected intent belongs
accuracy
float
Four decimals number, [0, 1]
Probability of returning the correct intent
Logging levels and usages
Level
Description
DEBUG
Log extra info and steps, for development purposes.
INFO
To follow the request with the most important steps and information
WARN
A non blocking error or special situation in the current request
ERROR
A blocking error in the current request
FATAL
A blocking error that avoids the normal behavior of the server, so it would be stopped.
Correlator policy
The correlator property in the log traces (corr) is mandatory in all levels but DEBUG, even though it is also highly recommended in that level.
We can use a correlator passed by in the original request (from other component), and we will use the same correlator in requests to other components (that allows them) in order to keep a cross-component correlator (the same correlator is used in the all the request-response flow, involving several APIs/servers/components).
If the correlator is not passed by the component creating the request, a new random correlator is generated at the beginning of the flow and is used from then on (even in requests to other components).
The correlator is used to relate all operations done in a request-response flow, that is all things done from the moment when the user starts the request untilaura-bot sends the last response (all operations logged must have the same corr).
There are a couple of exceptions to the last rule, as we are not able to get a correlator in all cases (some of them within a request-response flow, others outside it). Some special values are defined to know in which case the log trace is:
corr
Description
<uuid>
Normal case. The same correlator is used in all the request-response flow (passed in the original request, or random generated at the beginning).
no-correlator
The log trace does not have a correlator because it cannot have one (i.e., by technical limitations, such as HTTP request triggered by an event).
aura-system
The log trace is outside a request-response flow (such as server starting, server stopping, etc.)
null|undefined
These values are not permitted and are always considered a bug.
⚠️ Note: Do not abuse of the use of no-correlator, when the developer is lazy enough to get the current request-response correlator, and prefers to set no-correlator to avoid the bug detection when null or undefined. no-correlator means that it is impossible to get the correlator.
Standard logs
The log id must be:
The name of the class, if we are writing a class:
// to class AuraBridgeFlow
constlogger:AuraLog=newAuraLog('aura-bridge-flow');
The name of the file, if we are writing an utility file:
// to dialog whatsapp-utilities file
constlogger:AuraLog=newAuraLog('whatsapp-utilities');
The dialog identifier if we are writing a dialog:
// to dialog DisambiguationDialog
constlogger:AuraLog=newAuraLog(DisambiguationDialog.id);
Define the log outside the class to avoid attacks:
You can include an object as a string key to get more information about the case. In this scenario, avoid including very large objects to precent blocking the pods due to logs writing.
logger.info({msg:'Try to upload and validate attachments',fromId:from,auraId:auraUser.auraId,corr});
2 - Develop a bridge plugin
Develop a plugin and activate it in Aura Bridge
Guidelines for the generation of a new plugin and its activation/deactivation in Aura Bridge
Introduction
The process for the generation of a plugin and its activation or deletion in aura-bridge includes the following steps:
Execute the following command in the aura-bridge project path:
$ aucli bridge generate plugin
✔ Whats the origin of the process? (ex.: Directline, Whatsapp, ...)? · channel1
✔ Whats the destination of the process? (ex.: Directline, Whatsapp, ...)? · channel2
✔ The name of processor is correct? (y/N) · true
The following files will be generated in aura-bridge:
Add a swagger inside your plugin in the path: src/plugins/channel1-channel2-processor/swagger.yaml
openapi:3.0.0info:title:administration api to aura bridgedescription:Set of endpoints related to bridge administrationversion:1.0.0tags:- name:admindescription:Bridge administration pluginservers:- url:'http://localhost:8045'description:Local server- url:'https://svc-ap-current.auracognitive.com'description:ap-currentpaths:/aura-services/v1/testing/channel1:# include the path to the new pluginget:tags:- testing-channe1description:channel1 testing.operationId:getChannel1Messages# controller function name.x-router-controller:plugins# in path src/controllers/plugins.tsresponses:'200':description:OKheaders:{}'500':description:An internal server error has occurred.
Add a new function controller
Add all the controllers in the path:
src/controllers/plugins.ts
Every plugin to be loaded in aura-bridge must be defined in the plugin-config.json file.
⚠️ If this plugin has dependence with other plugins, they must also be added to the plugins list. If during the aura-bridge boot a plugin has a dependency with another not installed plugin, the bridge will fail in the boot process and will log an error indicating the lost dependency.
Add the plugin to the configuration file plugin-config.json (located at the root of the aura-bridge project):
To activate a new plugin during development, it is only necessary to add the plugin path in the plugin-config.json file:
["./lib/plugins/directline-service","./lib/plugins/directline-whatsapp-processor","./lib/plugins/whatsapp-incoming-processor","./lib/plugins/whatsapp-service",// New plugin location here.
]
⚠️ take into account the dependencies with other plugins, as explained in the previous section.
Activate a new plugin for production
To add a new plugin in production, it is necessary to define the list of plugins using the plugin-config.json key in the aura-bridgeConfigMap for deployment in Kubernetes.
Example of aura-bridgeConfigMap using a configuration file:
To remove a plugin, it is only necessary to delete the plugin path in the plugin-config.json file.
⚠️ In order to eliminate a plugin correctly, it is necessary to verify that no other plugins from those installed depends on the one we want to eliminate.
3 - Develop filters for Engine filter API
Develop a filter for Engine Filter API
Guidelines for the knowledge and generation of filters for Engine Filters API.
Introduction
A filter is a set of configurations and instructions that allow data to be grouped and queried in order to determine whether the flow of a use case should be changed at a given time. For example, if we have a use case that queries an LLM, we can create a filter to limit the use of these LLMs by user, or by channel at a given time.
For this documentation we are going to make a filter that limits the use of LLMs by user and use case, we will describe it later.
Engine Filter API
Anatomy of a filter
A filter is composed of a descriptive part, a configuration part, and an execution instruction part. Both parts are stored in JSON format. The service in charge of the storage is the Aura Configuration API. [TODO url]. The execution part is done by MongoDB instructions.
Descriptive Properties
Field name
Type
Required
Description
id
string
true
UUID that uniquely identifies a routing filter in Aura.
name
string
true
Name that uniquely identifies a routing filter in Aura.
description
string
false
Routing Filter description.
type
string
true
Contains the type of filter. Currently, there is only one type ‘userId’.
{"name":"preset-filter-stb-conversational-search","id":"4a879583-5f76-4e6b-87c1-6250e8743dda","description":"Limit the number of messages per user in a month for stb conversational search preset","type":"userId"....}
Configuration Properties
Field name
Type
Required
Description
entities
string[]
true
Contains at least one entity necessary to generate the data for the filter.
dataBase
object
true
Contains an object with the collections necessary to store and process data. See database object.
vars
object
true
Contains an object with the custom variables necessary for any phase of the filter. Encrypted field.
fields
object
true
Contains the field mapping for grouping and its relationship with the previously defined entities. Encrypted field.
sourceFilters
object
false
Contains an object to filter source data when loading from entity data. Encrypted field.
To see all the steps necessary to create and use the filter, let’s use the example we discussed in the introduction.
Let’s suppose that we need a filter so that the users have a limit of access to the LLM APIs per use case. We are going to set a limit of 150 interactions per month. And this filter should reset those results at the beginning of each month.
Create a filter. Steps
Filters use data generated for KPIs, so the most important thing before starting to define a filter, is to know which entity has all the necessary fields to be able to extract the necessary conditions for the filter. Note that currently only one Entity can be used per filter, in later versions this may change and support multiple entities.
For our example we will identify what data we need
Date of interaction
User identifier
Preset identifier
For our filter the only entity that contains all these data is GATEWAYMESSAGE
Iteration date -> MESSAGE_TM
User ID -> USER_ID
Preset identifier -> AURA_PRESET_NAME
The value of the preset to which we are going to add the filter will be: “ef3d0603-3fef-4109-a577-0ab92f9060df”.
Once the Entity has been identified we proceed to create the descriptive part of the filter.
"name":"preset-filter-for-users","id":"1a879583-5f76-4e6b-87c1-6250e874bbs","description":"Limit the number of messages per user in a month for LLM User Case","type":"userId"
As we have the entity we are going to use we set it:
We will set the limit to 150 and put it in a variable.
"vars":{"llm_execution_limit":150}
As the USER_ID field is optional, we will filter out those records that do not have USER_ID, i.e. USER_ID notEqual "".
This is done in the sourceFilters property. Although it is an array, it currently supports only one element. The available operators are: major, minor, equal and notEqual. SourceFilters also supports other data types for comparison, such as date or number. This is set with the cast: number|date property, otherwise the default is string.
We have already established all the values for the descriptive part and the configuration part, we must take into account that when we finish defining the execution part we will have to add some more data, such as indexes, if we want the filter execution to be done with the maximum performance.
Create Execution definition
Once a filter has been created, the KPI utilities will already know which entities have to send the Engine Filter API in order to process and group data and then validate the filters with them.
In our example, in order to group those data we need the dataFilterCollection collection that will contain the raw data extracted from the KPIs entity.
"forId": "USER_ID", -> itemId
"forMatchingField": "AURA_PRESET_NAME", -> fieldForMatch and valueForMatch
"forTime": "MESSAGE_TM" -> year, month, expires and seqId
In our example we have to look for those that have the valueForMatch equal to the one defined in fields MatchingValue and group the data by userId, month, year, counting the number of iterations, i.e. we are looking for something like this:
Before we start defining the execution properties, let’s explain the format in which they are defined.
JSON with PLACEHOLDERS
The JSON that compose the properties with execution parameters have placeholders to be able to inject variables and other values that have other properties.
Prefixes
vars: Refers to the properties defined inside the vars object. Example ````vars.llm_execution_limit``` in our example will have the value 150.
fields: Refers to the properties defined inside the fields object. Example: ````fields.MatchingValue```.
doc: It refers to the document returned by mongodb in the previous execution. It will only appear in the summary property since it is formed with the results of a previous query. Example: ````doc.itemId```. In order to use these properties they must be inside the $project property of the match property.
ctx: Contains variables that can be used to insert in execution properties, these are:
ctx.day: Today’s day.
ctx.month: Current month.
ctx.year: Current year.
ctx.lastSeqId: Last seqId processed to create summaries.
ctx.expiration: Expiration time of the data to be generated.
Data Types
number: Numeric, integer or float type. Example: {{ctx.month|number}}.
date: Date type. Example: {{var.EDate|date|date}}.
expires: Current date + placeholder value. Example: {{ctx.expires|expires}}.
string: String, if nothing is set it is the default string. Example: {{doc.itemId}} or {{doc.itemId|string}} .
lower: String in lowercase. Example: {{doc.itemId|lower}}.
upper: String in uppercase. Example: {{doc.itemId|upper}}.
trim: Removes whitespace from a string. Example: {{doc.itemId|trim}}.
Variables
- _DATE_NOW_: Contains the current date. Example: "updatedAt":"{{__DATE_NOW__}}".
Create execution properties: match, summary and summaryFilter
Once we have data in the raw data collection what we need is
1.- Get the data that has not been processed yet filtering by a certain condition and grouping them in this case by ItemId, month, year, and finally returning a data model to generate the summaries. This is achieved in the match property:
In our case this would be to select those elements that the field “AURA_PRESET_NAME” has the value “ef3d0603-3fef-4109-a577-0ab92f9060df” and whose seqId is greater than the last processed seqId.
The doc prefix refers to the result of the previous execution of match. Once the summary has been executed, we will already have in the dataSummaryCollection data to query if they comply or not with the filter.
3.- Obtain the data that meet the filter. summaryFilter
In this case it looks for records that comply with the current month and year and that exceed the limit configured in vars.llm_execution_limit. The filter engine API stores the elements that exceed the filter in a cache each time the dataSummaryCollection is updated.
Check filters
Filters can be queried via Configuration API calls: https://ENV.auracognitive.com/aura-services/v2/routing-filters/FILTER_ID/items/ITEM_ID, or within a context filter.
The target audience of the document are developers of global and local teams contributing to Aura Global repositories:
aura-bot developers
aura-bridge developers
aura-services-api developers
…
⚠️ This document assumes that the developers already have permissions to access Telefónica organization in Github.
Branches used by Aura
Aura uses 2 different types of branches:
Eternal or pseudo eternal branches
These are protected branches, that MUST be always deployable in a given environment.
Protected means:
Only code owners can push code to them
All changes MUST be pushed through a pull request, that MUST pass the continuous integration checks and count on, at least, 2 approval reviews of code owners.
These branches are:
master: it holds next release code. It MUST be deployable in the global development environments.
main/*: it holds the development of a unique User Story or set of User Stories.
release/*: it holds the given release code. It MUST be deployable in global development environments and also in PRE and PRO of the OBs involved in this release developments.
hotfix/*: it holds the resolution of an issue or set of issues happening in a pre or pro environment.
Temporary branches
As any change in eternal branches MUST be done though a Pull Request, it means that they must be done in a temporary or feature branch.
These branches are usually named after the type of change they are going to hold: feat/my-new-feature or fix/fixing-that.
This format is <type>/<description>[#issue_number] where:
type is one of feat, fix, chore and docs
description should be a short description, dash-separated words, of the change. If using Jira as issue manager, add the number of the issue being resolved after a #.
Process for generating a new branch
First of all, clone the repository and choose an eternal branch as base branch for your feature:
Split your changes in small tasks (max 1 day per task)
Use a feature (feat, fix, chore, …) branch per task
Do small commits with spotted functionality
Do small PRs with all the code of a given task
When asking a PR remember to include an explanation of the why, how, what has been done in the PR, in order to help reviewers to understand the change.
global, can be omitted, if affects all the packages or it’s a basic change.
the affected package in a multipackage repository.
the affected library, plugin or dialog in library repository.
if it affects a concrete feature in the repository: logging, kpis, authentication, etc.
it can also be omitted in global cases, in minor changes, etc.
Description: indicative sentence in lower case.
Body: only if description field is not enough to understand the changes
Folder layout
In general, all the repositories containing typescript code should have an src folder with all the typescript source files there, and those source files will be transpiled into javascript into folder lib.
Unit tests
Currently aura-bot team is moving to Jest as unittest runner, so a couple of versions of unittest can be found:
Projects running with mocha, such as aura-bot-platform.
Test files should be included in the same folder as the files to test, with a trailing .spec.ts in the file name (these files should be .npmignore, to avoid being in the final package).
Test assets required for tests (.json files, images, .env files, etc.) will be placed in a test folder (that should be .npmignore too), so they will be available for typescript/javascript files by getting the path ../test or similar.
Projects running with jest, such as aura-mocks-server.
There will be a specific folder called test, at the same level than src and that will hold all the unit tests following the same structure than src. As with mocha tests, they must be ignored in the production transpilation and the files published in npm.
More about src/ folder layout in aura-bot-platform can be found in this file.
4.1 - Access to Github repositories
Request access to Aura repositories in Github
Steps for L-CDO developers willing to access to aura-bot and aura-nlp Github repositories
How can L-CDO developers access to Github repositories?
All local CDO developers are enrolled into a specific team in Github for each country, which provides the access to the relevant Aura repositories in Github, such as https://github.com/Telefonica/aura-bot-libraries.
To gain access to your local CDO team in Github, you have to fulfill the following requirements:
Having a personal account in Github.
Two factor authentication (2FA) enabled for the account.
Your personal profile page in Github must fulfill the following requisites:
3.1. It must include a personal image, not an avatar, distinct from the one Github assigns by default.
3.2. It must include your name and surname.
3.3. It must include a “@telefonica.com” as the public personal email account associated to the Github profile.
ℹ️ Please, have a look at https://github.com/Korreca as an example of a valid personal profile page in Github.
Once the previous requirements have been successfully fulfilled, please send a message to the “[APE] AURA-LCDO-XY-GENERAL” chat in Microsoft Teams (where “XY” should be substituted by the 2 letters code associated to your country) requesting access to the Aura repositories in Github.
A member of the APE team will request your registration into the Telefónica organization in Github. Afterwards, you will receive an email message including your personal invitation to be registered in it.
Once you have been included in the Telefónica organization in Github, a member of the APE team will enroll yourself in the appropriate “Aura Development XY” (where “XY” should be substituted by the 2 letters code associated to your country) which will grant you access to all the relevant Aura repositories in Github.
4.2 - Contributing with code
How to write code in Aura Bot global repositories
Guidelines for developers willing to contribute to Aura Bot global repositories writing code
Introduction
The aim of this document is to establish common minimum rules to contribute in Aura Global repositories: code linting, used libraries and guidelines for working over main and master branches
Programming Languages
aura-bot code is written in typescript 4.3.4 that is a programming language developed by Microsoft on top of Nodejs. Currently we use node engine version 14 and npm version 6.
This means that we follow typescript rules of naming, but we have also added some extra ones, to unify our developments:
local variables: always in camelCase.
files: always in lower case letters, dash-separated.
classes: in PascalCase.
interfaces: in PascalCase, they do not start with I.
enums: named in PascalCase, each item also in PascalCase.
library or server names: always in lower case letters, dash-separated.
environment variables: in capital letters, underscore-separated.
functions: always in camelCase.
asynchronous programming: use await/async pattern when possible, avoid chains of promises.
adding a feature to an existing server: check the already existing instances of Singleton components and use them directly. Do not create a new instance, this will lead to an error during the start-up or in runtime.
All servers count on a ConfigurationManager where all its environment variables can be obtained.
Try to use the best programming pattern for each feature, applying always Single Responsibility Point defining small modules, functions or classes that do only one specific task. Find more references for programming patterns:
Database: mongodb, currently version 4.2 but moving in next releases to version 6.
UnitTest: we use mocha for already existing modules, but we are moving to jest, so all brand-new modules must use jest.
Mocks-Server: we have developed a mocks server (aura-mocks-server), that mimics the behavior of all external components (such as kernel, nlp, etc.). If something is missing, add it.
API definition:
We use API FIRST design. i.e. firstly we define the swagger of the API and then we build the routing of the API using the swagger directly from expressjs and express-openapi.
We use OpenApi v3 for swagger definition
We generate the client of the API and the related models automatically after a change in the swagger file.
Client package is published on the fly to the correspondant npm registry (github during the development phase and npm for master and release), so its code never exists in the server repository.
Models package is updated in the server repository, but just using the definitions in the swagger, no extra code should be written.
Swagger file must be called swagger.yml and must be created in the root of the server, at the same level of its package.json.
Each server that needs a client must have a client folder with just the package.json of the client, the rest of the code is autogenerated by aura’s CI system.
HTTP requests: we use superagent library to handle HTTP request, both in our generated API clients and in those done directly from the servers. Currently we use version 5.2.2, but we are moving to version 8 in following releases.
request and request-promise are strongly prohibited.
Work on a main branch
When you work on a main branch, the libraries are published on Github.
To install the dependencies of a package, you need the following steps:
Learn how to migrate a folder within a Git repository to another repository
Move files with history
This document explains how to migrate the history of the files to the new repository (that could or could not be desirable).
This process tells how to move a part of a repository (folder) to a new repository, maintaining the history of the files. A new fresh copy of the origin repository is required, as it will not be usable anymore.
These commands will prepare a repo with only the commits from a specific folder (removing anything else):
# Clone a fresh copy of origin repo and enter into itgit clone <giturl-repoA>
cd <repoA>
# Optional, to avoid pushing to the wrong remote repositorygit remote rm origin
# From the repo source, remove all the files and history outside the foldergit filter-branch --subdirectory-filter <folder-name> -- --all
Now, we can import in the new repository following these steps:
# Optional, clone a fresh copy of destination repo (or could use a previously existing one) and enter into itgit clone <giturl-repoB>
cd <repoB>
# Add a new remote pointing to local path with repo Agit remote add repoA <local-path-repoA>
# Pull from previous local repository the dialog files, branch master (replace if using a different one)git pull repoA master --allow-unrelated-histories
# Optional, create a new branch to merge intogit checkout -b <destination-branch>
# Remove local remote, not required anymoregit remote rm repoA
4.4 - Manage Jira
Guidelines for Jira management
Learn how to handle user stories and tasks in Jira
Analysis
Description
Tool
State
Create Analysis Task
JIRA
[X]
Create Analysis Documentation
Confluence
[X]
. . . Use cases
[X]
. . . Data model
[X]
. . . Component architecture
[X]
Create “doubt repository”
JIRA/Confluence
[X]
—
Have all doubts been resolved?
[X]
Has the result of the analysis been presented?
[X]
Have several alternatives been evaluated?
[X]
Has the chosen option been agreed upon?
[X]
Chosen solution meets the performance requirements?
[X]
Development
Description
Tool
State
Create branch “feat” for each task
GIT
[X]
Create unit tests
GIT
[X]
Create Documentation (README)
GIT
[X]
Add lint rules (standard)
GIT
[X]
Create jenkins tasks (if they don’t exist)
JENKINS
[X]
Check for missing jenkins tasks
JENKINS
[X]
—
Does the code pass all unit tests?
[X]
Do the unit tests cover 80% coverage?
[X]
Does the code comply with the lint rules?
[X]
Have the Jenkinsfile and pipelines been checked?
[X]
Is the README documentation up to date?
[X]
FAQs
What to do if you have to integrate with another development team?
If, for example, we are going to integrate something with the Cognitive Team, such as the new suggestions API, we will be talking to them integrated in the same team all the time.
But the moment we need them to pass us their swagger, whether their component is deployed or the update deployed, we will give them a task to do so.
How to upload the code to git?
To upload the code to git, Pull Request to master or release/* is performed depending on the moment.
We upload it when we have finished, with a disclaimer: if we are dependent on other teams that are going to call our API or use the modification for something, we will be especially careful and upload the changes when they are necessary and not to break their development.
Do I have to notify QA at the end of coding?
Yes, you have to notify them when a new feature or a fix is deployed in the corresponding environment.
If it is bug or complete US, passing them to resolved they are already warned, but I like more to give them a touch and tell them.
In addition to when we upload the code, I like to share the analysis and design pages with them, because the discussions are usually very productive.
Should we notify someone about the generated documentation?
We must notify María Eugenia, who is the documentary maker. The ideal is to leave indicated in each US/Bug which documentation has been modified with its development. María Eugenia takes the doc from aura-docs, from Confluence or from the corresponding repo and includes it in the product versions.
How do the tasks change states?
We must pass the US/bugs from “new” -> “in progress” -> “code review” -> “resolved”. Tasks in the “resolved” state go “closed” when QA validates them.
How do I change version of component or library?
We do this process with each release or phase change. The future is that everything is done automatically by Jenkins.
4.5 - Source Folder Layout
Source folder layout
Description of Aura Bot Platform src/ repository
Introduction
The repository aura-bot-platform has a folder layout inside src folder, to organize the code according to functionality groups.
In src root folder, only main files should be placed, such as index.ts, or main server file server.ts (app.ts in aura-bot).
src/ sub-folders
src/bots
Currently, only the file aura-bot.ts is included here, that is the main ActivityHandler (kind of replace of previous UniversalBot).
In a near future, more Bot Builder adapters could be located here.
src/config
In this folder, files that load configurations are included. These files are required in other places, such as channel configuration, env vars, etc.
src/db
In this folder, files that manage and connect to databases are located, such as MongoDB.
src/events
Files handling events (such as ModuleObserver subclasses) should be placed in this folder.
src/dialogs
This folder contains aura-bot main dialog and custom prompts, as use case dialogs will be located in libraries, loaded as dependencies.
src/make
In this folder, the code related with joining all the library-specific data with global ones is included, corresponding to the make-up process.
src/middlewares
This folder contains abstract middleware base classes, and specific middleware implementation (final classes).
Files located within this folder will have exported types and interfaces required in different parts of the code (types and interfaces required only within a file could be self-contained).
src/modules
This folder contains independent code blocks, that could be exported as a reusable packages if required in different components, such as cache manager, locale manager, etc.
src/plugin
This folder contains the modules in charge of loading plugins: charging dialogs, middlewares, delivery configuration for these components, etc.
src/routing
Here, code related with intent-to-dialog routing is located, as this is not part of Bot Builder anymore.
src/utils
This folder contains utility classes and methods that are not part or any other block. We should maintain this folder organized and tidy, to avoid lots of unspecific files.
5 - Upgrade Aura Bot
Upgrade Aura bot to node 18 and its dependencies
Guidelines for the upgrade of aura-bot to node 18 and the update of its major dependencies, in order to avoid end-of-life of current versions
Introduction
Current version of node being used is 14 LTS, that finished active support in October 2021 and security support in April 2023.
In order to avoid the inconveniences of working with an old version, without security updates, we will update to current latest LTS version 18, with active support until October 2023 and security support until April 2025.
Moreover, some major dependencies are also upgraded in order to keep the latest available versions, with new features and security updates.
:warning:
The upgrade of node and dependencies in aura-bot comes together with the migration to Bot Framework 4.19..
The latter implies that local use cases must be migrated to this new version to be operative. For this purpose, OBs must follow the guidelines Migrate use cases to Bot Framework 4.19.
Guidelines for upgrade
In this section, a list of mandatory and recommended changes will be explained, but consider applying them both, as it will prevent future incompatibilities.
Mandatory changes
Docker base
Docker base image was changed from node:14.17.1-alpine3.13 to node:18.16.0-alpine3.17, in bot Dockerfile and mini-bot Dockerfile.
If using a different container on deploys, it must be upgraded to node 18.
Internally, the jenkins slave was set to aura-node18-ubuntu22 on Jenkinsfile (only affects to the global team).
Node types update
Package with node types must be upgraded to version 18 in the package.json file:
{"@types/node":"^18.16.0"}
Upgrade Typescript
Typescript was upgraded in aura-bot to version 5, so it must be also upgraded in all packages and libraries. For this purpose, modify the typescript dependency in the required package.json files:
{"typescript":"~5.0.0"}
Update package-lock.json
After upgrading node version to 18, npm is also upgraded to version 9.5.1, and the package-lock.json is obsolete. Running a one-time npm install will upgrade lockfileVersion to 3, and the required content, such as sha512 integrity hashes instead of sha1.
It is also recommended to delete the whole file package-lock.json before npm install, to upgrade also dependencies to newer versions.
Recommended changes
In this section, other optional upgrades are shown, highly recommended in order to avoid future compatibility problems. We encourage applying them, in addition to the mandatory changes.
mongodb-memory-server
It is recommended to upgrade mongodb-memory-server to the latest version 8.12.2, due to deprecation of current version and, moreover, in order to prepare the code to near future upgrade of MongoDB (from version 4 to version 6). The upgrade requires a minimal code update.
npm-check-updates
It is also recommended to upgrade npm-check-updates to latest version 16.10.9, as current version has a transitive dependency with an obsolete node-gyp that causes incompatibility when using Python >= 3.11, that is the default version in Ubuntu 22.
6 - Release Train Manager
Aura Release Train Manager
Release Train Manager (RTM) implementation and architecture
Introduction to RTM development methodology
The “release train” software development methodology is a way of planning the delivery of software according to a predetermined and regular schedule, as if it were a “train timetable”. This schedule is public for all the teams that must contribute to the delivery and represents a commitment on their part, as the schedule must be adhered to.
If we use the metaphor: “the train will take the passengers who are ready to travel”, i.e., only the software that is ready to be integrated will be integrated. For this it is necessary to work on the different functionalities in parallel and in isolation without affecting the rest.
We start from a prioritized backlog. The tasks will enter the implementation phase in order of priority.
Github is used for the organization of the process. See here the details of Github as orchestrator.
The file and properties for the configuration of branches is included in Branch configuration.
Learn how locales files and the POEditor project are managed in RTM: Managing locales.
RTM overview
The important thing when we are working on a new feature is to be able to completely isolate the development, environment and testing. To do this, environments will be generated per feature and git branches will be used for the different repositories involved.
RTM Overview
INIT: initial processing of a feature (select repos involved, create branches on those repos and create environment).
TF (Tests Feature): daily smoke tests on feat/new-feature branches.
TFPR (Tests Feature PR): full regression tests on the master merge and the feat/new-feature branch.
TM (Tests Master): complete daily/weekly regression tests on the master branches.
PR Fx (PR Feature): PRs on the feat/new-feature branch.
PR Mx (PR Master): PRs on the master branch.
6.1 - Github as Orchestrator
Github as orchestrator
How to use a Github repository to orchestrate the Release Train Manager processes
Release Train Manager Repository
To organize the work, we use a Github repository. This repository contains classic branches for the RTM development cycle itself and a set of branches reserved for RTM orchestration. The default base branch of the orchestrator is master.
The classic branches contain the prefixes:
feat/description
fix/description
doc/description
etc.
The specific branches for the orchestrator contain a descriptive name and the JIRA identifier:
main/name_reference#id: To orchestrate new features.
release/name_reference#id: To orchestrate releases.
hotfix/name_reference#id: To orchestrate hotfix on a release.
fasttrack/name_reference#id: To orchestrate features on a release.
To manage the RTM processes, we use Pull Requests (PR) in this repository. Each of these PRs has a base branch and a target branch, depending on the operation to be performed:
This PR is generated automatically when a commit is made for the first time in one of these special branches.
Pull Request Body
In the body of the generated PR, all the configuration information for the branch is displayed and the Pull Request will contain all the history of processes executed on it in comment mode.
An example of PR body of a main branch
The information for the PR body is obtained from a configuration file and is updated as the different phases of the orchestrator are executed.
This folder contains Github Actions necessary to perform all operations on repositories, data persistence and communications with Jenkins, among others.
resource/templates
This folder contains templates for the branch-config.hml file for each branch type:
It contains the tools needed to build the docker actions and images.
src/toolkit
It contains the code SDKs as a complement to the Github actions. These utilities will eventually become actions someday.
src/toolkit/nodejs
It contains the NodeJS toolkit. More information in [NodeJS Toolkit].
⚠️ NodeJS toolkit section WIP
.github/workflows
We define stage as a set of steps to be performed in a coordinated manner by each team. Until all the teams involved in the phase have completed all the steps of a phase, it will not be possible to move on to the next phase.
For example, for example, we can define the INIT phase in which the teams have to create the specific branches in their repositories and activate an environment where they can deploy. We would have that each team would have an INIT phase in its configuration, with a step to generate the branch or branches in its repositories, and the Devops team would also have another set of steps to prepare the environment.
In order to synchronize the work between the teams and perform these operations we will use Github Workflows. These workflows are explained in the [Workflows] section.
⚠️ Workflows section WIP
6.2 - Branch Configuration
Branch configuration
Description of the process for the configuration of branches in the Release Train Manager
Branch configuration file
The configuration of branches is defined in the branch-config.yml file.
Each stage defined in the schema is fully described in the Stages section.
Protection that will be applied to main branch when created in the RTM repository. This configuration can be partial and will be merged with the default configuration. All the documentation concerning the protection of a branch can be found at Github
Default configuration property, this configuration will be merged between custom protection configurations of teams or main branches. All the documentation concerning the protection of a branch can be found at Github
Jira URLs where the tasks that the branch solves once the life cycle is completed are
Fake data
true
Team Object
It contains a set of common properties and others customized by each team. In the following table, the properties that are shared by all teams are described.
Security configuration of the branches to be generated in the repositories. This configuration can be partial and will be merged with the default configuration. Example:
List of custom tasks to be completed before closing the PR for the branch. These tasks will be converted into a checkbox component in the body of the Pull Request
Fake data
false
Team Bot Custom Properties
Description of the bot’s custom properties.
Property
type
description
default
mandatory
update_dependencies
Object[] of {name: string, version: string}
List of objects with name and version. This property is used to update dependencies on the modules in the repositories defined by aura-bot. All the modules that use one of the libraries in the list, will be updated to the version that has been specified.
[]
false
dependencies
Object[] of {name: string, version: string}
List of objects with name and version. This property contains those @telefonica libraries that are needed in the Github library repository to isolate dependencies. At the INIT stage, the system will pick up that library from NPM and publish it to Github.
[]
false
node_version
number
Version of node to use in the bot’s Workflows.
14
true
tools
Object
It currently contains only one property, POEditor, which specifies the creation of a separated project to manage the localization strings.
Description of the different stages for main branches in the Release Train Manager
The following diagram shows the Release Train Manager stages for main branches:
All phases of the main branches are managed by the workflow:
.github/workflows/push-to-main.yml.
name:Push To Mainon:push:branches:- "main/**"- "hotfix/**"jobs:load-configuration:######### INIT #########prepare-init-process:protect_branch_main:init-bot-module:init-devops-module:init-legacy-module:init-models-module:init-nlp-module:init-training-nlp-module:init-qa-module:########## START ##########start-bot-module:start-qa-module:##################### PREPARE TO MERGE #####################block-close:prepare-to-merge-bot-module:prepare-to-merge-training-nlp-module:prepare-to-merge-qa-module:prepare-to-merge-legacy-module:prepare-to-merge-devops-module:########## CLOSE ##########close-bot-module:close-devops-module:close-qa-module:close-legacy-module:close-nlp-module:close-training-nlp-module:############# FINALIZE #############finalize-bot-module:process-result:
Main Init stage
In this phase, the branches are generated in the repositories and, if there is a configured environment, it will be deployed. As in every stage, each team defines its own steps, but some of them are shared and executed by any of the teams.
Common steps
Create branch
This step gets the list of repositories of the team (from branch-config.yml file, teams.*.repositories) which step is being executed and creates the corresponding main branch in all of them. If the branch already exists, it just goes on.
Protect Branch
Once all the initial changes are applied in each package of each main branch, they are protected to force the creation of a Pull Request to merge changes on them. It is executed after every stage of the RTM, because it needs the branch to be unprotected during its execution, to make changes automatically.
Unprotect Branch
RTM stages need to remove repositories’ branches protection to apply changes automatically in the code. It is executed at the beginning of almost any stage of the RTM.
Check conflicts
This step is executed during prepare-to-master stage to check if there is any conflict between master and the main branch in any configured repository. Any conflict will be reflected in the RTM as a message in the corresponding PR.
Init Bot Steps
Bot: Install dependencies
In this step, the dependencies defined in teams.bot.dependencies are installed. The tgz of the NPM module is obtained and installed in the Github repository. This is necessary because the main branch works in isolation and uses Github as the main repository.
Bot: Store Locales
In this step, all the locales files of every package in every repository are stored in DB. They are stored in the collection called branch-locales.
Bot: Prepare Repository
In this step, all the dependencies of every package in every repository are updated. Internal node packages (those belonging to @telefonica NPM organization) in master branch are read from npm registry and a x.y.z version is used. But each main branch uses for the internal dependencies a private GitHub npm registry and the dependencies are tagged with the decodedName of the branch: version: 7.5.0-main-my-feature.0.
Packages are read in order, so first those packages without internal dependencies are processed. They are published in GitHub registry and those packages depending on them are then processed, going on recursively.
Bot: Create Changelog
This step is responsible for creating the changelog scaffolding for the current version.
Bot: Install Tools
A new project is created in POEditor to include the language resources to be contained in this main branch. The project is created by auradev@tid.es user and its name will be the branch decoded name: main/feature-desc#id -> main-feature-desc-id. This project is intended to contain all the changes (creations or updates) of locales for the issues being resolved in this main branch.
Main Start stage
In this phase, all the components configured in branch-config.yml and that were prepared during the init stage will be deployed if an environment has been configured for the current branch. Afterwards, the sanity test plan is launched just to validate that the environment is properly settled. If no environment is configured (for instance, if a branch is just to review documentation or to prepare something specifically for a given OB, such as language trainings) tests are not executed.
Start Bot Steps
Bot: Trigger Deploy Components
A push is launched in the repositories of the components to force them to be deployed from CI job running in Jenkins. After each component is deployed, Jenkins sends a notification to GitHub to inform of the status, that is updated in the RTM API database.
Bot: Validate Deploy Components
This step is in charge of validating that all the components that are configured for the current branch have been deployed. It reads the information from the RTM API database that is updated with every notification from Jenkins, as explained in the previous step.
Main Prepare-To-Merge stage
This phase aims to merge master changes in the current main branch, deploy the updated version in the branch in configured environment, if exists, and launch the complete functional test plan to validate the full code base before merging to master.
Prepare-To-Merge Bot Steps
Bot: Update Locales
This step is in charge of updating the locales files in all repositories doing a merge into the main poEditor project and Aura-Bot PoEditor project.
Bot: Merge and Update Versions
Once the eventual conflicts have been resolved in all repositories configured in the branch, the content of master and main branches are merged together. In the happening of any change, tagged versions are automatically updated to provoke a new deployment of the components.
Bot: Publish Modules
If new versions have been generated in any of the bot npm libraries, a new version of them is published and updated in all the components.
Main Close stage
In this phase, all the components will be deployed in master and the libraries published in NPM. The versions of the modules will be updated if the repositories have been changed along the development in the main branch.
Close Bot Steps
Bot: Prepare versions to master
As already mentioned, bot versions in master are not tagged but in x.y.z format. This step calculates the next version of each library and component automatically depending on the existing version in master, the original semantic version and the number of changes in its tagged version in the main branch. If a library or a component has been updated in a main branch, no matter the number of commits needed for the change, its semantic version in master will be increased by 1 in the minor counter. For instance, if the version in master for library my-library is 6.2.0, and my-library has some changes in the current branch, its version in master after merging the main branch, will be 6.3.0.
Bot: merge to master
If new versions have been generated in the libraries, changes are merged and the libraries are published in @telefonica private npm registry.
Finalize step
In this phase, all the branches created, packages installed in Github and the POEditor project are deleted.
Finalize Bot Steps
Bot: Remove Main Branches
The main branch created in the repositories is removed.
Bot: Uninstall Tools
This step removes the POEditor project created for the language resources of this main branch.
Bot: Remove Packages
This step removes the packages created in the branch for each library.
6.3 - Managing locales
Managing locales in RTM
How locales files and POEditor project are managed in RTM.
How to manage locales
To avoid updating the locales with the base project in the main branches in the RTM INIT process, there is a job store-locales, that is responsible for storing the locales of each component/library in the branch-locales collection with the following structure:
{"branch":"main/fake","authentication-api":{"de-de":"{\n \"authentication:authenticator-error.description\": [\n \"Bei der Authentifizierung ist ein Fehler aufgetreten. Bitte versuche es später noch einmal\"\n ],\n ....","es-cr":"{\n \"authentication:A1004.authenticator-error.description\": [\n \"Parece que has olvidado tu usuario o contraseña de acceso a Mi Movistar.\"\n ] ....}"},"aura-bot-platform":{"es-cr":"{\n \"context-filter:multimsisdn-users-not-allowed.onboarding\": [\n \"Pulsa el micrófono y di lo que quieras. Consulta la ayuda para saber qué cosas puedes preguntarme.\"\n ],\n \"context-filter:user-info-not-accessible\": [\n \"Sorry\"\n ],\n \"core:bypass.close.words\": [\n \"cancelar\",\n \"cerrar\",\n \"salir\",\n \"desconectar\"\n ],\n \"core:empty.response\": [\n \"Ok, estarei aqui sempre que você precisar.\"\n ] }\n","de-de":"{\n \"context-filter:multimsisdn-users-intent-not-allowed.text\": [\n \"Entschuldigung, auf diese Daten kann ich aufgrund deines Vertrages nicht zugreifen. Zukünftig kann ich das sicherlich, aber bis dahin kann ich dir mit Fragen rund um o2 Services weiterhelfen. Wenn du mehr Informationen über deinen Vertrag erhalten möchtest, gehe einfach in deinen [Mein o2 Bereich](www.o2online.de/meino2).\"\n ],\n \"context-filter:user-info-not-accessible\": [\n \"Sorry\"\n ],\n \"core:bypass.close.words\": [\n \"cancelar\",\n \"cerrar\",\n \"salir\",\n \"desconectar\"\n ],\n \"core:empty.response\": [\n \"Ok, estarei aqui sempre que você precisar.\"\n ] }\n"}}
During the development of the main branch, if it is necessary to update the locales of any component, it will be done using the locale-update script, which will have been updated by the following locale-importer command in the creation phase of the branches of each component/library:
aura-locale-importer -u -s Db -pkg <package_name> -m library -b core -d ./locale -f -mb <main-branch-name>
The environment variables SECRET_TOKEN and BRANCH_CONFIG_APIKEY are required for its execution.
This command will do a merge between the local files stored in DB for that component/library and the resources created/updated in the POEditor project main-branch-name generating the new locales files.
In the execution of the prepare-to-merge of the bot, an update-locales step has been added, which updates the locales of each component/library by merging the content of the base project in POEditor (Aura-Bot) with the POEditor project of the main branch.
Endpoints locales
For the management of the locales in BD, the following endpoints have been created in aura-release-train-branches:
- /branches/{branch}/locales
- /branches/{branch}/locales/{package}
6.4 - API definition
Release Train Manager API definition
Description of Release Train Manager API
This API is used to manage the release train process and stores both the configuration and state of all the main/hotfix/release/fasttrack branches handled by the release train manager, but also the status of the environments: component versions deployed, if any component has a new version waiting to be deployed, etc.
The API is used both from the release train manager and from the Jenkins CI jobs themselves.