Aura – mongodb

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Redis Mongo Synchronizer environment variables

Notation

List of environment variables defined for aura-redis-mongo-sync component

Properties marked in bold are mandatory.
Properties marked in italics are optional.

Property	Type	Description	Modifiable by OB?
AURA_LOGGING_EXTRA_TIME_METRICS	Boolean	Extra time metrics for logging. By default: `false`.	No
AURA_LOGGING_LEVEL	String	Level to be used in monitoring logs, from more to less verbose: `'TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL', 'OFF'`. By default: `INFO`.	Yes
AURA_MONGODB_BOT_COLLECTION_CACHES_INDEX_TTL	Number	Lifetime, in seconds, for the index of the caches collection. By default: `3600`	No. Only if validated with Aura Global team, because it is related to other values.
AURA_MONGODB_BOT_COLLECTION_CONTEXT	String	Name of the collection where the context is stored. By default: `aura-context`.	No
AURA_MONGODB_BOT_DATABASE	String	Name of the database where the bot collections are stored. By default: `aura-bot`	No
AURA_MONGODB_PASSWORD	String	Password of the user that accesses MongoDB.	No. Only if MongoDB credentials must be changed.
AURA_MONGODB_POOL_SIZE	Number	Pool size of the MongoDB. By default: `60`.	No.
AURA_MONGODB_SSL	Boolean	Flag to indicate if MongoDB connection uses SSL or not. By default: `true`.	No
AURA_MONGODB_URI	String	URI of the MongoDB	No
AURA_MONGODB_USERNAME	String	Username of the MongoDB	No
AURA_REDIS_CONFIGURATION_PREFIX	String	Prefix for Redis configuration keys. By default: `aura-config:`	No
AURA_REDIS_CONTEXT_CACHE_PREFIX	String	Prefix for Redis context cache keys. By default: `context:`	No
AURA_REDIS_SUBSCRIPTIONS_EVENTS	String	Comma-separated list of the types of Redis messages to trigger subscriptions events. By default: `expired,evicted`	No
AURA_REDIS_CONTEXT_CACHE_SHADOW_KEY_PREFIX	String	Prefix for Redis shadow key cache keys. By default: `shadow-key:`	No
AURA_REDIS_CONTEXT_CACHE_ACTIVE_CONTEXT_PREFIX	String	Prefix for Redis active context cache keys. By default: `active-context:`	No
AURA_REDIS_CONTEXT_CACHE_LOCK_PREFIX	String	Prefix for Redis lock context cache keys. By default: `lock-context:`	No
AURA_REDIS_CONTEXT_CACHE_MIN_SHARD_BY_NODE	Number	Number of minimal shards assigned to a service. By default: `1`	No
AURA_REDIS_CONTEXT_CACHE_CHECK_INDEX_SHARD_INTERVAL	Number	Number of seconds to wait before recalculate shards to subscribe events. By default: `60`.	No
AURA_REDIS_CONTEXT_CACHE_CHECK_UNPROCESSED_INTERVAL_MIN	Number	Minimum number of seconds to wait before checking unprocessed contexts. By default: `10`.	No
AURA_REDIS_CONTEXT_CACHE_CHECK_UNPROCESSED_INTERVAL_MAX	Number	Maximum number of seconds to wait before checking unprocessed contexts. By default: `120`.	No
AURA_REDIS_CONTEXT_CACHE_CHECK_UNPROCESSED_FACTOR	Number	Factor to increment or decrement interval to check unprocessed contexts. By default: `1.1`.	No
AURA_REDIS_CONTEXT_CACHE_UNPROCESSED_RANGE	Number	Number of unprocessed contexts to check by turn. By default: `10`.	No
AURA_REDIS_CONTEXT_CACHE_UNPROCESSED_RANGE_FACTOR	Number	Factor to divide the unprocessed range. By default: `5`.	No
AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT	Number	Number of shards to generate lists for storing unprocessed context. For performance reasons, it is important to configure the same number as the number of pods of the aura-redis-mongo-synchronizer.	No
AURA_REDIS_CONTEXT_CACHE_TTL	Number	Time in seconds to store a context in cache. By default: `60`.	No
AURA_REDIS_LOCK_CONTEXT_CACHE_TTL	Number	Time in seconds to store a lock context in cache. By default: `60`.	No
AURA_REDIS_MODE	String	Mode of Redis distribution. Values: `Cluster`, `Sentinel` or `Single`. By default: `Single`.	No
AURA_REDIS_PREFIX	String	Prefix that will be used by all Redis keys when using redis-connector. This allows mixing in a single Redis service messages coming from different environments in the same Azure subscription.	No
AURA_REDIS_SENTINEL_INSTANCE_NAME	String	Name of the Redis instance. Used in `Sentinel` mode.	No
AURA_REDIS_HOSTS	String	A string with a list of nodes (host:port) separated by comma. For example: “localhost:port,localhost2:port2”. Default: ‘127.0.0.0:6379’	No
AURA_REDIS_DATABASE	Number	Database number for Single or Sentinel mode. By default: `0`.
AURA_REDIS_PASSWORD	String	A string with the password of Redis.	No
AURA_REDIS_MAX_RECONNECT_RETRIES	Number	Number of retries to connect to Redis. By default: `25`.	No
AURA_REDIS_MAX_RECONNECT_INTERVAL	Number	Time in milliseconds to wait before reconnecting to Redis. By default: `5000`.	No
AURA_REDIS_USE_CONNECTION_POOL	Boolean	Flag to indicate to use a connection pool for Redis connections. Default: `true`.	No
AURA_REDIS_CONNECTION_POOL_MIN	Number	Minimum number of connections in the pool. Default: `2`.	No
AURA_REDIS_CONNECTION_POOL_MAX	Number	Maximum number of connections in the pool. Default: `100`.	No
AURA_REQUEST_TIMEOUT	Number	Number of seconds to wait a request. By default: `30 * 1000`.	No
AURA_SERVER_PORT	Number	Port where the server will be listening. By default: `8989`	No
AURA_SERVER_RETRIES	Number	Number of retries made by the server in case of error in an HTTP request. By default: `3`.	No
AURA_SERVER_RETRY_DELAY	Number	Delay between retries in case of error. By default: `100`.	No
AURA_SERVER_RETRY_FACTOR	Number	Factor to multiply delay for every HTTP request retried. By default: `10`.	No
AURA_SERVICE_ENVIRONMENT	Number	Type of environment: `'DEV', 'PRE', 'PRO'`. By default, `DEV`. It is used during locale translation, to get the correct text reference.	No
AURA_SHUTDOWN_GRACEFUL_TTL	Number	Time in milliseconds to complete the SHUTDOWN signal and p process all the messages in queue before SIGTERM. By default: `25 * 1000`	No
AURA_VERSION	String	Aura version. It is set by the deployment process.	No

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Bot databases

Description of databases used by Aura Bot

Introduction

Currently, aura-bot uses two databases, described below:

Mongo DB
Redis + MongoDB

MongoDB

In order to reuse the code, and clarify the development, all classes related with MongoDB connections are stored in the mongodb-storage.ts file within aura-bot-platform repository.

Here it can be found generic “connection” class, that is mongodb-connection.ts, that will handle the connection request, pooling them and configuring according the configuration (config is mandatory when initializing).

We can find also several “DAO” files, that will couple with a specific MongoDB database and collection, and will create optionally indexes or any other thing required in a specific domain.

MongoDbConnection

There will be only one instance of MongoDbConnection, creating it in the first call to getInstance (requiring config param) and returning the same instance in subsequent calls (where the config param is not required).

To create the instance properly, the following configuration variables are used:

Var name	Description	Required/Optional
AURA_MONGODB_URI	URI to connect to MongoDB, for example mongodb://localhost:27017/?authSource=admin	Required
AURA_MONGODB_USERNAME	User name to authenticate connection to MongoDB	Required
AURA_MONGODB_PASSWORD	Password to authenticate connection to MongoDB	Required
AURA_MONGODB_POOL_SIZE	MongoDB pool size	Optional (Default: 10)
AURA_MONGODB_SSL	Use SSL or not when connecting to MongoDB	Optional (Default: false)

This singleton module should be loaded the first time automatically by the system, and then could be accessed just by calling getInstance method, like:

const connection = await MongoDbConnection.getInstance();
const db = connection.dbConnect(DATABASE);
const collection = db.collection(COLLECTION);

MongoDbStorage

This DAO class will be used by Bot Builder v4 to store the conversation and user data. An index is created to delete documents older than 24 hours (specified by config var AURA_MONGODB_BOT_COLLECTION_CONTEXT_INDEX_TTL).

Following config variables are used to initialize this class:

Var name	Description	Required/Optional
AURA_MONGODB_BOT_DATABASE	Database name where the documents will be stored	Required
AURA_MONGODB_BOT_COLLECTION_CONTEXT	Collection name where the documents will be stored	Required
AURA_MONGODB_BOT_COLLECTION_CONTEXT_INDEX_TTL	Time (in seconds) that will exist the documents without updates, before being deleted	Optional (Default: 86440)

To create and initialize a MongoDbStorage, a code like following could be used:

const storage = await new MongoDbStorage().init(configuration);

Cache module

The cache module allows the generation of different types of cache:

AuraCacheLocal: only used for local data.
AuraCacheRemote: only used for remote data.
AuraCacheRemoteLocal: used for remote and local data, with priority for remote.
AuraCacheLocalRemote: used for local and remote data, with priority for local.

How to use cache module

It is possible to use any of the caches through the following commands:

// cache local.
const auraCacheLocal = new AuraCacheLocal('nameCacheLocal');
// cache remote.
const auraCacheRemote = new AuraCacheRemote('nameCacheRemote');
// cache local-remote
const auraCacheLocalRemote = new AuraCacheLocalRemote('nameCacheLocalRemote');
// cache remote-local.
const auraCacheRemoteLocal = new AuraCacheRemoteLocal('nameCacheRemoteLocal');

Variables for AuraCacheLocal

Only for local cache (AuraCacheLocal), it is possible to define a series of options:

stdTTL: standard TTL as number in seconds for every generated cache element. By default, 0 (0 = unlimited).
compressedData: data is to be compressed in remote cache. By default, false.
deleteOnExpire: boolean value that indicates whether variables will be deleted automatically when they expire or not. If true, the variable will be deleted. If false, the variable will remain. By default, true. You are encouraged to handle the variable upon the event expired by yourself.

const options = { stdTTL: 100, encryptData: true };
const auraCacheLocal = new AuraCacheLocal('nameCacheLocal', options);

Store a key

The following code sets a key value pair. It is possible to define a TTL (in seconds). It returns true on success.

const auraCacheLocal = new AuraCacheLocal('nameCacheLocal');
auraCacheLocal.set('key', 'value');

const auraCacheRemote = new AuraCacheRemote('nameCacheRemote');
await auraCacheRemote.set('key', 'value', 1800); // -> data expire at 30m.

Retrieve a key

The following code gets a saved value from the cache. It returns undefined if the value is not found or has expired. If the value is found, it returns an object with the corresponding key value pair.

const auraCacheLocal = new AuraCacheLocal('nameCacheLocal');
auraCacheLocal.set('key', 'value');
auraCacheLocal.get('key'); // -> value.

const auraCacheRemote = new AuraCacheRemote('nameCacheRemote');
await auraCacheRemote.set('key', 'value');
await auraCacheRemote.get('key'); // -> value.

For caches such as AuraCacheLocalRemote and AuraCacheRemoteLocal, it is possible to force it to obtain data different from the priority:

// cache local-remote
const auraCacheLocalRemote = new AuraCacheLocalRemote('nameCacheLocalRemote');
await auraCacheLocalRemote.set('key', 'value');
await auraCacheLocalRemote.getRemote('key'); // -> value.
auraCacheRemote.getLocal('key');
// cache remote-local.
const auraCacheRemoteLocal = new AuraCacheRemoteLocal('nameCacheRemoteLocal');
await auraCacheRemoteLocal.set('key', 'value');
auraCacheRemoteLocal.getLocal('key'); // -> value.
await auraCacheRemoteLocal.getRemote('key'); // -> value.

Delete a key

The following code deletes a key. It returns the number of deleted entries. A delete will never fail.

const auraCacheLocal = new AuraCacheLocal('nameCacheLocal');
auraCacheLocal.set('key', 'value');
auraCacheLocal.get('key'); // -> value.
auraCacheLocal.del('key'); // -> 1.
auraCacheLocal.get('key'); // -> undefined.

const auraCacheRemote = new AuraCacheRemote('auraCacheRemote');
await auraCacheRemote.set('key', 'value');
await auraCacheRemote.get('key'); // -> value.
await auraCacheRemote.del('key'); // -> 1.
await auraCacheRemote.get('key'); // -> undefined.

Flush all data

The following code deletes all data from a store.

const auraCacheLocal = new AuraCacheLocal('nameCacheLocal');
auraCacheLocal.set('key', 'value');
auraCacheLocal.get('key'); // -> value.
auraCacheLocal.flush(); // -> true.
auraCacheLocal.get('key'); // -> undefined.

const auraCacheRemote = new AuraCacheRemote('auraCacheRemote');
await auraCacheRemote.set('key', 'value');
await auraCacheRemote.get('key'); // -> value.
await auraCacheRemote.flush(); // -> true.
await auraCacheRemote.get('key'); // -> undefined.

Redis + MongoDB

If the configuration variable AURA_BOT_CONTEXT_DATABASE is set to REDIS-MONGODB, aura-bot uses a two-level cache to store the context through Aura Redis Mongo Synchronizer.

The following environment variables must be set for the two databases.

Variables

Var name	Description	Required/Optional
AURA_MONGODB_BOT_COLLECTION_CONTEXT	Name of the collection where the context is stored. By default: `aura-context`.	Required
AURA_MONGODB_BOT_DATABASE	Name of the database where the bot collections are stored. By default: `aura-bot`	Required
AURA_MONGODB_PASSWORD	Password of the MongoDB. Required	Required
AURA_MONGODB_POOL_SIZE	Pool size of the MongoDB. By default: `60`.	Required
AURA_MONGODB_SSL	Boolean value indicating if MongoDB connection uses SSL or not. By default: `true`.	Required
AURA_MONGODB_URI	URI of the MongoDB	Required
AURA_MONGODB_USERNAME	Username of the MongoDB	Required
AURA_REDIS_CONFIGURATION_PREFIX	Prefix for Redis configuration keys. By default: `aura-config:`	Required
AURA_REDIS_CONTEXT_CACHE_PREFIX	Prefix for Redis context cache keys. By default: `context:`	Required
AURA_REDIS_CONTEXT_CACHE_SHADOW_KEY_PREFIX	Prefix for Redis shadow key cache keys. By default: `shadow-key:`	Required
AURA_REDIS_CONTEXT_CACHE_ACTIVE_CONTEXT_PREFIX	Prefix for Redis active context cache keys. By default: `active-context:`	Required
AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT	Number of shards to generate lists for storing unprocessed context.	Required
AURA_REDIS_CONTEXT_CACHE_TTL	Time in seconds to store a context in cache. By default: `60`.	Required
AURA_REDIS_MODE	Mode of Redis distribution (`Cluster`, `Sentinel` or `Single`). By default: `Single`.	Required
AURA_REDIS_PREFIX	Prefix that will be used by all redis keys when using redis-connector. This allow mixing in a single redis service messages coming from different environments in the same Azure subscription.	Required
AURA_REDIS_SENTINEL_INSTANCE_NAME	Name of the Redis instance. Use in sentinel mode.	Optional
AURA_REDIS_HOSTS	A string with list of nodes (host:port) separate by comma. For example: “localhost:port,localhost2:port2”. Default: ‘127.0.0.0:6379’	Required
AURA_REDIS_DATABASE	Database number for `Single` or `Sentinel` mode. By default: `0`.
AURA_REDIS_PASSWORD	A string with the password of Redis.	Required
AURA_REDIS_MAX_RECONNECT_RETRIES	Number of retries to connect to Redis. By default: `25`.	Required
AURA_REDIS_MAX_RECONNECT_INTERVAL	Time in milliseconds to wait before reconnecting to Redis. By default: `5000`.	Required
AURA_REDIS_USE_CONNECTION_POOL	Used connection pool for Redis connections. Default: `true`.	Required
AURA_REDIS_CONNECTION_POOL_MIN	Minimum number of connections in the pool. Default: `2`.	Required
AURA_REDIS_CONNECTION_POOL_MAX	Maximum number of connections in the pool. Default: `100`.	Required

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Redis Mongo Synchronizer (ARMS)

aura-redis-mongo-synch (ARMS) is a service that allows the use of Redis and MongoDB to create a Bot Framework Context management as a two-level cache.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

To implement a two-level cache strategy:

Redis will be used as the first-level cache, with a short expiration time to speed up writing operations.
MongoDB will be used as the second-level cache, with a high expiration time.

aura-redis-mongo-synchronizer (ARMS) will be responsible for transferring expired data from Redis to MongoDB.

The service is implemented as a web server just to make it easier to handle it and to connect with Aura monitoring systems, such as Prometheus. It is in charge of managing the expired data in Redis using the Pub/Sub system provided by Redis.

Aura Redis Mongo Synchronizer components

The following figure shows the main components of the aura-redis-mongo-sync.

Server

The web server is implemented using fastify, a very fast web framework for NodeJS.

The web server works only as the service controller, while all the synchronization tasks are performed by a module outside the service called aura-redis-mongo-sync or ARMS.

Modules

Redis Mongo Sync

It is responsible for synchronizing data between Redis and MongoDB.

It uses a publish/subscribe system provided by Redis to notify the module when to synchronize data from Redis to MongoDB, based on events generated by Redis regarding the changes in the status of the documents.

Controllers & Services

When the previous step is finished, the request lands on the controller.

Each controller processes the request through a service in charge of implementing the logic. Once the request has been processed, the controller prepares and sends the response.

There is a controller to manage requests associated with the server itself (generic controller) and a controller for each module, in charge of handling the requests of that specific module.

Generic controller

As indicated above, the generic controller handles requests to retrieve information of the status of the server itself.

healthz: It is used to check if the server is online returning a simple status response: {"status": "ok"}.
lastevent: It returns the date of the last synchronization between Redis and MongoDB: { status: 'ok', lastEvent: DATE };
shutdown: Service to be invoked mainly by Kubernetes cluster to perform a server graceful shutdown.

Data Storage

A new class has been created that implements the Bot Framework storage interface called RedisMongoDbStorage.

This class writes the context data to Redis and firstly, it reads the data from Redis. If this data does not exist in Redis, it fetches it from MongoDB.

This class is included in the aura-common-utilities repository. Check the Github repository for further details.

Data structure in Redis

Redis stores its data in key:value structures. This value can support different data structures.

In the following sections, the ones used to store the bot context are explained.

These structures support an expiration time so that Redis deletes them when the expiration time is reached, so the content has expired. The expiration of a record in Redis emits an specific event, of expired type, that can be handled by an external module.

Context (Bot Framework Context)

The bot context is stored in a key:string structure:

Key: context:[MONGO DATABASE NAME]:[MONGO COLLECTION NAME]:[MONGO CONTEXT INDEX KEY]
Value: The content of the context is stored in a string obtained by converting the Object context in JSON to string.
Expiration: [EXPIRATION]

Where:

MONGO DATABASE NAME: Name of the database to use when ARMS writes this data in MongoDB.
MONGO COLLECTION NAME: Name of the collection to use when ARMS writes this data in MongoDB.
MONGO CONTEXT INDEX KEY: Key to use when ARMS writes this data in MongoDB.

When a key:value data expires in Redis, an event is emitted indicating which key contained the expired record (event => expired). The data can no longer be retrieved because Redis deletes value data before emitting the event and if this event is not processed, it will not recur, forcing us to use some auxiliary data structures in order to recover the expired data.

Shadow Keys

A Shadow Key is a key:void structure used to emit the event. As the expiration event returns a key, that will contain the current key of the context containing the data.

To improve efficiency, this shadow-key contains the key SHARD, which allows ARMS to subscribe only to certain shards to improve performance.

The number of shards is defined in the variable AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT, which indicates that the algorithm to calculate the shard based on the key will generate numbers between 1 and the value of the variable.

It is important to notice that the number of shards configured in AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT must be the same as the number of pods configured in the aura-redis-mongo-synchronizer.

Key: shadow-key:[SHARD]:context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY]
Value: empty
Expiration:  [REDIS CONTEXT DATABASE EXPIRATION]

When the service receives the expiration of this key, we can extract the shard and the real value from the context.

Active Context List

To ensure all data is synchronized, we need to keep in a list all the active context keys. By doing that, if an error occurs in the service and the expiration notifications are lost, we can recover data by periodically checking this list.

The data structure to be used is key: score-List. The score list contains a list of values with a string field and a numeric field by which the data can be sorted. The string field is used to store the key and the numeric field to store the expiration date.

In this way, we can retrieve those expired records that have not been processed, as items are removed from the list once they are synchronized.

Key: active-context:[SHARD_VALUE]
            context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY 1]     DATE_1
            context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY 2]     DATE_2
            ...
            context:[MONGO DATABASE NAME]:[MONGO COLLECTION CONTEXT]:[MONGO CONTEXT INDEX KEY N]     DATE_N

To store these elements, we will use partitioned lists in SHARD mode, that is, they will be included in the list that corresponds to them according to their KEY. The SHARD_VALUE is a number that is calculated from the key and the module obtained by dividing this number by the number of partitions that are configured.

For example, there is a series of lists to avoid collisions between PODS of the service (ARMS). That is, if the KEY1 key returns 301 (this data is obtained by performing an operation on the content of the key) this is divided by the number of partitions, 5 for example, and the module +1 is obtained, to avoid the 0. This indicates that the list where it will be stored will be active-context:2.

The execution of this process is based on the current system load. The variables associated to the execution period are:

AURA_REDIS_CONTEXT_CACHE_CACHE_CHECK_UNPROCESSED_INTERVAL_MIN, which indicates in seconds the minimum time that will be set.
AURA_REDIS_CONTEXT_CACHE_CACHE_CHECK_UNPROCESSED_INTERVAL_MAX, which indicates in seconds the maximum time for the interval.

The system will recalculate the next run based on the load. If the number of elements to be processed is very high, the interval will be reduced until it reaches the minimum, while if the load is low, the interval will be increased until it reaches the maximum.

SHARD variables

To manage the SHARD and the list to be selected by the ARMS, two variables come into play in Redis stored in the key:string structure.

AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT : Number of lists to generate (partitions).
AURA_REDIS_CONTEXT_LAST_SHARD_PROCESSED: Partition to manage at that moment, that is, when a service requests this data, one is added to it so that the next service that requests it can access the next partition, and so on.
AURA_REDIS_CONTEXT_LAST_INDEX_SHARD_PROCESSED: This variable is used to assign event subscriptions in an orderly fashion. For example, if we have two ARMS services and the AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT is set to four, then the first service will subscribe to 1 and 2 and the second service to 3 and 4. This order of assignment is done by incrementing this environment variable.

Synchronization Flows

Read Data

The Bot Framework components will read data in the following order:

They will try to read data from Redis.
If Redis does not return data, they will try it in MongoDB.

ARMS will read data in two situations:

When an expiration event is issued, ARMS reads the corresponding data, obtaining the context key contained in the expired KEY.
When the active context control service runs (it does it periodically) and it has some expired context that must be synchronized.

Write Data

Bot Framework components will write all their data in Redis.
ARMS will write the synchronized records in MongoDB.

NOTE: ARMS also writes data in Redis but only to control its core and internal behavior, incrementing SHARD, updating active context lists, etc.

Distribution of the event subscription

As previously mentioned, every time a Shadow-Key expires in Redis, it emits an event that will be received by the ARMS services that are subscribed to that event.

Now, as the Shadow-Key contains the SHARD of the real key of the data, we can distribute the events among different ARMS to improve performance and avoid network overload.

For the assignment of the events to subscribe, there is a method that is executed every certain time. The value is in the variable AURA_REDIS_CONTEXT_CACHE_CHECK_INDEX_SHARD_INTERVAL and is measured in seconds.

Steps to generate event subscription

Save the current state of the service: This saves, in a Redis structure, the identifier of the current ARMS and the current time.
Consult the number of ARMS services available: This is done by consulting the previous structure, obtaining those services that have been recently updated.
Calculate the number of events to subscribe: This is done by dividing AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT, which contains the number of partitions, by the number of services. There is a variable that can limit this assignment, AURA_REDIS_CONTEXT_CACHE_MIN_SHARD_BY_NODE: if this variable is greater than the previous result, the value of the variable is used. For example, if the result is 1, but this variable has the value 2, the service must subscribe to 2 events.
Assign the events: Once the number of events to subscribe to is determined, the AURA_REDIS_CONTEXT_LAST_INDEX_SHARD_PROCESSED variable is consulted and incremented. The value to assign is the module between the returned value and the value of AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT. In this way, it will assign consecutive values between 1 and the AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT.
Subscription to Redis: Once the SHARDs to be subscribed have been assigned, we only need to disconnect the events that do not match the current selection and connect the ones we do not have connected.

To obtain an optimal performance, ideally the number of AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT should at least match the number of ARMS services.

Example of distribution for AURA_REDIS_CONTEXT_CACHE_SHARD_COUNT = 5, and adding ARMS services until reaching 5.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

aura-mongo-handler utility

aura-mongo-handler utility is a tool to handle the MongoDB database

Introduction

aura-mongo-handler utility is the connector and cache for MongoDB.

Find more information in the Github repository: https://github.com/Telefonica/aura-common-utilities/tree/master/packages/aura-utilities/src/aura-mongo-handler/

Steps to Create a Two Level Cache

First, initialize the MongoDB Connector
Instantiate a new TwoLevelCache Object

Initialize MongoDB Connector

To initialize the Singleton of MongoDB connector, you need these variables in your configuration object:

AURA_MONGODB_URI
AURA_MONGODB_POOL_SIZE
AURA_MONGODB_SSL
AURA_MONGODB_USERNAME
AURA_MONGODB_PASSWORD
BRIDGE_MESSAGE_CACHE_TTL -> Time in miliseconds
BRIDGE_MONGODB_DATABASE_CACHE -> Database name in MongoDB for Cache
BRIDGE_MONGODB_COLLECTION_DL_CACHE -> Collection name in MongoDB for Cache
BRIDGE_MONGODB_INDEX_DL_CACHE -> Index field name in MongoDB Collection

To initialize, use the following command:


  Connector.init(configuration);

Instantiate a new TwoLevelCache Object

To instantiate a TwoLevelCache object, the following elements are needed:

A Mongo DataBase Name
A collection name
The property in the model to be saved in the cache to use as index in MongoDB. This index must be created by deploy scripts.

Constructor:

(name: string, localTTlMiliseconds: number, databaseName: string, collectionName: string, indexId?: string)

name: Name inside the logs generated by this instance.
localTTlMiliseconds: Local cache lifetime in in miliseconds.
databaseName: Name of the database in MongoDB.
collectionName: Name of the collection in MongoDB database.

const tlCache = new TwoLevelsCache<CustomModel>('LogName',30000, 'collectionName', 'DataBaseName', 'customId');

If you need more than one cache and use an instance for each of them, you can do this:

export class AppCaches {
    public static auraBotCache: TwoLevelsCache<AuraBotModel>;
    public static userDataCache:TwoLevelsCache<UserModel>;

    public static init(configuration: any) {
        AppCaches.auraBotCache = new TwoLevelsCache('AuraBotCache', 30000, 'AuraCacheDB','AuraCacheCollection', 'auraId');
        AppCaches.userDataCache = new TwoLevelsCache('AuraUserCache', 30000, 'UserCacheDB','UserCacheCollection''userId');
        return this;
    }
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Redis MongoDB Synchronizer metrics

List of metrics available in aura-redis-mongo-sync (ARMS)

aura_component_version

This metric is intended to store the number of aura-bot instances (pods) running each version of the code. It is stored as a Gauge in Prometheus.

Labels:

version: version field in the package.json file included in the running docker container.
component: name of the component that is writing the metric.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-redis-mongo-sync.

It is stored as a Counter in Prometheus.

Labels:

error: exception message that forced the unhandled error.

redis_mongo_sync_duration_milliseconds

This metric measures the data upload time from the service to the Mongo database.

It is stored as a Histogram in Prometheus. So every sample, besides the defined labels, also includes its duration.

The aura-redis-mongo-sync service contains a data collector that helps the event service move stale data from Redis to MongoDB. This collector sends the data in packets to optimize performance. This metric measures the time MongoDB takes to process the packet.

Labels:

status: HTTP status returned in the response. Values: success.
- success: if the status is success, the time is stored.

redis_mongo_synced_items_total

This metric is intended to store the registers synchronized between Redis and MongoDB by events.

It is stored as a Counter in Prometheus.

Labels:

type: register type. Values: event, active_context
- event: Items synchronized by event.
- active_context: Items synchronized by active context process.

redis_mongo_synced_errors

This metric is intended to store the errors that have occurred in the synchronization.

It is stored as a Counter in Prometheus.

Labels:

error: Values : create, syncData, executeBulk.
- create: If the error occurred when creating the service.
- syncData: If the error occurred when synchronizing the data.
- executeBulk: If the error occurred when uploading the data to MongoDB in bulk mode.

redis_mongo_sync_configuration_settings

This metric contains the service configuration data.

It is stored as a Gauge in Prometheus.

Labels:

setting_name: Values: shard_count, pod_count, active_context_ttl_seconds, redis_cache_ttl_seconds.
- shard_count: Current shard used to distribute the data to synchronize between pods.
- pod_count: Current number of services of aura-redis-mongo-sync.
- active_context_ttl_seconds: Time interval to run the data collector.
- redis_cache_ttl_seconds: Time in seconds that will be set to the context elements in the Redis cache.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

moduleId: Id of the module.
status: OK or ERROR