Aura –

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Bot make-up process

The current section describes the make-up processes that take place within Aura Bot

Introduction

The make-up process in aura-bot runs by using the following command:
npm run makeup

All the processes that are executed during the aura-bot make-up are detailed in the following sections.

Environments variables for Aura Bot make-up

Mandatory variables

Name
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY
AURA_DEFAULT_LOCALE
AURA_STATIC_RESOURCE_ENDPOINT
AURA_STATIC_RESOURCE_SAS_TOKEN
AURA_VERSION

Optional variables

These optional variables, together with their default values, are shown below. However, it can use some other optional variables, which can be checked out in the aura-bot make-up script in Github.

Name	Default value
AURA_MAKEUP_FILES_ORIGIN	blob
AURA_MAKEUP_FILES_DESTINATION	blob
AURA_AUTHORIZATION_HEADER
AURA_RESOURCES_PREFIX	’libraries'
AURA_SERVICE_ENVIRONMENT
AURA_MAKEUP_MODE	‘full’
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT
AURA_CONFIGURATION_CONTAINER	‘aura-configuration’
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT
AURA_CHANNELS_CONFIGURATION_BLOB	‘bot-response.json’
AURA_DIALOG_CONFIG_FILENAME	‘merged-dialog-config.json’
AURA_LOCAL_RESOURCES_PATH	’local-resources'
AURA_LOCAL_RESOURCES_PORT	3000

Load External Libraries

In this process, which is fully automated, the instance will be made-up, joining general configuration and i18n with library-specific ones.

The external libraries loaded by aura-bot are the different dialogs assigned to the use cases. This list of libraries is listed in the plugin-config.json file.

plugin-config.json

[
    "@telefonica/aura-bot-common-library",
    "@telefonica/aura-bot-disambiguation-library",
    "@telefonica/aura-bot-generic-library",
    {
        "exclude": [
            "./handover-genesys/handover-bypass-dialog",
            "./handover-genesys/handover-status-dialog"
        ],
        "packagePath": "@telefonica/aura-bot-handover-library"
    },
    "@telefonica/aura-bot-miscellaneous-library",
    "@telefonica/aura-bot-none-library",
    "@telefonica/aura-bot-onboarding-library",
    "@telefonica/aura-bot-tv-library",
    {
        "exclude": [
            "./wifi-check-connectivity-dialog",
            "./wifi-connect-to-guest-dialog",
            "./wifi-enable-disable-guest-dialog"
        ],
        "packagePath": "@telefonica/aura-bot-wifi-library"
    },
    "@telefonica/aura-bot-la-amazon",
    "@telefonica/aura-bot-la-iberia",
    "@telefonica/mi-movistar-beta",
    "@telefonica/generic-faq"
]

The make-up process is in charge of loading all these libraries and unifying the different resources they contain. Each environment contains the libraries associated with the required use cases.

Manage MongoDB Indexes

The indexes configuration file is in /settings/makeup/aura-bot-mongodb-indexes.json.

aura-bot-mongodb-indexes.json (partial)

{
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      },
      "aura-caches": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_item_caches",
            "keys": {
              "item": 1
            },
            "unique": true
          },
          {
            "version": 1,
            "name": "idx_ttl_caches",
            "keys": {
              "lastAccess": 1
            },
            "expireAfterSeconds": 3600
          }
        ]
      },
      "aura-context": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_aura_last_access",
            "keys": {
              "lastAccess": 1
            },
            "expireAfterSeconds": 3600
          }
        ]
      },
      "aura-external-kpi":
      .....

Once the make-up is executed, this file will be uploaded to the assigned repository [AURA-STORAGE]/aura-configuration/[AURA_VERSION]/aura-bot-mongodb-indexes.json. If this remote file exists, it has priority over the local file.

This configuration is assigned in the make-up process by means of the following model:

  const options: MakeupMongoIndexManagerOptions = {
        mongodbConfiguration: {
            AURA_MONGODB_URI: configuration.AURA_MONGODB_URI,
            AURA_MONGODB_DB_NAME: configuration.AURA_MONGODB_BOT_DATABASE,
            AURA_MONGODB_USERNAME: configuration.AURA_MONGODB_USERNAME,
            AURA_MONGODB_PASSWORD: configuration.AURA_MONGODB_PASSWORD,
            AURA_MONGODB_POOL_SIZE: configuration.AURA_MONGODB_POOL_SIZE,
            AURA_MONGODB_SSL: configuration.AURA_MONGODB_SSL
        },
        fileConfiguration: {
            containerName: 'aura-configuration',
            localPath: 'settings/makeup',
            remotePath: '',
            files: [{ name: 'aura-bot-mongodb-indexes.json', mimeType: 'application/json' }]
        },
        replaceLocalWithRemote: true,
        storageCredentials: {
            storageKey: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY,
            storageName: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT
        },
        uploadFromLocalToRemote: true,
        auraVersion: configuration.AURA_VERSION,
        ignoreDownloadErrors: true,
        ignoreUploadErrors: true,
        dataBaseSuffix: configuration.AURA_ENVIRONMENT_NAME
    }

Prepare Locale Resources

Each library may contain a settings/locale folder, with one or more i18n locale files (settings/locale/<lang>.json).

These locale files will be merged by language (in case of conflict, the latest loaded library overrides the previous one), and then merged with global locale files in aura-bot/locale/<lang>.json, taking precedence the global version in case of conflict.

All keys are sorted in resulting JSON files, to increase the readability.

For these local files to be merged, it is required to expose them in the locale property when registering the plugin. This will be done within the file index.ts, as shown in the following example:

import * as libraryUtil from '@telefonica/aura-bot-utilities/lib/aura-bot-library-util';
(...)
const settingsPath = path.resolve(__dirname, '..', 'settings');
register(null, {
    my_library: {
        locale: libraryUtil.readLocaleFolder(path.resolve(settingsPath, 'locale')),
        (...)
    }
});

The language resources in aura-bot are located in /locale/:

 de-de.json
 en-gb.json
 es-es.json
 pt-br.json

When starting aura-bot, these files are loaded remotely from [AURA-STORAGE]/static-resources/[AURA_VERSION]/locale/.

In the make-up process, the language resources of each loaded external library are merged into one file per language and then uploaded to Azure Storage using uploadFile method.

File example:

{
    "common:common.error.main": [
        "Aura no está disponible en estos momentos. Por favor, inténtalo un poco más tarde"
    ],
    "common:common.goodbyes.main": [
        "¡Hasta la vista!",
        "¡Hasta pronto! Espero haber podido ayudarte"
    ],
    "common:common.greetings.main": [
        "Hola, buenas",
        "Hola, estoy aquí para ayudarte"
    ]
}

Note that all the keys should start with the name of the library followed by a colon (":"). In the bot, this library name will be appended automatically, so only the last part of the key must be used in the dialogs.

Set Environment Variables

Each library may contain .env files within the folder settings or settings/{DEV|PRE|PRO} (some vars could have a specific value per environment).

If the AURA_SERVICE_ENVIRONMENT environment variable is set, the settings/.env file will be merged with settings/<AURA_SERVICE_ENVIRONMENT>/.env (taking precedence the env-specific values in case of conflict).

Every file is optional. After that, the resulting .env file from each library will be merged together and written down in aura-bot/.env.libraries. Note that this file has the lowest precedence when loading environmental variables in aura-bot.

Multiline values are escaped in order to allow the dotenv library to successfully read the values, even though it is recommended not using them.

For these .env files to be merged, it is required to expose them in the env property when registering the plugin. This will be done within the file index.ts, as shown in the following example:

import * as libraryUtil from '@telefonica/aura-bot-utilities/lib/aura-bot-library-util';
(...)
const settingsPath = path.resolve(__dirname, '..', 'settings');
register(null, {
    my_library: {
        env: libraryUtil.readEnv(settingsPath),
        (...)
    }
});

All the environment variables of each external library are merged together with the aura-bot environment variables and assigned as environment variables to the container or containers that will be in charge of raising their corresponding Pods in Kubernetes.

File example:

MYLIBRARY_VERSION=1.2.3
MYLIBRARY_ENDPOINT=https://my-library.test/v1
Note that to avoid conflicts, the env vars specific for a library should start with the library name in uppercase followed by an underscore ("_").

Prepare Remote Channel Config

Each library may contain dialog-config.json files or dialog-config.<AURA_BOT_DEFAULT_LOCALE>.json files.

The AURA_DEFAULT_LOCALE environment variable is required while AURA_SERVICE_ENVIRONMENT is optional. If AURA_SERVICE_ENVIRONMENT is set, these 4 files will be fetched and configuration is merged (the highest in the list means more priority in case of conflict):

(1) settings/<AURA_SERVICE_ENVIRONMENT>/dialog-config.<AURA_BOT_DEFAULT_LOCALE>.json
(2) settings/<AURA_SERVICE_ENVIRONMENT>/dialog-config.json
(3) settings/dialog-config.<AURA_DEFAULT_LOCALE>.json
(4) settings/dialog-config.json

In case AURA_SERVICE_ENVIRONMENT is not set, only #3 and #4 files are fetched and merged.

After that, the resulting dialog config from each library will be merged together. Then, the config is added to every channel in the config file pointed out by AURA_CHANNEL_CONFIGURATION_ENDPOINT (except the dialogs containing an only_in string array property, that are not added to channels not present in that property) replacing dialog_libraries from each channel. Then the resulting file is uploaded to Azure, replacing the previous one.

For these files to be merged, it is required to expose them in the config property when registering the plugin. This will be done within the file index.ts, as shown in the following example:

import * as libraryUtil from '@telefonica/aura-bot-library-util';
(...)
const settingsPath = path.resolve(__dirname, '..', 'settings');
register(null, {
    my_library: {
        config: libraryUtil.readDialogConfig(settingsPath),
        (...)
    }
});

Use onlyIn property

In some cases, the dialog configuration can differ among channels for the same country. In these cases, the onlyIn property must be used in the dialog-config.<AURA_DEFAULT_LOCALE>.json file:

{
    "name": "tv",
    "authorization": {
        "purposes": "customer-self-service",
        "scopes": "subscribed-products-user-read"
    },
    "dialogs": [
        {
            "id": "tv-carousel-info",
            "suggestions": false,
            "triggerConditions": [
                {
                    "intent": "intent.carousel.info"
                },
                {
                    "intent": "intent.tv.details"
                }
            ],
            "onlyIn": [
                "movistar-plus"
            ]
        }
   ]
}

This configuration file is built by merge each channel configuration from each external library and uploaded to [AURA-STORAGE]/aura-configuration/bot-response.json using uploadStringAsBlob and/or (according to AURA_MAKEUP_FILES_DESTINATION) uploading channel information in aura-configuration-api.

bot-response.json (partial)

[
  {
    "channel_id": "45494a5b-835a-4fff-a813-b3d2be529dbe",
    "fpa_auth_purposes": "customer-self-service detect-abnormal-usage device-recommendations-v3 sim-upgrade-suggestion aura-read-insight-events identify-customer bolt-on-suggestion",
    "fpa_auth_scopes": "device-catalog:devices-read device-stock:stock-read",
    "name": "novum-mytelco",
    "nlp": {
      "enabled": true,
        "enabled": true
      }
    },
    "prefix": "nov",
    "dialogLibraries": [
      {
        "name": "generic",
        "dialogs": [
          {
            "id": "get-generic",
            "triggerConditions": [
              {
                "intent": "intent.greetings"
              }
            ]
          }
        ]
      },
      {
        "name": "bill",
        "dialogs": [
          {
            "id": "balance-check",
            "suggestions": true,
            "authorization": {
              "purposes": "customer-self-service",
              "scopes": "mobile-balance-read"
            },
            "triggerConditions": [
              {
                "intent": "intent.balance.check",
                "contextFilters": [
                  {
                    "name": "Anonymous redirect to linking",
                    "type": "type",
                    "conditions": "/type eq 'anonymous'",
                    "true": {
                      "name": "Anonymous redirect to linking",
                      "breakDialogExecution": true,
                      "breakFilterEval": true,
                      "redirectToIntent": "intent.account.linking",
                      "suggestions": false
                    }
                  },
                  {
                    "name": "user_type_multimsisdn_not_allowed",
                    "type": "user_type_filter",
                    "conditions": "/type eq 'multimsisdn'",
                    "true": {
                      "name": "user_type_not_allowed_action_true",
                      "breakDialogExecution": true,
                      "breakFilterEval": true,
                      "resource": "context-filter:multimsisdn-users-intent-not-allowed.text",
                      "suggestions": false
                    }
                  },
...

Process Resources

Each library may contain a resources folder with images or any other binary files. If present, those files are uploaded to Azure container pointed by AURA_SERVICE_ENVIRONMENT env var, within libraries/<library> virtual path.

This uploading process could last longer than the other make-up steps, so the process could throw errors after showing that the make-up process is finished (with pending uploading processes).

The following snippet shows how to use those files within the dialogs (it will take AURA_STATIC_RESOURCE_ENDPOINT and AURA_STATIC_RESOURCE_SAS_TOKEN env vars from the environment):

import * as libraryUtil from '@telefonica/aura-bot-library-util';
(...)
const wholeUrl = getResourcePath('my_library', 'icon.png');

If we need to retrieve an image with specific device resolution as coming in the imageSettings resolution field of activity channelData, we should use getImageUrl method as follows:

import { getImageUrl } from '@telefonica/aura-bot-library-util';
[...]
const wholeUrlWithResolution = getImageUrl(stepContext.context, 'my_library', 'icon.png', configuration);

In libraries, the resources path must be exposed in the resources property when registering the plugin. This will be done within the file index.ts, as shown in the following example:

register(null, {
    my_library: {
        resources: path.resolve(__dirname, '..', 'resources'),
        (...)
    }
});

In the aura-bot make-up process the resources of each external library are uploaded to Azure Storage so that they can be later consumed by them. Remote path: [AURA-STORAGE]/static-resources/libraries/[LIBRARY_NAME].

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Bridge make-up process

The current section describes the make-up processes that take place within Aura Bridge

Environment variables for Aura bridge make-up

Mandatory environment variables

AURA_ENVIRONMENT_NAME
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT
AURA_MONGODB_PASSWORD
AURA_MONGODB_URI
AURA_MONGODB_USERNAME
AURA_VERSION

Optional environment variables

These optional variables, together with their default values, are shown below.

AURA_LOCALE_FOLDER                      # Default: ./locale
AURA_LOCALE_REMOTE_CONTAINER            # Default: 'static-resources
AURA_LOCALE_REMOTE_CONTAINER_PREFIX     # Default: aura-bridge/locale
AURA_MAKEUP_MODE                        # Default: full
AURA_MICROSOFT_AZURE_STORAGE_CONFIGURATION_CONTAINER  # Default: aura-configuration
AURA_MONGODB_POOL_SIZE                  # Default: 60
AURA_MONGODB_SSL                        # Default: false
AURA_SWAGGER_LOCAL_CORE_PATH            # Default: swagger-core.yaml
AURA_SWAGGER_LOCAL_PATH                 # Default: swagger.yaml
AURA_SWAGGER_REMOTE_CONTAINER_PREFIX    # Default: swagger
BRIDGE_MONGODB_DATABASE_CACHE           # Default: aura-bridge

Manage MongoDB Indexes

In this task, the make-up is responsible for creating the necessary indexes in the MongoDB database. These indexes will only be created if the AURA_MAKEUP_MODE variable is configured to “full”.

The indexes configuration file is in /settings/makeup/aura-bridge-mongodb-indexes.json. An example of this file is shown below.

{
    "version":2.3,
    "databases": {
        "aura-bridge-caches": {
            "directline-cache": {
                "version": 1,
                "indexes": [
                    {
                        "version": 1,
                        "name": "idx_item_caches",
                        "keys": {
                            "item": 1
                        },
                        "unique": true
                    },
                    {
                        "version": 1,
                        "name": "idx_ttl_caches",
                        "keys": {
                            "lastAccess": 1
                        },
                        "expireAfterSeconds": 3600
                    }
                ]
            },
            "dev-behavior-cache": {
                "version": 2,
                "indexes": [
                    {
                        "version": 2,
                        "name": "idx_ttl_caches",
                        "keys": {
                            "lastAccess": 1
                        },
                        "expireAfterSeconds": 3600
                    }
                ]
            },
            "bridge-message-queue": {
                "version": 2.3,
                "indexes": [
                    {
                        "version": 2.2,
                        "name": "idx_ttl_caches",
                        "keys": {
                            "expiresAt": 1
                        },
                        "expireAfterSeconds": 0,
                        "force": true
                    },
                    {
                        "version": 2.2,
                        "name": "idx_queue_search",
                        "keys": {
                            "conversationId": 1,
                            "requestId": 1
                        },
                        "force": true,
                        "unique": false
                    },
                    {
                        "version": 2.2,
                        "name": "idx_first_message",
                        "keys": {
                            "conversationId": 1,
                            "requestTimestamp": 1,
                            "responseTimestamp": 1
                        },
                        "unique": false,
                        "force": true
                    },
                    {
                        "version": 2.2,
                        "name": "idx_ack_find",
                        "keys": {
                            "responseId": 1
                        },
                        "unique": false,
                        "force": true
                    },
                    {
                        "version": 2.3,
                        "name": "idx_ack_find_async_id",
                        "keys": {
                            "asyncResponseId": 1
                        },
                        "unique": false,
                        "force": true
                    }
                ]
            }
        }
    }
}

Once the make-up is executed, this file will be uploaded to the assigned repository [AURA-STORAGE]/aura-configuration/[AURA_VERSION]/aura-bridge-mongodb-indexes.json. If this remote file exists, it has priority over the local file.

This configuration is assigned in the make-up process by means of the following model:

    const options: MakeupMongoIndexManagerOptions = {
        mongodbConfiguration: {
            AURA_MONGODB_URI: configuration.AURA_MONGODB_URI,
            AURA_MONGODB_DB_NAME: configuration.BRIDGE_MONGODB_DATABASE_CACHE,
            AURA_MONGODB_USERNAME: configuration.AURA_MONGODB_USERNAME,
            AURA_MONGODB_PASSWORD: configuration.AURA_MONGODB_PASSWORD,
            AURA_MONGODB_POOL_SIZE: configuration.AURA_MONGODB_POOL_SIZE,
            AURA_MONGODB_SSL: configuration.AURA_MONGODB_SSL
        },
        fileConfiguration: {
            containerName: 'aura-configuration',
            localPath: 'settings/makeup',
            remotePath: '',
            files: [{ name: 'aura-bridge-mongodb-indexes.json', mimeType: 'application/json' }]
        },
        replaceLocalWithRemote: true,
        storageCredentials: {
            storageKey: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY,
            storageName: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT
        },
        uploadFromLocalToRemote: true,
        auraVersion: configuration.AURA_VERSION,
        ignoreDownloadErrors: true,
        ignoreUploadErrors: true,
        dataBaseSuffix: configuration.AURA_ENVIRONMENT_NAME
    }

Prepare Locale Resources

This task will only be created if the AURA_MAKEUP_MODE variable is configured to full.

The language resources in aura-bridge are located in /locale/:

 en-gb.json
 es-es.json

When starting an aura-bridge, these files are loaded remotely from [AURA-STORAGE]/static-resources/aura-bridge/[AURA_VERSION]/locale/.

In the make-up process, the language resources are uploaded to Azure Storage.

Prepare unified swagger

aura-bridge is composed of a set of plugins in which each one is responsible for a single task. Processor or API plugins can have a swagger that defines the complete definition of the service.

In this task, the make-up process is responsible for unifying all these swaggers defined in each plugin in a single and unified swagger. This swagger is used by the oastool in the server’s start-up phase.

To create this unified swagger, the make-up process uses a swagger-core.yaml file that defines the base of the final swagger file.

The process for the creation of this unified swagger is as follows:

The swagger-core.yaml file is used as a base.
Each swagger.yaml file of each plugin is merged with the swagger-core.yaml:
- Fields 'components.schemas', 'components.securitySchemes' and 'paths' are merged using Object.assign.
- Field 'tags' is added using Array.push.

graph LR
    subgraph Unified swagger from plugins and swagger-core.yaml
      A[swagger-core.yaml] --> Z[swagger.yaml]
      B[swagger.yaml -> admin-plugin] --> Z[swagger.yaml]
      C[swagger.yaml -> whatsapp-incoming-processor] --> Z[swagger.yaml]
      D[swagger.yaml -> genesys-directline-processor] --> Z[swagger.yaml]
    end

It is important to know that entities with the same name defined in different plugins will be overwritten and the last processing will be used during the Swagger unification process (as in paths and securityschemes).

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura authentication server make-up process

The current section describes the make-up processes that take place within Aura authentication server

Manage MongoDB Indexes

The indexes configuration file is in /settings/makeup/aura-authentication-mongodb-indexes.json.

Once the make-up is executed, this file will be uploaded to the assigned repository [AURA-STORAGE]/aura-configuration/[AURA_VERSION]/aura-authentication-mongodb-indexes.json. If this remote file exists, it has priority over the local file.

This configuration is assigned in the make-up process by means of the following model:

   const options: MakeupMongoIndexManagerOptions = {
        mongodbConfiguration: {
            AURA_MONGODB_URI: configuration.AURA_MONGODB_URI,
            AURA_MONGODB_DB_NAME: configuration.AURA_MONGODB_USER_DB,
            AURA_MONGODB_USERNAME: configuration.AURA_MONGODB_USERNAME,
            AURA_MONGODB_PASSWORD: configuration.AURA_MONGODB_PASSWORD,
            AURA_MONGODB_POOL_SIZE: configuration.AURA_MONGODB_POOL_SIZE,
            AURA_MONGODB_SSL: configuration.AURA_MONGODB_SSL
        },
        fileConfiguration: {
            containerName: 'aura-configuration',
            localPath: 'settings/makeup',
            remotePath: '',
            files: [{ name: 'aura-authentication-mongodb-indexes.json', mimeType: 'application/json' }]
        },
        replaceLocalWithRemote: true,
        storageCredentials: {
            storageKey: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY,
            storageName: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT
        },
        uploadFromLocalToRemote: true,
        auraVersion: configuration.AURA_VERSION,
        ignoreDownloadErrors: true,
        ignoreUploadErrors: true,
        dataBaseSuffix: configuration.AURA_ENVIRONMENT_NAME
    };

Prepare Locale Resources

The language resources in Aura Authentication Service are located in /locale/:

de-de.json
en-gb.json
es-es.json

When starting an Aura Authentication Service, these files are loaded remotely from [AURA-STORAGE]/static-resources/authentication/locale/[AURA_VERSION]/.

In the make-up process, the language resources are uploaded to Azure Storage.

Process Resources

In the make-up process of Aura Authentication Service, the resources are loaded from /settings/resources/ and upload to [AURA-STORAGE]/static-resources/authentication/resources/[AURA_VERSION]/.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Make-up index manager

Make-up Index Manager allows managing the indexes of a MongoDB database

Introduction

The Make-up Index Manager is an utility for the management of indexes of a MongoDB database. This configuration can be loaded remotely.

Find more information in the Github repository: https://github.com/Telefonica/aura-common-utilities/tree/master/packages/aura-utilities/src/aura-makeup-index-manager/

Configuration

This module uses a MakeupMongoIndexManagerOptions model to manage the process.

MakeupMongoIndexManagerOptions

Property	Type	Description
storageCredentials	StorageCredentials	Credentials of Azure Storage.
fileConfiguration	FileConfiguration	The configuration file structure to manage the files.
mongodbConfiguration	any	MongoDB config (*)
replaceLocalWithRemote	boolean	If `true` and if a remote file exist, this replaces the local one.
uploadFromLocalToRemote	boolean	If `true` and if a remote file does not exist, it uploads the local one.
ignoreDownloadErrors	boolean	If `true`, if any file cannot be downloaded, it stops the process and returns an error.
ignoreUploadErrors	boolean	If `true`, if any file cannot be uploaded, it stops the process and returns an error.
auraVersion	string	Aura version, used to store the remote file in a folder with this value. Optional.
dataBaseSuffix	string	Used to rename the MongoDB with a suffix of Aura Environment. Example: for the value “ap-current”, the aura-bot database will be “aura-bot-ap-current”. There is an environment variable `AURA_ENVIRONMENT_NAME` in aura-bot to get this value. Optional.

(*) The MongoDB configuration has not got a model defined yet. This is the used by MongoDB Index Manager:

  AURA_MONGODB_URI
  AURA_MONGODB_DB_NAME
  AURA_MONGODB_USERNAME
  AURA_MONGODB_PASSWORD
  AURA_MONGODB_POOL_SIZE
  AURA_MONGODB_SSL

An example is shown below:

   const options: MakeupMongoIndexManagerOptions = {
        mongodbConfiguration: {
            AURA_MONGODB_URI: configuration.AURA_MONGODB_URI,
            AURA_MONGODB_DB_NAME: configuration.AURA_MONGODB_BOT_DATABASE,
            AURA_MONGODB_USERNAME: configuration.AURA_MONGODB_USERNAME,
            AURA_MONGODB_PASSWORD: configuration.AURA_MONGODB_PASSWORD,
            AURA_MONGODB_POOL_SIZE: configuration.AURA_MONGODB_POOL_SIZE,
            AURA_MONGODB_SSL: configuration.AURA_MONGODB_SSL
        },
        fileConfiguration: {
            containerName: 'aura-configuration',
            localPath: 'settings/makeup',
            remotePath: '',
            files: [{ name: 'aura-bot-mongodb-indexes.json', mimeType: 'application/json' }]
        },
        replaceLocalWithRemote: true,
        storageCredentials: {
            storageKey: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY,
            storageName: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT
        },
        uploadFromLocalToRemote: true,
        auraVersion: configuration.AURA_VERSION,
        ignoreDownloadErrors: true,
        ignoreUploadErrors: true,
        dataBaseSuffix: configuration.AURA_ENVIRONMENT_NAME
    };

    const makeupIndexManager = new MakeupIndexManager(options);
    await makeupIndexManager.process();

This utility uses downloadFiles to download index files.

MongoDB Index Manager

This module allows managing the indexes of a MongoDB database manager through a configuration file.

Configuration

IndexManagerModel

Property	Type	Description
version	number	Index manager’s version.
databases	object	Object with databases names as properties.
forceToVersion	number	Force to use a base version to update indexes and collections. Optional.
rollback	boolean	Flag to initialize rollback procedure. Optional.

VersionControlModel

Property	Type	Description
version	number	Index manager’s version.
name	string	Internal name module for supporting version control in other modules.
timestamp	Date	Date of version applied.

IndexManagerCollectionModel

Property	Type	Description
version	number	Version of changes.
indexes	IndexModel[]	Array of indexes to generate or update.
isPrefix	boolean	Check if prefix if present or not. Optional.
removeIndex	string[]	Array of indexes names. For deleting all indexes in a collection, use ["*"]. Optional.
removeAllData	string	Parameter to remove all collection data. Use the collection name in upper case for validation. Optional.

IndexModel

Property	Type	Description
version	number	Version of changes.
name	string	Name of the index.
keys	IndexKeyModel	Document fields to generate the indexes.
unique	boolean	It indicates if the key is unique for a document or not. Optional.
partialFilterExpression	boolean	If the index is partial, expression which filters specific documents to apply the index. Optional.
expireAfterSeconds	number	Number of seconds when the index expires. For this, `keys` must contain a field `datetime`. Optional.
collation	boolean	It indicates if language-specific rules for string comparison to index apply or not. Optional.

An example is shown below:

{
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

Manage index with specific version

When an index configuration file is applied, the affected database generates or updates a collection called aura-control-version. In this collection, the history of index application and the last version applied are stored. This is to avoid generating indexes that have already been created.

If we want to force the generation of an index for a specific version, we can indicate it in the forceToVersion field of the configuration file. For example, if we are in version 1 and we want to generate a new index to the collection without modifying this version, we must force to the version previous to the one that currently has the database.

Example:

The database has the version 1 and we need apply a index with this version: we need to force it.

{
  "databases": {
    "forceToVersion": 0,
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

Rollbacks

Rollbacks are possible with the MongoDB Index Manager. A rollback forces MongoDB to generate the indexes on one or more other databases exactly as they are described in the configuration file, regardless of the version number.

In order to perform a rollback, it is necessary to have a configuration file. This configuration file can be generated manually or via the Mongo Index Manager Client.

Example of file configuration to make a rollback:

{
  "rollback": true,
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

First of all, the indexes will be removed and then will be regenerated as described in the configuration file.

Mongo Index Manager Cli

This utility allows us to perform index management operations of a MongoDB database. You can also generate the index configuration file of one or several MongoDB databases.

Command Line

node .\lib\mongo-index-manager-cli.js <options>

Options

-f, –fileIndex: Input or output file with the Index Configuration File path.
-a, –appName: Suffix for the databases. Example: “ap-current”. Optional.
-r, –rollback: Generates an exact index structure to the configuration file regardless of versions. Optional.
-d, –deleteOnly: Only delete all indexes. Optional
-c, –connection: MongoDB Uri: mongodb://[username:password@]host1[:port1][,…hostN[:portN]][/[defaultauthdb][?options]].
-u, –mongoUser: MongoDB Username. Optional but mandatory if the user does not exist in connection parameter.
-p, –mongoPassword: MongoDB Password. Optional but mandatory if the password does not exist in connection parameter.

Examples

Generate an Index Configuration File from MongoDB

node ./lib/mongo-index-manager-cli.js  -g -c mongodb://localhost:27017 -u username -p password -f index-configuration-file.json

Remove Indexes: Remove all indexes of collections that exist in the Configuration File.

Configuration File (file.json)

  {
      "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
  }

      node ./lib/mongo-index-manager-cli.js  -d -c mongodb://localhost:27017 -u username -p password -f file.json

Rollback Indexes: Generate the indexes in all collections defined in the Configuration File

Configuration File (file.json)

  {
  "rollback": true,
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

    node ./lib/mongo-index-manager-cli.js  -d -c mongodb://localhost:27017 -u username -p password -f file.json

Update or generate indexes in MongoDB by Configuration File

Configuration File (file.json)

  {
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 2,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

  node ./lib/mongo-index-manager-cli.js  -c mongodb://localhost:27017 -u username -p password -f file.json