Make-up index manager

Introduction

The Make-up Index Manager is an utility for the management of indexes of a MongoDB database. This configuration can be loaded remotely.

Find more information in the Github repository: https://github.com/Telefonica/aura-common-utilities/tree/master/packages/aura-utilities/src/aura-makeup-index-manager/

Configuration

This module uses a MakeupMongoIndexManagerOptions model to manage the process.

MakeupMongoIndexManagerOptions

(*) The MongoDB configuration has not got a model defined yet. This is the used by MongoDB Index Manager:

  AURA_MONGODB_URI
  AURA_MONGODB_DB_NAME
  AURA_MONGODB_USERNAME
  AURA_MONGODB_PASSWORD
  AURA_MONGODB_POOL_SIZE
  AURA_MONGODB_SSL

An example is shown below:

   const options: MakeupMongoIndexManagerOptions = {
        mongodbConfiguration: {
            AURA_MONGODB_URI: configuration.AURA_MONGODB_URI,
            AURA_MONGODB_DB_NAME: configuration.AURA_MONGODB_BOT_DATABASE,
            AURA_MONGODB_USERNAME: configuration.AURA_MONGODB_USERNAME,
            AURA_MONGODB_PASSWORD: configuration.AURA_MONGODB_PASSWORD,
            AURA_MONGODB_POOL_SIZE: configuration.AURA_MONGODB_POOL_SIZE,
            AURA_MONGODB_SSL: configuration.AURA_MONGODB_SSL
        },
        fileConfiguration: {
            containerName: 'aura-configuration',
            localPath: 'settings/makeup',
            remotePath: '',
            files: [{ name: 'aura-bot-mongodb-indexes.json', mimeType: 'application/json' }]
        },
        replaceLocalWithRemote: true,
        storageCredentials: {
            storageKey: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY,
            storageName: configuration.AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT
        },
        uploadFromLocalToRemote: true,
        auraVersion: configuration.AURA_VERSION,
        ignoreDownloadErrors: true,
        ignoreUploadErrors: true,
        dataBaseSuffix: configuration.AURA_ENVIRONMENT_NAME
    };

    const makeupIndexManager = new MakeupIndexManager(options);
    await makeupIndexManager.process();

This utility uses downloadFiles to download index files.

MongoDB Index Manager

This module allows managing the indexes of a MongoDB database manager through a configuration file.

Configuration

IndexManagerModel

VersionControlModel

IndexManagerCollectionModel

IndexModel

An example is shown below:

{
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

Manage index with specific version

When an index configuration file is applied, the affected database generates or updates a collection called aura-control-version. In this collection, the history of index application and the last version applied are stored. This is to avoid generating indexes that have already been created.

If we want to force the generation of an index for a specific version, we can indicate it in the forceToVersion field of the configuration file. For example, if we are in version 1 and we want to generate a new index to the collection without modifying this version, we must force to the version previous to the one that currently has the database.

Example:

The database has the version 1 and we need apply a index with this version: we need to force it.

{
  "databases": {
    "forceToVersion": 0,
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

Rollbacks

Rollbacks are possible with the MongoDB Index Manager. A rollback forces MongoDB to generate the indexes on one or more other databases exactly as they are described in the configuration file, regardless of the version number.

In order to perform a rollback, it is necessary to have a configuration file. This configuration file can be generated manually or via the Mongo Index Manager Client.

Example of file configuration to make a rollback:

{
  "rollback": true,
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

First of all, the indexes will be removed and then will be regenerated as described in the configuration file.

Mongo Index Manager Cli

This utility allows us to perform index management operations of a MongoDB database. You can also generate the index configuration file of one or several MongoDB databases.

Command Line

node .\lib\mongo-index-manager-cli.js <options>

Options

• -f, –fileIndex: Input or output file with the Index Configuration File path.
• -a, –appName: Suffix for the databases. Example: “ap-current”. Optional.
• -r, –rollback: Generates an exact index structure to the configuration file regardless of versions. Optional.
• -d, –deleteOnly: Only delete all indexes. Optional
• -c, –connection: MongoDB Uri: mongodb://[username:password@]host1[:port1][,…hostN[:portN]][/[defaultauthdb][?options]].
• -u, –mongoUser: MongoDB Username. Optional but mandatory if the user does not exist in connection parameter.
• -p, –mongoPassword: MongoDB Password. Optional but mandatory if the password does not exist in connection parameter.

Examples

• Generate an Index Configuration File from MongoDB

  node ./lib/mongo-index-manager-cli.js  -g -c mongodb://localhost:27017 -u username -p password -f index-configuration-file.json
  

• Remove Indexes: Remove all indexes of collections that exist in the Configuration File.

  Configuration File (file.json)

  {
      "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
  }

      node ./lib/mongo-index-manager-cli.js  -d -c mongodb://localhost:27017 -u username -p password -f file.json

• Rollback Indexes: Generate the indexes in all collections defined in the Configuration File

  Configuration File (file.json)

  {
  "rollback": true,
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 1,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

    node ./lib/mongo-index-manager-cli.js  -d -c mongodb://localhost:27017 -u username -p password -f file.json

• Update or generate indexes in MongoDB by Configuration File

  Configuration File (file.json)

  {
  "databases": {
    "aura-bot": {
      "aura-version-control": {
        "version": 1,
        "removeIndex": [],
        "removeAllData": "",
        "indexes": [
          {
            "version": 2,
            "name": "idx_name",
            "keys": {
              "name": 1
            },
            "unique": true
          }
        ]
      }
    }
  }
}

  node ./lib/mongo-index-manager-cli.js  -c mongodb://localhost:27017 -u username -p password -f file.json