Aura –

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Check Hugging Face embedding models downloading

Guidelines to check if the Hugging Face models used in ATRIA are downloaded during the generate-db process

Introduction

The free embedding templates we are currently using in ATRIA are paraphrase-multilingual-MiniLM-L12-v2 and multi-qa-distilbert-cos-v1 both from Hugging Face. (These models are the ones used with the following embeddings by default available in ATRIA: Local Sentence Transformer and Distilbert-based Local Sentence Transformer).

During the generate-db process, these models are loaded into memory and the process may fail if there is a connection problem with Hugging Face. In this error scenario, the only solution is to wait until the service is again up and running.

In the current document, we include the instructions to check if the embedding models can be downloaded, in order to detect the process failure.

Prerequisites

Install huggingface-cli
```
pkgx install huggingface-cli
```

Check if the Hugging Face models are downloaded properly

The way to check if the service is up is by launching the following command:

huggingface-cli download sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

If the download starts, the service is up, and you can restart the generate-db process.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Migrate Aura Bot and Aura Groot context to Linkin Park

Guidelines for the migration of aura-bot and aura-groot database context schemas from releases previous to Linkin Park to Linkin Park onwards. Additionally, the rollback procedure is included.

Prerequisites

Mandatory requisite: A valid kubeconfig for the environment must be available.
Recommendation: Create a snapshot/backup of the current DB data.

Process for context DB migration

Access MongoDB console

To access and open MongoDB console we can use the kubectl command below to export the intended environment, with MongoDB adminuser password. Use the right environment and database name in the commands.

$ export ATLAS_DEPLOYMENT_NAME=aura-br-pro-70 # Get from kubectl get -n aura-system atlasdeployments.atlas.mongodb.com
$ export ATLAS_DATABASE_NAME=aura-groot-br-pro # Set with the correct DB of your deployment
$ export MONGODB_USER=$(kubectl get -n aura-system atlasdatabaseusers.atlas.mongodb.com mongodb-adminuser-password -o jsonpath='{.spec.username}')
$ export MONGODB_PASSWORD=$(kubectl get -n aura-system secret mongodb-adminuser-password -o go-template='{{ .data.password | base64decode }}')
$ export MONGODB_URI="$(kubectl get -n aura-system atlasdeployments.atlas.mongodb.com $ATLAS_DEPLOYMENT_NAME -o jsonpath='{.status.connectionStrings.standardSrv}'|sed "s|srv://|srv://$MONGODB_USER:$MONGODB_PASSWORD@|g")/${ATLAS_DATABASE_NAME}"
$ kubectl -n aura-system run fix-script -i -q --rm --restart=Never --image=mongo:latest --command mongosh ${MONGODB_URI}

Migrate aura-bot context collection to new schema

Update documents to the new format:

> db["aura-context"].updateMany(
      {
        document: {
          $exists: true
        }
      },
      [
        {
          $set: {
            eTag: "$document.eTag",
            conversationData: { $mergeObjects: ["$conversationData", "$document.conversationData"] },
            dialogState: { $mergeObjects: ["$dialogState", "$document.dialogState"] },
            userData: { $mergeObjects: ["$userData", "$document.userData"] },
            userAuthData: { $mergeObjects: ["$userAuthData", "$document.userAuthData"] }
          }
        },
        { $unset: "document" }
      ]
    );

Migrate aura-groot context collection to new schema

Update documents to the new format:

> db["aura-context"].updateMany(
      {
        document: {
          $exists: true
        }
      },
      [
        {
          $set: {
            eTag: "$document.eTag",
            activeSkill: { $mergeObjects: ["$activeSkill", "$document.activeSkill"] },
            conversationHistory: { $concatArrays: [{ $ifNull: ["$conversationHistory", []] }, { $ifNull: ["$document.conversationHistory", []] }] },
            conversationDataProperty: { $mergeObjects: ["$conversationDataProperty", "$document.conversationDataProperty"] },
            dialogContext: { $concatArrays: [{ $ifNull: ["$dialogContext", []] }, { $ifNull: ["$document.dialogContext", []] }] },
            skillConversationReference: { $mergeObjects: ["$skillConversationReference", "$document.skillConversationReference"] }
          }
        },
        { $unset: "document" }
      ]
    );

Rollback aura-bot to the previous schema

If a rollback is needed, documents should be modified to have the previous schema and maintain the users’ context data:

> db["aura-context"].updateMany(
      {},
      [
        {
          $set: {
            document: {
              eTag: "$eTag",
              conversationData: { $mergeObjects: ["$document.conversationData", "$conversationData"] },
              dialogState: { $mergeObjects: ["$document.dialogState", "$dialogState"] },
              userData: { $mergeObjects: ["$document.userData", "$userData"] },
              userAuthData: { $mergeObjects: ["$document.userAuthData", "$userAuthData"] }
            }
          }
        },
        {
          $unset: [ "eTag", "conversationData", "dialogState", "userData", "userAuthData"]
        }
      ]
    );

Rollback aura-groot to the previous schema

If a rollback is needed, documents should be modified to have the previous schema and maintain the users’ context data:

> db["aura-context"].updateMany(
      {},
      [
        {
          $set: {
            document: {
              eTag: "$eTag",
              activeSkill: { $mergeObjects: ["$document.activeSkill", "$activeSkill"] },
              conversationHistory: { $concatArrays: [{ $ifNull: ["$document.conversationHistory", []] }, { $ifNull: ["$conversationHistory", []] }] },
              conversationDataProperty: { $mergeObjects: ["$document.conversationDataProperty", "$conversationDataProperty"] },
              dialogContext: { $concatArrays: [{ $ifNull: ["$document.dialogContext", []] }, { $ifNull: ["$dialogContext", []] }] },
              skillConversationReference: { $mergeObjects: ["$document.skillConversationReference", "$skillConversationReference"] }
            }
          }
        },
        {
          $unset: [ "eTag", "activeSkill", "conversationHistory", "conversationDataProperty", "dialogContext", "skillConversationReference"]
        }
      ]
    );

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Migrate Aura Groot context (Dire Straits)

Guidelines for the migration of aura-groot database context schema from releases previous to Dire Straits to Dorian release onwards. Additionally, the rollback procedure is included.

Prerequisites

Mandatory requisite: A valid kubeconfig for the environment must be available.
Recommendation: Create a snapshot/backup of the current DB data.

Process for context DB migration

Access MongoDB console

$ export ATLAS_DEPLOYMENT_NAME=aura-br-pro-70 # Get from kubectl get -n aura-system atlasdeployments.atlas.mongodb.com
$ export ATLAS_DATABASE_NAME=aura-groot-br-pro # Set with the correct DB of your deployment
$ export MONGODB_USER=$(kubectl get -n aura-system atlasdatabaseusers.atlas.mongodb.com mongodb-adminuser-password -o jsonpath='{.spec.username}')
$ export MONGODB_PASSWORD=$(kubectl get -n aura-system secret mongodb-adminuser-password -o go-template='{{ .data.password | base64decode }}')
$ export MONGODB_URI="$(kubectl get -n aura-system atlasdeployments.atlas.mongodb.com $ATLAS_DEPLOYMENT_NAME -o jsonpath='{.status.connectionStrings.standardSrv}'|sed "s|srv://|srv://$MONGODB_USER:$MONGODB_PASSWORD@|g")/${ATLAS_DATABASE_NAME}"
$ kubectl -n aura-system run fix-script -i -q --rm --restart=Never --image=mongo:latest --command mongosh ${MONGODB_URI}

Migrate context collection to new schema

Update documents to the new format:

> db["aura-context"].aggregate([
      {
        $match: {
          _id: {
            $regex: "/conversations/",
          },
          "document.activeSkill.conversationId": {
            $exists: true,
          },
        },
      },
      {
        $lookup:
          {
            from: "aura-context",
            localField:
              "document.activeSkill.conversationId",
            foreignField: "_id",
            as: "document.skillConversationReference",
          },
      },
      {
        $set: {
          "document.skillConversationReference": {
            $arrayElemAt: [
              "$document.skillConversationReference",
              0,
            ],
          },
        },
      },
      {
        $set: {
          "document.skillConversationReference":
            "$document.skillConversationReference.document",
        },
      },
      {
        $merge: "aura-context",
      },
    ])

Check if changes have been applied. The following query should return n documents:

  > db["aura-context"].find({"document.skillConversationReference": { $exists: true }})

Rollback to the previous schema

If a rollback is needed, documents should be modified to have the previous schema and maintain the users’ context data:

> db["aura-context"].aggregate([
    {
      $match:
        {
          "document.activeSkill.conversationId": {
            $exists: true,
          },
          "document.skillConversationReference": {
            $exists: true,
          },
        },
    },
    {
      $project:
        {
          _id: "$document.activeSkill.conversationId",
          document:
            "$document.skillConversationReference",
          expiresAt: "$expiresAt",
        },
    },
    {
      $merge: "aura-context",
    },
  ])