Aura deployment troubleshooting

1: Check Hugging Face embedding models downloading
2: Migrate Aura Bot and Groot context database to Linkin Park
3: Migrate Aura Groot context database (Dire Straits)

Deployment troubleshooting

Documents related to troubleshooting tasks in Aura deployment

Index

Check Hugging Face embedding models downloading

1 - Check Hugging Face embedding models downloading

Check Hugging Face embedding models downloading

Guidelines to check if the Hugging Face models used in ATRIA are downloaded during the generate-db process

Introduction

The free embedding templates we are currently using in ATRIA are paraphrase-multilingual-MiniLM-L12-v2 and multi-qa-distilbert-cos-v1 both from Hugging Face. (These models are the ones used with the following embeddings by default available in ATRIA: Local Sentence Transformer and Distilbert-based Local Sentence Transformer).

During the generate-db process, these models are loaded into memory and the process may fail if there is a connection problem with Hugging Face. In this error scenario, the only solution is to wait until the service is again up and running.

In the current document, we include the instructions to check if the embedding models can be downloaded, in order to detect the process failure.

Prerequisites

Install huggingface-cli
```
pkgx install huggingface-cli
```

Check if the Hugging Face models are downloaded properly

The way to check if the service is up is by launching the following command:

huggingface-cli download sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

If the download starts, the service is up, and you can restart the generate-db process.

2 - Migrate Aura Bot and Groot context database to Linkin Park

Migrate Aura Bot and Aura Groot context to Linkin Park

Guidelines for the migration of aura-bot and aura-groot database context schemas from releases previous to Linkin Park to Linkin Park onwards. Additionally, the rollback procedure is included.

Prerequisites

Mandatory requisite: A valid kubeconfig for the environment must be available.
Recommendation: Create a snapshot/backup of the current DB data.

Process for context DB migration

Access MongoDB console

To access and open MongoDB console we can use the kubectl command below to export the intended environment, with MongoDB adminuser password. Use the right environment and database name in the commands.

$ export ATLAS_DEPLOYMENT_NAME=aura-br-pro-70 # Get from kubectl get -n aura-system atlasdeployments.atlas.mongodb.com
$ export ATLAS_DATABASE_NAME=aura-groot-br-pro # Set with the correct DB of your deployment
$ export MONGODB_USER=$(kubectl get -n aura-system atlasdatabaseusers.atlas.mongodb.com mongodb-adminuser-password -o jsonpath='{.spec.username}')
$ export MONGODB_PASSWORD=$(kubectl get -n aura-system secret mongodb-adminuser-password -o go-template='{{ .data.password | base64decode }}')
$ export MONGODB_URI="$(kubectl get -n aura-system atlasdeployments.atlas.mongodb.com $ATLAS_DEPLOYMENT_NAME -o jsonpath='{.status.connectionStrings.standardSrv}'|sed "s|srv://|srv://$MONGODB_USER:$MONGODB_PASSWORD@|g")/${ATLAS_DATABASE_NAME}"
$ kubectl -n aura-system run fix-script -i -q --rm --restart=Never --image=mongo:latest --command mongosh ${MONGODB_URI}

Migrate aura-bot context collection to new schema

Update documents to the new format:

> db["aura-context"].updateMany(
      {
        document: {
          $exists: true
        }
      },
      [
        {
          $set: {
            eTag: "$document.eTag",
            conversationData: { $mergeObjects: ["$conversationData", "$document.conversationData"] },
            dialogState: { $mergeObjects: ["$dialogState", "$document.dialogState"] },
            userData: { $mergeObjects: ["$userData", "$document.userData"] },
            userAuthData: { $mergeObjects: ["$userAuthData", "$document.userAuthData"] }
          }
        },
        { $unset: "document" }
      ]
    );

Migrate aura-groot context collection to new schema

Update documents to the new format:

> db["aura-context"].updateMany(
      {
        document: {
          $exists: true
        }
      },
      [
        {
          $set: {
            eTag: "$document.eTag",
            activeSkill: { $mergeObjects: ["$activeSkill", "$document.activeSkill"] },
            conversationHistory: { $concatArrays: [{ $ifNull: ["$conversationHistory", []] }, { $ifNull: ["$document.conversationHistory", []] }] },
            conversationDataProperty: { $mergeObjects: ["$conversationDataProperty", "$document.conversationDataProperty"] },
            dialogContext: { $concatArrays: [{ $ifNull: ["$dialogContext", []] }, { $ifNull: ["$document.dialogContext", []] }] },
            skillConversationReference: { $mergeObjects: ["$skillConversationReference", "$document.skillConversationReference"] }
          }
        },
        { $unset: "document" }
      ]
    );

Rollback aura-bot to the previous schema

If a rollback is needed, documents should be modified to have the previous schema and maintain the users’ context data:

> db["aura-context"].updateMany(
      {},
      [
        {
          $set: {
            document: {
              eTag: "$eTag",
              conversationData: { $mergeObjects: ["$document.conversationData", "$conversationData"] },
              dialogState: { $mergeObjects: ["$document.dialogState", "$dialogState"] },
              userData: { $mergeObjects: ["$document.userData", "$userData"] },
              userAuthData: { $mergeObjects: ["$document.userAuthData", "$userAuthData"] }
            }
          }
        },
        {
          $unset: [ "eTag", "conversationData", "dialogState", "userData", "userAuthData"]
        }
      ]
    );

Rollback aura-groot to the previous schema

If a rollback is needed, documents should be modified to have the previous schema and maintain the users’ context data:

> db["aura-context"].updateMany(
      {},
      [
        {
          $set: {
            document: {
              eTag: "$eTag",
              activeSkill: { $mergeObjects: ["$document.activeSkill", "$activeSkill"] },
              conversationHistory: { $concatArrays: [{ $ifNull: ["$document.conversationHistory", []] }, { $ifNull: ["$conversationHistory", []] }] },
              conversationDataProperty: { $mergeObjects: ["$document.conversationDataProperty", "$conversationDataProperty"] },
              dialogContext: { $concatArrays: [{ $ifNull: ["$document.dialogContext", []] }, { $ifNull: ["$dialogContext", []] }] },
              skillConversationReference: { $mergeObjects: ["$document.skillConversationReference", "$skillConversationReference"] }
            }
          }
        },
        {
          $unset: [ "eTag", "activeSkill", "conversationHistory", "conversationDataProperty", "dialogContext", "skillConversationReference"]
        }
      ]
    );

3 - Migrate Aura Groot context database (Dire Straits)

Migrate Aura Groot context (Dire Straits)

Guidelines for the migration of aura-groot database context schema from releases previous to Dire Straits to Dorian release onwards. Additionally, the rollback procedure is included.

Prerequisites

Mandatory requisite: A valid kubeconfig for the environment must be available.
Recommendation: Create a snapshot/backup of the current DB data.

Process for context DB migration

Access MongoDB console

$ export ATLAS_DEPLOYMENT_NAME=aura-br-pro-70 # Get from kubectl get -n aura-system atlasdeployments.atlas.mongodb.com
$ export ATLAS_DATABASE_NAME=aura-groot-br-pro # Set with the correct DB of your deployment
$ export MONGODB_USER=$(kubectl get -n aura-system atlasdatabaseusers.atlas.mongodb.com mongodb-adminuser-password -o jsonpath='{.spec.username}')
$ export MONGODB_PASSWORD=$(kubectl get -n aura-system secret mongodb-adminuser-password -o go-template='{{ .data.password | base64decode }}')
$ export MONGODB_URI="$(kubectl get -n aura-system atlasdeployments.atlas.mongodb.com $ATLAS_DEPLOYMENT_NAME -o jsonpath='{.status.connectionStrings.standardSrv}'|sed "s|srv://|srv://$MONGODB_USER:$MONGODB_PASSWORD@|g")/${ATLAS_DATABASE_NAME}"
$ kubectl -n aura-system run fix-script -i -q --rm --restart=Never --image=mongo:latest --command mongosh ${MONGODB_URI}

Migrate context collection to new schema

Update documents to the new format:

> db["aura-context"].aggregate([
      {
        $match: {
          _id: {
            $regex: "/conversations/",
          },
          "document.activeSkill.conversationId": {
            $exists: true,
          },
        },
      },
      {
        $lookup:
          {
            from: "aura-context",
            localField:
              "document.activeSkill.conversationId",
            foreignField: "_id",
            as: "document.skillConversationReference",
          },
      },
      {
        $set: {
          "document.skillConversationReference": {
            $arrayElemAt: [
              "$document.skillConversationReference",
              0,
            ],
          },
        },
      },
      {
        $set: {
          "document.skillConversationReference":
            "$document.skillConversationReference.document",
        },
      },
      {
        $merge: "aura-context",
      },
    ])

Check if changes have been applied. The following query should return n documents:

  > db["aura-context"].find({"document.skillConversationReference": { $exists: true }})

Rollback to the previous schema

If a rollback is needed, documents should be modified to have the previous schema and maintain the users’ context data:

> db["aura-context"].aggregate([
    {
      $match:
        {
          "document.activeSkill.conversationId": {
            $exists: true,
          },
          "document.skillConversationReference": {
            $exists: true,
          },
        },
    },
    {
      $project:
        {
          _id: "$document.activeSkill.conversationId",
          document:
            "$document.skillConversationReference",
          expiresAt: "$expiresAt",
        },
    },
    {
      $merge: "aura-context",
    },
  ])