Aura – aura-nlp

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP Adapters

What are Aura NLP adapters

Adapters are used in order to modify the response achieved by the rest of the pipeline stages.

The different types of adapters are included in the following sections.

Description

The specific intent recognition stage (CLU, Exact match, Grammars, etc.) recognizes the user’s intent.
Aura NLP will provide as an output a recognized intent and an entity ID.

Path

Files

Keys: intent
Value: the value for each key corresponds to an ID from different sources.

An example is shown below. Each intent is mapped with a specific ID.

```typescript
{
"intent.exact-match.faq.1": "1",
"intent.exact-match.faq.2": "2",
"intent.exact-match.faq.3": "3"
}
```

Afterwards, aura-bot will receive the recognized intent and the entity ID as an output from the NLP stage.

Review that all the included IDs in this file are existing in the corresponding sources and the matching between the intents and entities for this ID.
Review that the intent name is previously defined.
Include the intents in alphabetical order.
It is recommended to include in the E2E tests phrases to validate that the mapping is correctly done.

Configuration

This stage requires the following configuration in the nlp.json configuration file, in which the field intent_template should point to the use case intent.

As an example, if Exact match is the intent recognizer stage, it can be:

        "intent_template":"intent.exact-match.faq"
        }

Intent Entity Mapper

Description

It can be used in:

Personalized experiences to configure a particular entity based on a specific intent.

In both scenarios:

An intent recognition stage (CLU, Exact match, Grammars, etc.) recognizes the user’s intent.
Intent Entity Mapper adapter is trained to map the intent with an entity name and label.
Aura NLP provides as an output the recognized intent and entity.

Path

auracog_pipelines.stage_wrappers.adapter_wrapper.intent_entity_mapper_wrapper.IntentEntityMapperWrapper

Files

This stage requires an intent_entity_mapper.json file. It maps a specific intent with an entity name and label:

The keys are the intent names.
Values for each key contain another key-value pair:
- entity_name (mandatory): The name of the new entity.
- entity_label (optional): The label of the new entity. If not provided, it uses the label defined in the nlp.json configuration file.

An example is shown below:

{
    "intent.exact-match.faq.1": {
        "entity_name": "1",
    },
    "intent.exact-match.faq.2": {
        "entity_name": "2",
        "entity_label": "test-label" 
    },
    "intent.exact-match.faq.3": {
        "entity_name": "3"
    }
}

Afterwards, aura-bot will receive the recognized intent and the entity name as an output from the NLP stage.

Best practices to edit `intent_entity_mapper.json`

Review that all the included entity names in this file exist in the corresponding sources.
Review the matching between the intents and entities.
Review that the intent name is previously defined.
Include the intents in alphabetical order.
It is also recommended to include it in the E2E test phrases to validate whether the mapping is correctly done.

Configuration

This stage requires the following settings in the nlp.json configuration file, under the intent_entity_mapper key:

intent_template (mandatory): The intent name to replace the original intent.
entity_label_template (optional, default None): A default label name for the entities not defined in the intent_entity_mapper.json file.
entity_type_template (optional, default faq): The type of the new entities.
1. The OpenaiEmbeddingsRecognizer stage is included in the same pipeline.

Here is an example:

"intent_entity_mapper": {
    "intent_template": "intent.tv.search",
    "entity_label_template": "REAL MADRID",
    "entity_type_template": "ent.audiovisual_sports_team",
}

None Handler

Description

None Handler is a stage used when the intent recognized by the pipeline stages is None. It modifies the None intent by the intent predefined in the file none_mapper.json for the specific domain. You can select any intent defined in the system to be pointed in this adapter.

Path

auracog_pipelines.stage_wrappers.adapter_wrapper.none_handler_wrapper.NoneHandlerWrapper

Files

None Handler requires one file: none_mapper.json that indicates, within a specific domain, which intent must be set if the recognized intent is None. Therefore, if the domain is already defined and the system recognizes the intent None, then the intent is replaced by the value indicated in the file.

In this file:

Keys: different domains
Values: value for each key is the intent mapped with this domain.

Developers can select any intent defined in the system to be pointed in this adapter.

An example is shown below:

{
  "domain.tv_content": "intent.tv.none",
  "domain.tv_desco": "intent.tv.none",
  "domain.tv_record": "intent.tv.none",
  "domain.tv_wh3": "intent.tv.none"
}

In the example, if the domain is domain.tv_content and the system recognizes the intent None, then this intent is replaced by intent.tv.none.

Configuration

This stage does not require any configuration.

Standard Threshold

Description

Standard Threshold allows the establishment of a threshold for the scores provided by preceding pipeline stages. If the score obtained by the previous stages is lower than the established threshold, the Standard Threshold provides score 0 and the intent is replaced by the default value set in the configuration.

This stage is useful to prevent false positives and can be included in any place of the pipeline. The NLP Global Team recommends to set this threshold to 0.6.

Path

auracog_pipelines.stage_wrappers.adapter_wrapper.standard_threshold_wrapper.StandardThresholdWrapper

Files

Standard Threshold does not require any file.

Configuration

This stage requires the following configuration in the nlp.json file, depending on the language and the channel:

{
  "es-es": {
    "mp": {
      "threshold": {
         "default": {
           "threshold": 0.1,
           "intent": "intent.default" 
        },
         "intent.test": {
            "threshold": 0.8,
            "intent": "None" 
        }
      } 
    }     
  }

The fields are explained below:

threshold: value between 0 and 1 indicating the limit that triggers the action of the adapter.
- This field can be defined per intent, thus having a different threshold for each intent.
- The default intent must be always specified and, additionally, you can define a different threshold for other specific intent in order to improve the recognition process.
- The NLP Global Team recommends to set this threshold to 0.6.
intent: this field contains an internal string identifier, that is associated if the score value is lower than the threshold.

In the previous example, the default config applies to every intent, with accuracy 0.1 and intent intent.default. But, specifically for the intent intent.test, the applied configuration has accuracy 0.8 and the associated intent is None.

Entity Tagger Adapter

Description

Entity Tagger Adapter is a stage that allows entities tagging through the definition of aliases and labels on them.

Path

auracog_pipelines.stage_wrappers.adapter_wrapper.entity_tagger_adapter_wrapper.EntityTaggerAdapterWrapper

Files

The Entity Tagger Adapter stage requires four files:

sdict_items: Items dictionary including the canonical values of entities.
sdict_aliases: Alias dictionary including the aliases for an entity.
ner_entity_translation.json
ner_aliases.json

ner_entity_translation.json

ner_entity_translation.json indicates, for each entity type, if canon and label have been defined for the entity type or not.

It is a JSON file that must be generated manually to define properties for each entity type.

The keys are entity types.
Values for each key contain another key-value pair:
- Keys: canon and label
- Values: boolean operators (true/false) depending on whether a canon and/or label has been defined for a given entity type or not.

An example of ner_entity_translation is shown below:

{
  "DEFAULT": {
    "canon": false,
    "label": false
  },
  " ent.audiovisual_actor ": {
    "canon": true,
    "label": false
  }
}

Where default represents the default value if the entity type is not included in the dictionary. The default field is not mandatory if all the entity types are defined in the file.

When a new entity is added, it must be included in the ner_entity_translation.json file with the expected behavior for the canon and label. Likewise, if the behavior for canon and label of an entity changes, it is required to update this file.

As best practices, entities should be ordered alphabetically.

ner_aliases.json

ner_aliases.json is an optional file required if you need to match/map the canon value to some other value requested by an API or search engine (currently, it is used by Spain to search content in the M+ database).

It is a JSON dictionary that must be generated manually, where:

Keys: entity types.
The value of each key includes another key-value pair:
- Keys: labels
- Value for each key: list of values contemplating just the canonical form(s) of the entity label

When adding a new entity, in case the entity should have a label, the label has to be assigned to the canons that we want to be identified by this label.

An example of ner_aliases dictionary is shown below:

{
    "ent.audiovisual_genre": {
        "CN": [
            "movies",
            "films"
        ],
        "CT": [
            "shorts",
            "short film"
        ]
    },
}

The following example shows the mapping between alias-canon-label and their corresponding files:

   docu (`sdict_aliases`) -> documentary (`sdict_items`)-> DC  (`ner_aliases`)

Best practices for the edition of ner_aliases.json

Canon names should be expressed as in sdict_items.json, including capitalization, diacritic marks and punctuation.
Entities should be ordered alphabetically.
Labels inside entities should be ordered alphabetically.

Configuration

No configuration is required.

Description

If the disambiguation process cannot discern between multiple entities, then the intent returned by Aura NLP to the bot will be a disambiguation intent as a top intent at first level. In this intent, each option is composed of the original top intent and one entity per option. These options will be presented to the user for him to choose the most appropriate one.

The input for this stage includes: a list of intents; 1 top intent (intent recognized with the higher score); a list of entities.
The output from this stage includes: the top intent and different options (options in the data model) of recognized entities.

Here is an example of the input and output data models for the disambiguation by entities stage, where entities are IDs.

Input data model:

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [
            {
                'entity': '14', 'type': 'faq', 'score': 0.90, 'start_index': 1, 'end_index': 1, 'canon': '14', 'label': null
            },
            {
                'entity': '16', 'type': 'faq', 'score': 0.88, 'start_index': 1, 'end_index': 1, 'canon': '16', 'label': null
            },
            {
                'entity': '12', 'type': 'faq', 'score': 0.60, 'start_index': 1, 'end_index': 1, 'canon': '12', 'label': null
            }
        ],
        'intents': [
        ]
    },
    'domain_result': {}
}

Output data model:


    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [],
        'top_result': {'intent': 'intent.disambiguation', 'score': 1.0},
        'intents': [
            {'intent': 'intent.disambiguation', 'score': 1.0}
        ]
    },
    'domain_result': {},
    'options': [
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': '14', 'type': 'faq', 'score': 0.90, 'start_index': 1, 'end_index': 1, 'canon': '14', 'label': null
                    }
                ],
                'intents': [
                ]
            }
            'domain_result': {},
            'options': []
        },
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': '16', 'type': 'faq', 'score': 0.88, 'start_index': 1, 'end_index': 1, 'canon': '16', 'label': null
                    }
                ],
                'intents': [
                ]
            },
            'domain_result': {},
            'options': []
        }
    ]
},

Path

Files

Configuration

This stage requires the following configuration in the nlp.json file depending on the language and the channel:

{
  "es-es": {
    "mp": {
        "disambiguation_margin": 0.2,
        "exact_match": 0.92,
        "intent_template": "intent.disambiguation",
      } 
    }     
  }
}

The fields determine how the disambiguation process is carried out:

intent_template: This field contains an internal string identifier corresponding to the intent resulting from this stage if no disambiguation can be carried out.
exact_match: value between 0 and 1. This value is used to check if any recognized entity score is above this value.
disambiguation_margin: value between 0 and 1 used to set an interval.

The combination of these two last parameters provides different scenarios:

The score of certain entities is equal or higher than exact_match (left graphic).
- Only the entity with the best score is considered.
- In case of tie (more than one entity with the highest score), all of them are returned.
The score of all entities is below the exact_match (right graphic). Only those entities whose score is in the interval: ([top score], [top score - disambiguation_margin]) (both included) are considered.

Intent Disambiguation Adapter

Description

The goal of this stage is to disambiguate when, in an utterance, several intents are recognized.

If the disambiguation process cannot discern between multiple intents, then the intent returned by this stage will be a disambiguation intent as a top intent at first level. In this intent, each option is composed of one intent (that fulfils the conditions to be disambiguated) and a list of entities. These options will be presented to the user to choose the most appropriate one.

The general behavior of this stage is explained below:

The input for this stage includes: a list of intents and a list of entities.
The output for this stage includes: the different options for intents recognized during the disambiguation (options in the data model) and the original list of entities.

However, if black lists are defined, the behavior is explained in the following sub-section.

Intent disambiguation with a blacklist of intents

Aura NLP allows the integration of configurable blacklists of intents for a custom behavior of disambiguation. In this case, the disambiguation mechanisms will not apply for the intents included in the blacklist. The use case constructors can edit a blacklist of intents in the nlp.json configuration file, filling the parameter intent_blacklist.

When there is a blacklist of intents, the disambiguation process behaves as explained below:

a. If the top scored intent is included in the intent_blacklist, the pipeline will return this unique intent (no disambiguation is launched).

b. If the top scored intent is not included in the intent_blacklist, then the predefined values of the configuration parameters come into play:

If the score of the top scored intent is higher than exact_match, then this intent is returned. In case of tie (more than one intent with the highest score), all of them are returned.
If the score of the top scored intent is lower than exact_match, then all the intents whose score is in the interval between the top score and the disambiguation_margin and are not in the intent_blacklist are returned.
In this case, the final intent will be the one described in intent_template (with a score of 1.0) and the selected intents will be placed in the options of the result.

Path

auracog_pipelines.stage_wrappers.adapter_wrapper.intent_disambiguation_wrapper.IntentDisambiguationAdapterWrapper

Files

No files are required.

Configuration

This stage requires the following configuration in the nlp.json file for each country and channel, within the key intent_disambiguation.

The following parameters are required for this stage:

exact_match: Float number, value between 0 and 1. If the intent with the highest score is greater than this value, the result is this intent (if this intent is not included in the intent_blacklist).
disambiguation_margin: Float number. Margin between the highest score and the lower score considered for the response.
intent_template: String. Name of the intent that the stage returns when there are multiple options as response.
intent_blacklist: List of intents for which the disambiguation mechanisms will not apply. This parameter is mandatory. If there are no blacklisted intents, it will have to be an empty list.
⚠️ none intent must always be included in the blacklist, as it is not going to be offered as an option to disambiguate.

See an example of nlp.json file configuration for this stage:

{
  "es-es": {
    "mp": {
      "intent_disambiguation": {
         "disambiguation_margin": 0.2,
         "exact_match": 0.92,
         "intent_template": "intent.disambiguation",
         "intent_blacklist": ["intent.blacklisted_intent1", "intent.blacklisted_intent2"]
      },
      ...
    }
  }
}

Examples of intent disambiguation adapter

Basic example for intent disambiguation adapter inclusion in a pipeline

{
  "name": "IntentDisambiguationExample",
  "initial_node_id": "IntentDisambiguationExamplePipeline",
  "elements": {
    "IntentDisambiguationExamplePipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
    "RecognizerWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.recognizer_wrapper"
    },
    "IntentDisambiguationAdapterWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.intent_disambiguation_wrapper.IntentDisambiguationAdapterWrapper"
    }
  },
  "links": {
    "IntentDisambiguationExamplePipeline": [
      "RecognizerWrapper",
      "IntentDisambiguationAdapterWrapper"
    ]
  }
}

General behavior of intent disambiguation stage (with no blacklist of intents)

Here is an example of the input and output data models for the intent disambiguation stage, belonging to the OpenAI embeddings stage, where the disambiguation margin is 0.2:

Input data model

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [
            {
                'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1,         'end_index': 2, 'canon': 'canon1', 'label': 'label1'
            },
            {
                'entity': 'name2', 'type': 'type2', 'score': 0.78, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label2'
            }
        ],
        'top_result': {'intent': 'intent.tv.search', 'score': 0.96},
        'intents': [
            {'intent': 'intent.tv.search', 'score': 0.96},
            {'intent': 'intent.tv.display', 'score': 0.90},
            {'intent': 'intent.tv.launch', 'score': 0.60}
        ]
    },
    'domain_result': {}
}

Output data model (2 intents fulfil the predefined criteria)

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [],
        'top_result': {'intent': 'intent.disambiguation', 'score': 1.0},
        'intents': [
            {'intent': 'intent.disambiguation', 'score': 1.0}
        ]
    },
    'domain_result': {},
    'options': [
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
                    },
                    {
                        'entity': 'name2', 'type': 'type2', 'score': 0.78, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label2'
                    }
                ],
                'top_result': {'intent': 'intent.tv.search', 'score': 0.96},
                'intents': [
                    {'intent': 'intent.tv.search', 'score': 0.96}
                ]
            },
            'domain_result': {},
            'options': []
        },
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
                    },
                    {
                        'entity': 'name2', 'type': 'type2', 'score': 0.78, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label2'
                    }
                ],
                'top_result': {'intent': 'intent.tv.display', 'score': 0.90},
                'intents': [
                    {'intent': 'intent.tv.display', 'score': 0.90}
                ]
            },
            'domain_result': {},
            'options': []
        }
    ]
}

Behavior of intent disambiguation stage with a blacklist of intents.
Scenario 1: top intent is included in the blacklist

Input data model

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [
            {
                'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
            }
        ],
        'top_result': {'intent': 'intent.common.greetings', 'score': 0.96},
        'intents': [
            {'intent': 'intent.common.greetings', 'score': 0.96},
            {'intent': 'intent.tv.display', 'score': 0.90},
            {'intent': 'None', 'score': 0.60}
        ]
    },
    'domain_result': {}
}

Output data model: the top intent is included in the blacklist, therefore, no disambiguation is launched.

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [
            {
                'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
            }
        ],
        'top_result': {'intent': 'intent.common.greetings', 'score': 0.96},
        'intents': [
            {'intent': 'intent.common.greetings', 'score': 0.96},
            {'intent': 'intent.tv.display', 'score': 0.90},
            {'intent': 'None', 'score': 0.60}
        ]
    },
    'domain_result': {}
}

Behavior of intent disambiguation stage with a blacklist of intents
Scenario 2: top intent is included in the blacklist

Input data model

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [
            {
                'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
            }
        ],
        'top_result': {'intent': 'intent.tv.search', 'score': 0.96},
        'intents': [
            {'intent': 'intent.tv.search', 'score': 0.96},
            {'intent': 'None', 'score': 0.90}
            {'intent': 'intent.tv.display', 'score': 0.89}
        ]
    },
    'domain_result': {}
}

Output data model: the top intent is not included in the blacklist and more than one intent fulfil the condition for disambiguation » The options in the blacklist are ignored and the remaining intents are disambiguated.

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [],
        'top_result': {'intent': 'intent.disambiguation', 'score': 1.0},
        'intents': [
            {'intent': 'intent.disambiguation', 'score': 1.0}
        ]
    },
    'domain_result': {},
    'options': [
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
                    }
                ],
                'top_result': {'intent': 'intent.tv.search', 'score': 0.96},
                'intents': [
                    {'intent': 'intent.tv.search', 'score': 0.96}
                ]
            },
            'domain_result': {},
            'options': []
        },
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
                    }
                ],
                'top_result': {'intent': 'intent.tv.display', 'score': 0.89},
                'intents': [
                    {'intent': 'intent.tv.display', 'score': 0.89}
                ]
            },
            'domain_result': {},
            'options': []
        }
    ]
}

Length Adapter

Description

The objective of this stage is to control the maximum and minimum length of a phrase in order to avoid it to be too short or too long. The maximum/minimum number of characters is configurable. The stage discards the out-of-range phrases, as they are not recognized properly by Aura NLP, thus saving time and resources in the recognition process.

The Length Adapter returns a configurable intent template if the length of the received phrase in the pipeline message is higher than the maximum number of configurable characters or lower than the minimum one. The intent template is also configurable.

Path

auracog_pipelines.stage_wrappers.adapter_wrapper.lenght_adapter_wrapper.LenghtAdapterWrapper

Files

No files required.

Configuration

This stage requires a specific configuration in the pipeline configuration file pipeline.json, within the args section of this file, that contains the following fields:

max: maximum number of characters in the phrase of the received pipeline message.
min: minimum number of characters in the phrase of the received pipeline message.
intent_template: intent name to be returned if the number of characters is lower than the min value or higher than max value.

      "args": {
         "max": 50,
         "min": 1,
         "intent_template": "intent.example"
      }

You can also configure more than one stage of the Length Adapter to return different intents for max or min length characters.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

How to obtain Azure credentials for OpenAI

This process may be required in the first step for training the understanding model: Set up configuration properties.

Prerequisites

Pre-requisites:

Azure account with permissions for applications registration.
Azure CLI installed.

Guidelines

Review azure-cli documentation to validate the commands and parameters.

Follow the guidelines below for obtaining the Azure credentials for OpenAI:

Run the login command (documentation): az login
Sign in with your account credentials in the browser.
You will obtain the different subscriptions within Azure corresponding to the logged account.

Select the specific subscription to be used, with its corresponding field id, and execute the following command to switch to this subscription (documentation): az account set --subscription <subscription_id>
- <subscription_id> is the id of the selected subscription
Create a resource group (documentation): az group create --name <name_resource_group> --location <location>
- <name_resource_group>: name of the resource group
- <location>: one location available for Azure (i.e., northeurope)
Create app (documentation): az ad app create --display-name <display_name>
- <display_name>: name of the service principal
  From the output of az ad app create, we can obtain the field appId. This value is used for the variable OAI_AZURE_TOKEN_CLIENT_ID.
Create password for app (documentation): az ad app credential reset --id <app_id>
- <app_id>: app_id obtained from previous app creation
  From the output of az ad app credential reset, we can obtain the field password. This value is used for the variable OAI_AZURE_TOKEN_CLIENT_SECRET.
  From the output of az ad app credential reset, we can obtain the field tenant. This value is used for the variable OAI_AZURE_TOKEN_TENANT.
Create service principal (documentation): az ad sp create --id <app_id> - AppId: app_id obtained from previous app creation
Assign role contributor (documentation): az role assignment create --assignee <appId> --role Contributor --scope <scope> - <app_id>: app_id obtained from previous app creation
- <Scope>: scope of the role assignment. Read more in (documentation). A possible value is the of the resource group, you can obtain it with the command az group show --name <name_resource_group> | jq .id (documentation).
Create the OpenAI application (documentation): az cognitiveservices account create --kind "OpenAI" --name <name_openai> -g <name_resource_group> --sku s0 -l <location>
- <name_openai>: resource name
- <name_resource_group>: name of resource group (previously generated)
- <location>: location available for Azure (i.e., northeurope)

The values for the parameters required to fill in the build_local_variables.sh script for OpenAI execution must be obtained from the above-defined steps:

export OAI_ID_SUBSCRIPTION="$(az account show | jq -r .id)"
export OAI_RESOURCE_GROUP="<name_resource_group>"
export OAI_ACCOUNT_NAME="<name_openai>"
export OAI_AZURE_TOKEN_CLIENT_ID="<app_id>"
export OAI_AZURE_TOKEN_CLIENT_SECRET="<password>"
export OAI_AZURE_TOKEN_TENANT="$(az account show | jq -r .tenantId)"

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Guidelines for the generation of Grammars in Unitex

Guidelines and best practices for working with Unitex for the generation of the Grammars to be included in the NLP model.

General guidelines

Grammars is an Aura NLP stage that has its own path, files and configuration required to be included in the NLP model.

Firstly, if your pipeline contains the Grammar stage, you need to work with Unitex Gramlab and Grape NLP, which are included in the NLP Virtual Machine.

After that, linguists can proceed to create the grammars associated to the new use case. This process will be similar for global and local grammars.

The intent, entities and utterances defined for the new use case must be considered. A representative set of utterances will be selected and represented in Unitex through the creation of connected boxes that will contain, from left to right, different options for expressing each token of the selected utterances. The combination of different connected boxes provides a full variability of utterances to be recognized.

It is necessary to bear in mind that grammar engine only provides an exact recognition of utterances previously integrated in the model. Therefore, it is necessary to build up a rich and realistic utterance database to cover all the representative users’ utterances for a given use case.

Once the grammars have been developed in Unitex, the grammar engine Grape NLP goes through the grammar from the beginning (left side of the graph) and compares box by box the user’s utterance with the grammar to evaluate the matching.

The output will be a set of labels together with a start and end index.

Intents and entities tagging

Tag an intent in the grammar interface

In order to tag an intent in a grammar graph, a box previous to the closing box of the graph should be created with the following information and format:

<E>/<intent.[intent_name]/>

Tag an entity in the grammar interface

Two separate boxes need to be created: one before and one after the entity values.
We need two entity tags because we need to wrap the entity values in order to know its position in the user’s utterance.
Opening entity tag should have the following information and format:
```
<E>/<ent.[entity_name]> 
```
Closing entity tag requires the following information and format:
```
<E>/</ent.[entity_name]>
```
Consider the difference between the opening and closing entity tag and remember that the entity tags need to be included within the entity graph and not outside of it.

Best practices for graphs generation

We highly recommend you the webinar Implementing new use cases: Grammar guidelines.
The graphs must be as visually clear as possible.
Avoid crosslines.
The verbal graphs should be vertically aligned and the arrows connecting boxes should be horizontally aligned.
Be careful when using too much optionality (“Epsilon” symbol), this may lead the grammar to recognize unwanted strings, collisions between UCs, etc.
All graphs should have an appropriate size not to leave info/boxes out of the them.
Try not to repeat the same box structure several times. Try to reuse it for different paths or to create a subgraph that can be reused anywhere in the intent axiom. Hence, avoid creating two or more paths recognizing the same input.
Use comments if needed to clarify, for instance, if a path has some limitations due to potential conflicts with other UCs or just as explanatory notes of what a path is contemplating. For creating a comment within a graph, create a box and do not connect it to any other box. This way, you will see that the characters of comment message appear in red colour.
Avoid leaving empty boxes in any graph.
Avoid typos within the boxes info.
The opening and closing entity tags used for wrapping the entity values should be contemplated in the entity graph and not outside of it.
Make sure the intent and entity tags have been properly included.
When adding prepositions and articles in boxes, put them separately. That is, create a box for the prepositions and another one for the articles.
The circumstantial complements (e.g., time, location, manner…) are optional on many occasions regardless of whether they are in initial, middle or final position.
No graph must recognize the “Epsilon” symbol:
```
 <E>
```
So in case of optional subgraphs, the optionality should be in the graph where it is called and not in the subgraph.
If the grammar makes use of the NER dictionaries to do the matching value > canon > label, the values contemplated in the different dictionaries should be also contemplated in the entity graphs of the grammar for the matching process to be successful.
- It is crucial to consider here that the grammar values contemplated in the entity graph should be normalized (same process as the normalization pipeline carries out except for the normalization of upper-case characters) in order to be recognized.
- That is, if a value in the dictionaries is ‘Mr. Robot’, since the normalization pipeline erases punctuation marks, the value that should be included in the entity graph should be ‘Mr Robot’.
There are special symbols that have specific meaning for the grammar and should be escaped (to check these symbols go to “Encoding of special characters in the graph editor” section of the Unitex Gram Lab official documentation).
An example of a special character would be “+” that needs to be escaped by using ““ (See figure below).
It is highly recommendable to compile the grammar before pushing changes into the Pull Request. This way, the NLP developer will see if there is any error in the call to the subgraphs, if the grammar recognizes an empty path (the grammar recognizes: " “) or if there is any corrupt file.
When compiling the grammar, some files are generated. These files have different extensions (.fst2, .snt, .diff) and should be avoided. Thus, the NLP developer should erase them locally before committing further changes into the PR.
For main verbs or list of keywords, create another graph.
Try to reuse basic structures (grammar block) from one graph to another.
Try to avoid ungrammatical paths if possible.
If one graph gets too complex, try to split it into smaller blocks/subgraphs.

Best practices for the generation of .grf files

Create as many folders as existing domains/intents.

Graphs

Call axiom.grf to the main graph of the whole grammar (general graph that calls to the different domains).
Generate another axiom.grf file in the subfolder of each specific domain, which will be the main graph for this domain (graph that calls to the UCs related to that domain).
Generate another axiom.grf file for each use case/intent. Remember that, if different subgraphs are created to contemplate different structures or entity combinations for a given UC, the intent tag should be found in this general UC/intent axiom and not in the individual subgraphs.

Domain folder

The name of the domain folder should be identical to the name of the corresponding domain.
This way, when opening the main graph of the whole grammar, one could quickly see the domains that have, at least, some UC developed through the grammar engine. Make sure when including a new domain to make the proper call to it in the main axiom of the whole grammar.

Intent folder

The name of the intent folder should contemplate the name of the corresponding intent.
For example: If the intent name is intent.common.greetings, the intent folder name would be greetings.
If a given intent has different sub-use cases or the intent is divided into different graphs according to different linguistic structures/entities combination, the intent tag should be only tagged once in the main axiom of the use case and not in each of the different subgraphs.
Make sure when including a new intent to make the proper call to it in the domain axiom.

Entity graphs

The entity graphs should have the name of the corresponding entity name, that is, if an entity name is ent.device_tv, the name of the graph in the grammar folder should be ent.device_tv.grf.
If an entity is only used in one use case/intent, the entity graph should be located in the intent folder.
Otherwise, if an entity is used in different intents/UCs of the same domain, the entity graph should be located in the main folder of the domain.
If an entity is used in different domains, the entity graph should be in the main folder of the whole grammar.

Verbal structure and nomenclature

Introduction

Verbal graphs need to be adapted based on the target language the NLP developer is working with.

The following verbal forms and tenses have been provided in Spanish as illustrative examples because the Spanish language varies morphologically depending on the person/number and tense info.

Nomenclature of auxiliary verbs:
- Auxiliary verbs: They serve, among other things, to form the compound tenses, the progressive forms, the passive voice, as well as negations and questions (e.g., “I would like to eat an apple”).
- Main/Full verbs: They add meaning to the sentence and are essential for understanding the statement (e.g., “I eat two apples every morning”).

To this end, the NLP team has been working on an efficient structure and nomenclature to have them contemplated in the grammar.

Auxiliary verbs

aux_W: auxiliary verb + infinitive tense

It contemplates all the possible auxiliary verbs that can be found before a verb in infinitive tense.

This graph should be always optional since the infinitive tense without the auxiliary verb is also acceptable in linguistic terms (e.g., “I want to check my agenda” & “Check my agenda” are both linguistically correct).

aux_Y: auxiliary expressions/verbs + imperative tense

aux_Y2s: second person singular in imperative tense.
aux_Y3s: third person singular in imperative tense.
aux_Y2p: second person plural in imperative tense.
aux_Y3p: third person plural in imperative tense.

All these graphs contain possible auxiliary expressions that may be found before verbs in imperative tense.

These graphs should be also optional as the verbs in this tense can be also found in isolation (e.g., “Go and bring me some water” & “Bring me some water”). All these graphs should be found within the graph containing all imperative tenses of a given verb (e.g., verb_Y.grf -> buy_Y.grf).

aux_SQT: auxiliary verb + present/past imperfect subjunctive tense

It contains all possible expressions that can be found before a verb in present imperfect subjunctive tense (S) and past imperfect subjunctive tense (QT).

This graph should be always mandatory in these two tenses as the structure “I would like you to bring me some water” vs. “I like you to bring me some water” would be agrammatical without the modal verb.

Main verbs

This graph is used for ambivalent verbs that could work both as auxiliary and main/full verbs.

When this graph is used, all these verbs are conceived as being main/full verbs and thus do not need to be accompanied by another verb but by a complement in the form of a noun (e.g., “I want some water” vs. “I want to drink some water”).

Verbal graphs

The names of the verbal graphs should be in English. This also applies to the name of the domain, intent, keywords and complement graphs.
Before creating any verbal graph, make sure it is not repeated, that is, it is not already contemplated in any other of the grammar UCs. If a given verb is only used in a particular UC, place the verbal graphs within the folder of the UC. If a verb is shared by several UCs of the same domain, place the verbal graph at the domain folder level.
If, on the other hand, a given verb is used by UCs belonging to different domains, place the verbal graphs in the folder of the whole grammar.
If you create different verbal subgraphs for different verbal tenses, make sure you include all the verbs in each tense. The basic verbal tenses included for each verb are:

In case the developer needs to contemplate more verbal tenses, please consult the section “Common inflectional codes” in Unitex Gram Lab official documentation.

[verb]_main_graph structure

To ease the grammar development process, we propose a common structure for all verbs.

The name of the verbal graph would be [verb]_main_graph.grf and would have the following structure:

As it can be appreciated in the example, some pronouns have been added between aux_SQT and the verbal boxes for those tenses. This also happens before P3s/p, P2s and C2s verbal boxes.
These pronouns are needed for sentences such as: “Quiero que me compres este vuelo” (“I want you to buy me this flight”). Besides, the interrogative particle “cómo” has been added before aux_W and before P1s/p tenses for questions such as “¿Cómo puedo comprar este ticket?” (“How can I buy this ticket”) and “¿Cómo compro este ticket?” (“How do I buy this ticket”).

In case the clitic forms of verbs are needed, create a separate graph for them. For this, create a verbal graph called [verb]_clitic_forms.grf. As in the previous example, some pronouns have been included in some of the paths, but the main difference here would be that the boxes containing the clitic pronouns have been included in a mandatory way.

As seen in the figure, this graph would recognize sentences such as: “Quiero que me lo compres” (“I want you to buy it for me”).

Guidelines for testing Grammars in Unitex

There are two alternative ways of testing the grammars generated with Unitex:

1. Testing grammars using the Unitex interface

Useful for checking the potential overlaps among the different use cases developed through the grammar engine.

For this purpose, a .txt file should be created with the testing statements (sentences, phrases or isolated words) to be tested (each one on a different text line).

Keep in mind that when testing the grammar in the Unitex interface, the testing statements are not going to be processed with the normalization pipeline and, thus, must consider capitalization, accentuation marks, etc.

Afterwards, the following instructions should be carried out:

Open the Unitex interface and go to the menu found in the upper section of the interface.
Click on “Text > Open” and open the .txt previously created (say “OK” to Process Text).
Once the .txt file is selected, go to “Text > Locate Pattern” and a window will pop up in which it is required to select the graph to be used to process the testing examples. The “Merge with input text option” should be also selected to see the intent and entity tags.
Besides, we need to select the “Activate debug mode option” to see the paths activated in each of the testing examples as well as the “Longest matches option” to replicate how the system works.
Click on “Search” and another window will pop up with the results.

If all the provided examples have obtained the expected intent, that means that the grammar engine has the expected behaviour.

2. Testing grammars through run_local_pipeline.sh scripts

The second option would be to launch queries through the run_local_pipeline.sh script to validate their intent assignment (remember that you must previously execute the build_local.sh script in order to train the pipeline).

Since the grammar engine is deterministic, based on the score (confidence) obtained, it is possible to know whether an utterance has been solved by CLU or by the grammar engine:

Utterances detected by the grammar module obtain a score of 1.0
The scores of utterances detected by CLU are float numbers between 0 and 1 (e.g. 0.98, 0.65…).

This testing option is the most recommendable one since this way we can see the faithful output of how the system works end-to-end.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Logical connectors

Introduction

Connectors are components that connect different NLP stages and control the flow of the pipeline. Specifically, logical connectors use the logical connectives to combine different stages.

They can be classified in four main groups:

Competitive connectors

HigherScorePipeline

Description

Two or more stages competing in a parallel way, in which the result of the stage providing the higher score is selected.

Path

auracog_pipelines.pipelines.joint.competitives.HigherScorePipeline

Configuration

No configuration is required

Conditional connectors

Stages are strung together and executed in a specific way depending on the result of the boolean operators and/or/not.

There are three different types of conditionals connectors, whose result is shown in the figure below.

AndPipeline

Description

This connector is used if all the input conditions must be met in order to have the output true (see figure above).

Path

auracog_pipelines.pipelines.joint.conditionals.AndPipeline

Configuration

No configuration is required

OrPipeline

Description

This connector is used to express that, as long as one out of two or more conditions are met, the value for the output is true (see figure above).

Path

auracog_pipelines.pipelines.joint.conditionals.OrPipeline

Configuration

No configuration is required

NotPipeline

Description

This connector is used to reverse the input value from true to false and from false to true (see figure above).

Path

auracog_pipelines.pipelines.joint.conditionals.NotPipeline

Configuration

No configuration is required

Boolean Connectors

FalsePipeline

Description

The purpose of this connector is to execute in sequential order the stages that the connector contains and to return the status false. This connector ignores the status of the different stages which are contained.

Path

auracog_pipelines.pipelines.joint.conditionals.FalsePipeline

Configuration

No configuration is required.

An example of how to integrate the connector in a pipeline is shown below:

{
  "initial_node_id": "FalseBooleanPipeline",
  "elements": {
    "FalseBooleanPipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.joint.booleans.FalsePipeline"
    },
      "type": "stage",
    }
  },
  "links": {
    "FalseBooleanPipeline": [
    ]
  }
}

TruePipeline

Description

The purpose of this connector is to execute in sequential order the stages that the connector contains and to return the status true. This connector ignores the status of the different stages which are contained.

Path

auracog_pipelines.pipelines.joint.conditionals.TruePipeline

Configuration

No configuration is required.

An example of how to integrate the connector in a pipeline is shown below:

{
  "initial_node_id": "FalseBooleanPipeline",
  "elements": {
    "TrueBooleanPipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.joint.booleans.TruePipeline"
    },
      "type": "stage",
    }
  },
  "links": {
    "TrueBooleanPipeline": [
    ]
  }
}

Sequential connectors

These connectors work as follows: Stage B input is the output of its preceding stage A, with stage B output the result of summing both stages result.

The way the different stages are connected defines how the interactions between them are carried out. For example, two or more stages can run in a simultaneous competitive way, in which the winner is the stage with higher score or stages can be executed in a sequentially way in which a first stage generates information used by the succeeding stage.

BasePipeline

Description

BasePipeline is the simplest connector in charge of the sequential execution of the different stages composing the pipeline. These stages are executed in the specified order.

Path

auracog_pipelines.pipelines.base.BasePipeline

Configuration

No configuration is required

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Nabro normalization pipeline

Description and stages

Nabro is a pipeline used for the normalization of the user’s utterance through the execution of the following normalizers:

PunctuationNormalizer
SplitPunctNormalizer
SpaceNormalizer
CurrencyNormalizer
UnicodeNormalizer
LowercaseNormalizer

Configuration

This stage requires the following configuration in the nlp.json configuration file:

For the specific language and channel, in the nlp field of this JSON file, the key normalizer_pipeline_class must be filled in with the value: auracog_pipelines.pipelines.normalization.nabro.NabroPipeline

{
  "es-es": {
    "mp": {
         "nlp": {
         "normalizer_pipeline_class": "auracog_pipelines.pipelines.normalization.nabro.NabroPipeline"
      }
    }
  }
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Catalog of NLP stages

NLP stages to compose the NLP pipeline

Aura Platform Team has implemented a set of natural language processing (NLP) stages in order to configure different pipelines.

Select your intended stage in the left menu. Each of them is characterized by its description, path, files and configuration.

Section	Content	Role in the NLP process
Description	Identification and objective of the stage in the recognition process	Descriptive purpose of the stage in the recognition process
Path	Class path (Python class) of an element (stage or connector)	The path of each stage of the pipeline must be included in the file `pipeline.json` for building up the NLP dynamic pipeline
File	Specific training files and test set files for the NLP stage required to train and validate the NLP model	Linguists must generate these files for the training and the validation of the NLP model during the data resources definition
Configuration	Required configuration for each NLP stage	Configuration of each stage of the NLP model

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites for working with Aura NLP

Key requirements that are essential to configure the Aura NLP development environment, prior to the generation and training of an understanding model

Introduction

Before starting the development of use cases over Aura NLP, there are certain tasks that must be carried out in order to install and configure this component:

Get sure your Aura system includes the mandatory technical resources for working with Aura NLP.
Install the Aura NLP Virtual Machine.
Generate a local branch for the Aura NLP data repository.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Python repositories migration

Migration of Python repositories from Python 3.9 to Python 3.13 version for PyUtils, Aura NLP, Aura Complex Logic Framework plugin packages and ATRIA repositories

Introduction

This document describes the migration process of Python repositories from Python 3.9 to Python 3.13 ¹ for PyUtils, Aura NLP, Aura Complex Logic Framework plugin packages, ATRIA and other productive Python repositories that has taken place in Prince 10.0.0 release.

This migration is required due to the end of life of Python 3.9, set in October 2025.

PyUtils migration

PyUtils contains multiple repositories that have been migrated to Python 3.13 version by Aura Global NLP Team.

The use of these libraries is the same as before the migration, but OBs constructors must test the use cases to ensure that everything is working correctly.

NLP packages migration

The Aura Global NLP team has migrated the Python repositories to Python 3.13 version and has generated the different NLP packages for the OBs.

Two branches will live together:

release/7.0: Compatible branch with Python 3.9 version. If the deployed OB release is previous to Prince 10.0.0.
release/8.0: Compatible branch with Python 3.13 version. After Prince 10.0.0 release delivery.

The release/8.0 branch will be launched together with Prince 10.0.0 delivery. However, release/7.0 will remain active until the OB deploys Prince in production environment and even after that, for any issue or hotfix required.

The repositories of NLP training has a JenkinsFile ² that defines the pipeline in Jenkins for the continuous integration of the repositories. For release/7.0 branch, the JenkinsFile of Aura NLP repositories will be configured for working with Python 3.9 version in the pipelines defines in Jenkins, to allow OBs to continue creating NLP packages, until they deploy this new release.

When deploying Prince release and changing from release/7.0 to release/8.0, both branches must be synchronized to preserve all modifications made in the older one.

Constructors should not do any change, as no trainings with the QnA and LUIS stages should exist (these stages are deprecated and must have already been replaced by the OpenAIEmbeddings and CLU stages respectively).

With the NLP packages provided by Aura Global NLP team, the OB constructors must test and validate that all the use cases are working correctly.

QnA and LUIS recognizers deprecation

The QnA and LUIS recognizers are deprecated and have been replaced by the OpenAIEmbeddings and CLU recognizers respectively in all trainings.

NLP Resources

The repositories resources listed below, which are the ones in production environment, have been migrated:

AP
- resources_pizza
- resources_ap-demo
- resources_cr
ES
- resources_tiempo
- resources_real-academia-historia
- resources_core-dialogs
- resources_bingo
- resources_movistar-shop
- resources_movistar-gaming
- resources_es-nov
- resources_memory
- resources_leia
- resources_eset
- resources_chester
- resources_mis-tokens
- resources_movistar-cloud
- resources_estadio-infinito
- resources_es
BR
- resources_br-b2c
- resources_br-b2b
- resources_br
- resources_br-rh
- resources_br-rcs
- resources_br-stb
- resources_br-easy
- resources_br-app
- resources_br-wa
- resources_br-nov

Migration process for CLF plugins

The Aura Global NLP team and constructors have carried out the following steps for the migration of CLF plugins:

Aura Global NLP team:
- Updated in requirements.txt the packages to the versions generated for Prince release.
- Updated tools and scripts to work with the Prince release.
- These repositories work the same way as the NLP training repositories: a release/7.0 branch has been launched before the Prince release to make changes in previous versions. For Prince release, a release/8.0 branch is available.
- Updated JenkinsFile:
  - The JenkinsFile is set for versions previous to Prince release (release/7.0).
  - In Prince release, there is no need to update JenkinsFile as it is already configured in master.
Constructors:
- Reviewed the Python changelog for Python 3.13 version.
- Executed tests with the new version of Python and its libraries.
- Executed plugins and tested the locally.
- Adapted code.
- Generated a new version of the plugin package of the constructors and propagated to the corresponding environment.
- Tested plugins in the environments.

Currently, there are no constructors outside the global team. For this reason, the Aura Global NLP team will be responsible for the migration of the CLF Plugins.

CLF plugins repository list

Below, the list of migrated CLF plugins repositories is included:

aura_clf_video

Migration process for ATRIA

The ATRIA repositories have been migrated to Python 3.13 by Aura Global Team.

The configuration inside ATRIA repositories must be the same as before migration, but constructors must test the use cases to ensure that everything is working correctly.

Information about the migration process for any productive repositories of Python language

Get sure that any productive repository that is not currently being used has been migrated from Python 3.9 to 3.13 version before being put into production.

Python 3.13 is the latest version of Python Programming Language at this moment. Cognitive components are being migrated to this version. ↩︎
JenkinsFile defines the pipeline in Jenkins for the continuous integration of the repositories. ↩︎

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Abacus 1.0.0. initial set-up

Using Abacus for the first time? Follow these guidelines for its set-up.

⚠️ This process must be done only once, the firt time you open Abacus.

Prerequisites for using Abacus

Check that you fulfil these requisites before Abacus set-up.

You have the NLP Virtual Machine installed and working, with your aura-nlpdata working branch generated.
The understanding model is already defined and configured.
You have generated the specific training files and E2E test files required for each stage of your NLP pipeline.
You have already filled the variables required for training in the configuration file build_local_variables.sh.

Steps for Abacus set-up

Access the NLP Virtual Machine.
Enter your local working project, henceforth {Root_project}, whose name should be:
aura-nlpdata-[country-code]
Copy the local variables configuration template: "${Root_project}/tools/build_local_variables.sh.tpl"
and rename as:
"${Root_project}/tools/build_local_variables.sh"
```
cd ${Root_project}/tools
cp build_local_variables.sh.tpl build_local_variables.sh 
```

Access the file build_local_variables.sh in the tools/ folder.
Fill in the variables inside the “run_web_trainings” section:
- TRAINING_WEB_AZURE_BASE_URL and TRAINING_WEB_AZURE_SAS_TOKEN variables:
  Ask the APE Team for them.
- GITHUB_TOKEN variable:
  - Generate a GitHub token:
    - Enter GitHub with your profile
    - Verify your email address if you haven’t done it yet
    - Go to GitHub token settings and generate a new token
    - Fill in the name and expiration date of the token
    - Select the scopes (Recommended: activate repo)
    - Click Generate token
  - Copy the token in the variable GITHUB_TOKEN.
    ⚠️ Make sure you copy it, as it will only shown once.

Run the script file:

${Root_project}/tools/run_web_trainings.sh

When the script is finished, Abacus is opened automatically.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Technical resources for working with Aura NLP

Mandatory resources required by NLP experts of linguists in order to work with Aura NLP

Resources list

🔹 Aura NLP technical resources 🔹
Aura installation - Latest Aura Platform release
Operating systems - Linux over distribution Ubuntu 18.04 LTS (with Java preinstalled)
Configuration of development environment - Python 3.9. - Pip3 - virtualenv - For Linux distributions: libsqlite3-dev liblzma-dev libbz2-dev
Software - GitHub licence - Text editor: Pycharm or similar - Use of Grammars: Unitex/Gramlab open-source corpus processing suite - Grammars engine: GrapeNLP - CLU stage: Microsoft CLU account - OpenAI stage: Azure OpenAI Service account
NLP Training and testing tool Abacus 1.0.0.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP Virtual Machine user guide

Guidelines for the installation of Aura NLP Virtual Machine, required for the configuration of the NLP development environment

Installation and operation guidelines

Aura Platform Team has generated a Virtual Machine that allows the configuration of the Aura NLP development environment following a semi-automatic process.

Follow the steps explained below to install and operate it:

Install VirtualBox (or other software).
Import OVA formatted files that contain all the technical requirements for the installation.
Once the VirtualBox is installed, follow these steps:

3.1. Select “File” -> “Import Appliance”

3.2. Access Aura NLP Virtual Machine for your release in the Sharepoint L-CDO repository and select the corresponding aura-nlpdata.ova file for importing it.

3.3. Click on “Next >” button and, afterwards, select the “Import” button.

3.4. When the importation process has finished, in the main view of VirtualBox the new Virtual Machine is shown. Click over the new virtual machine and then press “Start”.
Ask the APE Team for the user and password to access.
Generate a new SSH key to operate the Virtual Machine
In order to operate with the NLP Virtual Machine, it is required to generate a new SSH (Secure Shell Protocol) key. For this purpose, follow the steps described in the section Generate a new SSH key and add it to the ssh-agent.

Be aware of the time update, as if the Virtual Machine is paused, the time is not updated and it can lead to errors with certificates.

Generate a new SSH key and add it to the ssh-agent

In order to operate with the NLP Virtual Machine, certain configuration is needed to allow users to access GitHub NLP repositories.

You can access and write data in repositories on GitHub using SSH (Secure Shell Protocol). When you connect via SSH, you authenticate using a private key file on your local virtual machine.

SSH configuration guidelines

Follow the steps explained below to configure the SSH keys:

Open VirtualBox and run the NLP Virtual Machine by clicking in the “Start” button.
After login into the VM (if you do not remember the password, ask APE team), open the terminal.
In the terminal, type in the command: ssh-keygen -t ed25519 -b 4096 -C your-email@telefonica.com, replacing your-email by your corporate e-mail. This command generates a new pair of SSH keys using the ed25519 algorithm. After introducing the command, press “enter” as shown in the following image.
Now that we have our SSH key pair generated, introduce the following command: ssh-add ~/.ssh/id_ed25519. This command adds the private key id_ed25519 to the SSH agent, allowing SSH connections to use that key without requiring the password each time we need to access the repositories.
The next step is to display the content of the id_ed25519.pub file, which contains the public SSH key. To do this, type in the following command: cat ~/.ssh/id_ed25519.pub. We need to copy the displayed key to add it to GitHub in the following steps. Copy everything except your email address, displayed at the end of the string.
Open GitHub in a web browser and log in.
In the top-right corner, click on your profile image, and select “Settings”.
In the new page, on the left-hand side bar, click on “SSH and GPG keys”, and then click on “New SSH key” button (in the top right corner).
Now, add a title for the SSH key, for example, “VM NLP”. The key type should be Authentication Key, and in the key field, add the key copied from the terminal in step 5. Finally, click on “Add SSH key” button.
In the list of SSH keys shown after clicking “Add SSH Key” in the previous step, identify the recently created SSH key and, on the right-hand side, click on the “authorize” button, as shown in the following image. GitHub will prompt you to verify with your password in order to authorize the Telefónica organization.
Now that we have our SSH keys set up, we can proceed with cloning the repositories we need to work with. Remember that to clone the repositories from the VM, we must click on “SSH” to clone via SSH and copy the URL.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

How to obtain Azure credentials for CLU

This process may be required in the first step for training the understanding model: Set up configuration properties.

Prerequisites

Pre-requisites:

Azure account with permissions for applications registration.
Azure CLI installed.

Guidelines

Follow the guidelines below for obtaining the Azure credentials for CLU:

Create the CLU application:
az cognitiveservices account create --kind "TextAnalytics" --name <clu_name> -g <name_resource_group> --sku S -l <location> --custom-domain <clu_name>
- <clu_name>: resource name
- <name_resource_group>: name of resource group (previously generated)
- <location>: location available for Azure (i.e., northeurope)

The value for the parameters required to fill in the build_local_variables.sh script for CLU execution must be obtained from the above-defined steps:

export CLU_USER="<user_name>"
export CLU_RESOURCE_NAME="<clu_name>"
export CLU_SUBSCRIPTION_KEYS="$(az cognitiveservices account keys list --name <clu_name> -g <name_resource_group> | jq -r .key1)"

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP basic concepts and components

Basic concepts related to Aura NLP, components in Aura NLP architecture, catalogs and dictionaries

Utterance

An utterance is any textual input produced by the user that Aura receives through a specific communication channel and needs to understand, interpret and act accordingly. It may be a whole sentence, a phrase, or a single word. There can be many utterance variations for a particular intent.

Examples of utterances:

“I want to watch Frozen”; “Play the movie Frozen”; “Frozen”; “Aura, search Frozen”
“Show my bill”; “I want my bill”; “Bill”; “Hello, check my bill”

Use case

A use case is a written description of a certain functionality in Aura that is launched both by direct request from a user or through the data analysis of the user’s behavior.

A certain use case can be expressed by the user in a large variety of utterances through natural language, therefore Aura is intended to understand all those possible requests.

Examples of use cases:

TV search
Check my bill
Change subtitles to English

Intent

The intent identifies the concrete action requested by the user, among a set of supported services. In other words, it is what the user is asking for and expects Aura to carry out and is usually defined with a verb.

The general format of an intent name in Aura is: intent.[DOMAIN].[INTENT]
In this format, [DOMAIN] is used for the categorization of use cases belonging to the same topic (i.e., for Telco use cases, different domains can be defined such as billing, data usage, etc.).

Example of intents:

Pay my bill –> intent.billing.pay
Make a phone call –> intent.communications.call
Turn off the TV –> intent.tv.off

When developing a use case, linguists must firstly define the intent associated to their use case:

Check if the intent already exists in Existing_Intents_n_Entities database and, in that case, use it.
If not existing, define a new one following the format above. At this stage, it is highly recommended that the intent name is reviewed by Aura Platform Team, in order to avoid further inconsistencies.

An example of an intent and certain associated training statements is shown below:

"intent.billing.check": [
      "Bill",
      "Billing information",
      "Can I see my bill?",
      "Check my bill",
      "How do I access my bill?",
      "Invoice details",
      "Show me my last invoice"]

Entity

An entity contains detailed information that is relevant in an utterance and provides specific arguments required to run the service (intent).

Depending on the NLP pipeline stages, entities can be expressed in different formats:

If CLU stage is used to extract entities, the general format for an entity name is: [entity_value:entity_type]
If CLU does not extract entities and an external entity extractor stage is used before CLU (Standard NER, Gazetteer NER or Grammars), entities are defined as: [entity_type]
In both cases, the general format for the entity_type is ent.[object]

Example of entities:

Pay my bill –> bill
Make a call –> phone call
Turn off the TV –> TV

When developing a use case, linguists must define the entities associated to their use case:

Check if the entity already exists and, in that case, use it.
If not existing, define new ones following the format above. At this stage, it is highly recommended that the entity name is reviewed by Aura Platform Team, in order to avoid further inconsistencies.

A special situation, when the user’s utterance is recognized by means of the Grammar stage and there are different entities of the same type in the utterance, the format for entities is described in Grammars documentation: Recognition of utterances with several entities in Grammars.

Entity types for CLU

Conversational Language Understanding (CLU) uses the following entity types.

They are fully described in Microsoft documentation: CLU entity components and summarized below:

learned: Dict field to include entities of learned component type. This is actually not an entity type, but a feature. Therefore, they are uploaded as model_features and, at the same time, as simple entities (using the same name for both). An example of are those words referring to an audiovisual genre, such as movie, series or documentary.
list: Dict field to include entities of list component type. Fixed, closed set of related words along with their synonyms.
prebuilts: Dict field to include entities of prebuilt component type.
regex: Dict field to include entities of regex component type.
combination: Field for the combination of components as one entity when they overlap.

Aura NLP basic components: stages, connectors and pipelines

Stages

Aura NLP provides a set of stages that encompasses different methods for natural language processing.

Each stage carries out a specific process to be executed with the user’s utterance with the final goal of recognizing the user’s intent and associated entities.

You can use different stages in an NLP model, both internal stages developed by Aura Cognitive Team or external ones such as Microsoft CLU for intents recognition, NER (Named Entity Recognition) stages for entities recognition, Grammars engine (GrapeNLP & Unitex, that provide a deterministic recognition of intents, etc.

📃 Check the current available NLP stages.

Connectors

Connectors are components that connect different NLP stages and control the flow of the complete pipeline.

Aura NLP integrates several types of connectors, that provide a different behavior to the pipeline: logical connectors, selection connectors and disambiguation connectors.

📃 Check the current available NLP connectors.

NLP pipeline

NLP stages and connectors are integrated into a key component of Aura NLP: the pipeline. An Aura NLP pipeline is a set of wired stages composing a big topology that defines the processing to be performed during the natural language recognition phase.

In the current release, Aura NLP includes:

Normalization pipelines: Pipelines composed of different stages used for the normalization of the user’s utterance.
Dynamic pipeline: Pipeline designed using different stages and connectors. The pipeline must be defined for each channel and included in a file named pipeline.json.

Catalogs and dictionaries

NLP dictionaries

The recognition of entities in the Aura NLP model is based on dictionaries: knowledge bases of entities that are included in the NLP model as part of stages for entities recognition.

They are automatically generated from catalogs through the execution of a script.

Access to detailed information regarding what are Aura NLP dictionaries, types and guidelines for the generation or update of dictionaries in generation of Aura NLP dictionaries.

Catalogs

Catalogs are the source for entities to be included in an NLP model. Entities in catalogs are the input for the script that generates the NLP dictionaries.

There are two types of data in catalogs, at least one of them is required: manual catalogs and automatic catalogs that fetch data from Kernel URM database.

Access to detailed information regarding what are entities catalogs, types and guidelines for the generation or update of catalogs in generation of Aura NLP catalogs.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Stages in use cases development over Aura NLP

Guidelines that describe the orderly steps required for the development of a use case over Aura NLP, with the objective of making Aura understand the users’ utterances.

Introduction

These steps correspond to 3 main overall stages:

Build the understanding model and train it, that is, teach Aura to understand
Test the model through an ongoing and cyclical process until the accuracy in terms of intents and entities recognition is good enough
Certify the model and publish it

Prerequisites

Firstly, check that all the prerequisites are fulfilled:

Technical resources are available
Aura NLP Virtual Machine is installed and working
NLP data repository local branch is generated

1. Build up the NLP dynamic pipeline

For the development of a new use case, you must design a dynamic pipeline (pipeline.json file) through the most appropriate combination of stages and connectors for the recognition of intents and entities in the use case.

For this purpose, follow the guidelines in the succeeding sections.

1.1. Select the elements composing your NLP pipeline

Select the elements composing the pipeline (stages, connectors, normalization pipelines) depending on the recognition process required for the use case and its associated channel, and combine them for the design of the NLP pipeline.

Catalog of components for NLP pipelines

1.2. Generate the pipeline.json file

The base file for the dynamic pipeline is pipeline.json, that must be generated in the following path from the NLP repository:
aura-nlpdata-[country_code]/data/[language]/[channel]/pipeline.json

Edit this file including the required fields from all your selected stages and connectors and indicating the hierarchy between them:

name: Unique string that identifies the pipeline.
initial_node_id: Key of the element where the pipeline starts. It must appear as the first one also in the fields elements and links.
elements: Include in this field each element composing the pipeline (stages and connectors) and characterize them with two attributes:
- type: Two feasible values:
  - stage: pipeline stage.
  - joint: connector between pipeline stages.
- classpath: Class path of the specific element, that is, Python class reference from the root directory that must be included in order to use this stage.
  To obtain the classpath of each element of your pipeline:
  - Access to the NLP catalog of components
  - Look for your specific stage, connector or normalization pipeline
  - Copy the classpath in the corresponding Path section. Take into account:
  - The name of the element is free, but it should be auto-descriptive of its content.
  - The first element must be the one specified in initial_node_id.
  - The elements must be ordered: after a parent, its children must be included.
args: This field is only required for the configuration of three NLP components: Length Adapter; Domain Selector connector; Disambiguation connector
links: This field includes the hierarchy of the pipeline and connections between its elements.
- Each link item contains the connectors (as keys) and their children are the stages or other connectors they deal with.
- Each key in links must be of joint type.

See below a practical examples of the pipeline.json file:

Example 1. Garua pipeline.json file

The pipeline hierarchy can be seen in the boxes that contain other elements (PygrapeCanlaonPipeline and AcotangoNothresholdPipeline). Diamond boxes represent joint stages in the pipeline.

{
  "name": "Garua",
  "initial_node_id": "GaruaPipeline",
  "elements": {
    "GaruaPipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
    "FromConfigNormalizerWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.normalizer_wrapper.from_config_normalizer_wrapper.FromConfigNormalizerWrapper"
    },
    "PygrapeGrammarWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.grammar_wrapper.pygrape_grammar_wrapper.PygrapeGrammarWrapper"
    },
    "AcotangoNoThresholdPipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
    "StandardNerWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.ner_wrapper.standard_ner_wrapper.StandardNerWrapper"
    },
    "FullEntityORDCCLUPipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.joint.conditionals.OrPipeline"
    },
    "FullEntityRecognizerWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.recognizer_wrapper.full_entity_recognizer_wrapper.FullEntityRecognizerWrapper"
    },
    "DCCLUPipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
    "DomainClassifierWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.domain_classifier_wrapper.domain_classifier_wrapper.DomainClassifierWrapper"
    },
    "CluRecognizerWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.recognizer_wrapper.clu_recognizer_wrapper.CluRecognizerWrapper"
    },
    "EntityTaggerAdapterWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.entity_tagger_adapter_wrapper.EntityTaggerAdapterWrapper"
    },
    "StandardThresholdWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.standard_threshold_wrapper.StandardThresholdWrapper"
    },
    "NoneHandlerWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.none_handler_wrapper.NoneHandlerWrapper"
    },
    "ExactMatchOrRestStages": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.joint.conditionals.OrPipeline"
    },
    "ExactMatchRecognizerWrapper": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.recognizer_wrapper.exact_match_recognizer_wrapper.ExactMatchRecognizerWrapper"
    }
  },
  "links": {
    "GaruaPipeline": [
      "FromConfigNormalizerWrapper",
      "ExactMatchOrRestStages",
      "EntityTaggerAdapterWrapper",
      "StandardThresholdWrapper",
      "NoneHandlerWrapper"
    ],
    "ExactMatchOrRestStages": [
      "ExactMatchRecognizerWrapper",
      "PygrapeGrammarWrapper",
      "AcotangoNoThresholdPipeline"
    ],
    "AcotangoNoThresholdPipeline": [
      "StandardNerWrapper",
      "FullEntityORDCCLUPipeline"
    ],
    "FullEntityORDCCLUPipeline": [
      "FullEntityRecognizerWrapper",
      "DCCLUPipeline"
    ],
    "DCCLUPipeline": [
      "DomainClassifierWrapper",
      "CluRecognizerWrapper"
    ]
  }
}

Example 2. Configuration for the stage Length Adapter

See below an example of how to integrate this stage in a pipeline:

{
  "name": "Example",
  "initial_node_id": "ExamplePipeline",
  "elements": {
    "ExamplePipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
    "LengthAdapterThreshold": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.length_adapter_wrapper.LengthAdapterWrapper",
      "args": {
         "max": 50,
         "min": 1,
         "intent_template": "intent.example"
      }
    }
  },
  "links": {
    "ExamplePipeline": [
      "LengthAdapterThreshold"
    ]
  }
}

The following snippet shows how to configure more than one stage of the Length Adapter to return different intents for max or min length characters.

  "name": "Example",
  "initial_node_id": "ExamplePipeline",
  "elements": {
    "ExamplePipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
    "LengthAdapterMaxThreshold": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.length_adapter_wrapper.LengthAdapterWrapper",
      "args": {
        "max": 50,
        "intent_template": "intent.max.example"
      }
    },
    "LengthAdapterMinThreshold": {
      "type": "stage",
      "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.length_adapter_wrapper.LengthAdapterWrapper",
      "args": {
        "length_threshold_map": {
          "min": 1,
          "intent_template": "intent.min.example"
        }
      }
    }
  },
  "links": {
    "ExamplePipeline": [
      "LengthAdapterMaxThreshold",
      "LengthAdapterMinThreshold"
    ]
  }
}

1.3. Validate your pipeline.json file

At this stage, it is recommended to validate the generated pipeline.json file in order to assure that it is consistent and that all stages and joint operations are correctly related.

For this purpose, the following verifications are recommended:

Each item of links includes dicts, where the key is a name and the values are lists of class names.
Each item of elements has type and classpath.
initial_node_id is a key in links.
Each key in links is a joint stage (by having type equals to joint in elements).
Each class belonging to the values of a links item is present in elements.

Different examples of invalid pipeline.json files

Invalid pipeline as WrongPipeline key does not have a list of class names in links section:

{
   "name": "WrongPipelineExample",
   "initial_node_id": "WrongPipeline",
   "elements": {
     ...
   },
   "links": {
     "WrongPipeline":"DCCLUPipeline",
     "DCCLUPipeline": [
       "DomainClassifierWrapper",
       "CluRecognizerWrapper"
     ]
   }
 }

Pipeline contains an element without type:

{
  "name": "WrongPipelineExample",
  "initial_node_id": "WrongPipeline",
  "elements": {
    "WrongPipeline": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
   "DCCLUPipeline": {
      "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
    },
...

Pipeline MissingPipeline is not included as a key in the field links:

{
   "name": "WrongPipelineExample",
   "initial_node_id": "MissingPipeline",
   "elements": {
     ...
   },
   "links": {
     "WrongPipeline": [
      "DCCLUPipeline",      
      "EntityTaggerAdapterWrapper",
       "StandardThresholdWrapper",
       "NoneHandlerWrapper"
     ],
     "DCCLUPipeline": [
       "DomainClassifierWrapper",
       "CluRecognizerWrapper"
     ]
   }
 }

Pipeline with a stage (DomainClassifierWrapper) as key in links and not a joint:

{
   "name": "WrongPipelineExample",
   "initial_node_id": "WrongPipeline",
   "elements": {
     "WrongPipeline": {
       "type": "joint",
       "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
     },
    "DCCLUPipeline": {
       "type": "joint",
       "classpath": "auracog_pipelines.pipelines.base.BasePipeline"
     },
    "DomainClassifierWrapper": {
       "type": "stage",
       "classpath": "auracog_pipelines.stage_wrappers.domain_classifier_wrapper.domain_classifier_wrapper.DomainClassifierWrapper"
     },
     "CluRecognizerWrapper": {
       "type": "stage",
       "classpath": "auracog_pipelines.stage_wrappers.recognizer_wrapper.clu_recognizer_wrapper.CluRecognizerWrapper"
     },
     "EntityTaggerAdapterWrapper": {
       "type": "stage",
       "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.entity_tagger_adapter_wrapper.EntityTaggerAdapterWrapper"
     },
     "StandardThresholdWrapper": {
       "type": "stage",
       "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.standard_threshold_wrapper.StandardThresholdWrapper"
     },
     "NoneHandlerWrapper": {
       "type": "stage",
       "classpath": "auracog_pipelines.stage_wrappers.adapter_wrapper.none_handler_wrapper.NoneHandlerWrapper"
     }
   },
   "links": {
     "WrongPipeline": [
      "DCCLUPipeline",      
      "EntityTaggerAdapterWrapper",
       "StandardThresholdWrapper",
       "NoneHandlerWrapper"
     ],
     "DCCLUPipeline": [
       "DomainClassifierWrapper",
       "CluRecognizerWrapper"
     ],
    "DomainClassifierWrapper": [
       "CluRecognizerWrapper"
     ] 
  }
 }

2. Configure your NLP model

It is required to configure every element composing the NLP pipeline:

All NLP stages (excepting Length Adapter) and normalization pipelines are configured in the file nlp.json file for each language and channel placed in the [NLP repository]:
aura-nlpdata-[country_code]/config/etc/nlp_config/nlp.json
As an exception, Length Adapter stage, Domain Selector connector and Disambiguation connector need a specific configuration in the file pipeline.json(args field) placed in the [NLP repository]:
aura-nlpdata-[country_code]/data/[language]/[channel]/pipeline.json

To obtain the configuration of each element of your pipeline:
- Access to the NLP catalog of components
- Look for your specific stage, connector or normalization pipeline
- Copy the classpath in the corresponding Configuration section.
If dictionaries are included in the NLP model, they must be also configured:
- Configuration of dictionaries.

3. Define your data resources

Every NLP stage needs particular resources for its training and testing that must be generated through the edition of a specific file for each of them.

3.1. Generate the files for each NLP stage

Generate the specific file for each stage composing your NLP pipeline and for each language and channel.
- Access to the NLP catalog of components
- Look for your specific stage
- Find in the Files section the specific files required for this stage
- Edit them
Place these files in:
aura-nlpdata-[country_code]/data/[language]/[channel], where:
- language corresponds to the culture code of all the languages supported by Aura (e.g., es-es, en-gb, de-de, pt-br).
- The channel variable in the pattern is the channel code used to identify the specific channel (for example, mh (Movistar Home), mp (Movistar Plus)).

3.2. Generate dictionaries

If your NLP pipeline contains entities recognition stages (Entity Tagger Adapter; Standard NER and Gazetteer NER), it is needed to use the dictionaries sdict_items.json and sdict_aliases.json which are automatically generated from two sources: manual catalogs and/or URM data.

Learn how to do it in generation of Aura NLP dictionaries.

4. Define your end-to-end tests

E2E test files perform the evaluation of accuracy in the recognition of domains, intents and entities, with two approaches:

Measurement of the overall accuracy of the pipeline (mandatory)
Measurement of the accuracy of the different stages of the pipeline (optional)

Once generated, when running the corresponding pipeline with the user’s utterance as the pipeline input, the system will compare the result provided by the pipeline with the expected values declared in the file (intent, entities and domain) to calculate the pipeline accuracy.

4.1. Define E2E test set files

You must define a file for the end-to-end evaluation of the system as well as regression tests: testset.json and regression.json. They both are dictionaries with the same structure:

phrase: Statement (sentence, phrase or isolated word) to be tested.
domain: Inferred domain for the user’s utterance. Possible values:
- <domain_name>: name of the identified domain.
- null: when domain is not of application.
- default: in case there is only one domain or in case the grammar engine recognizes the whole utterance.
  ⚠️ The value for the field domain must be included using quotation marks for every value excepting for null.
intent: Expected intent.
options: List of .json containing certain packages of intents and entities to disambiguate (optional field).
entities: Expected entities. It is a list of json with the following fields:
- value: entity value to be recognized.
- e_type: entity type expected.
- start_index: Initial position of the entity in the filled phrase.
- end_index: final position of the entity in the filled phrase.
- canon: expected canon for a given entity. If canon is deactivated or the entity recognizer does not work with canon, this field must be completed with the same value of the field value but normalized (e.g., “Film”, value: Film; canon: film).
  - This field is currently used in Spain for those use cases related to TV content searches (e.g., “search for films”) in which there are specific codes (labels) for searching particular content types that the API needs to find in the corresponding catalogs to resolve the petition.
  - For instance, in the utterance “I want to watch an action movie”, “movie” is the value of the entity whereas “movies” may be its canon and “MV” the label the API needs to find this type of content in the catalogs. The same could be applied to the genre “action”.
- label: expected label. It can have the value null. The same use as canon is currently applied.

4.1.1. testset.json

At this stage, you should define the testset.json file in: pipeline_eval/ob/[country_code]/resources/[language]/[channel]/testset.json

It must include the testing statements for the E2E evaluation of the system’s accuracy (sentences or isolated words) and in order to identify potential problems (e.g., unmatching, low confidence/score).

You can generate different testset.json files for different purposes, for instance, for evaluation of metrics or to carry out regression tests at a later stage. To calculate the metrics, all the different files are considered as a unique one.

Example of testset.json file:

[
  {
    "phrase": "put the film Coco",
    "domain": "default",
    "intent": "intent.tv.search",
    "entities": [
      {
        "value": "Coco",
        "e_type": "ent.audiovisual_film_title",
        "start_index": 13,
        "end_index": 17,
        "canon": "coco",
        "label": null
      }
    ],
   "options": []
  },
  {
    "phrase": "show me my bill",
    "domain": "default",
    "intent": "intent.billing.check",
    "entities": [
      {
        "value": "bill",
        "e_type": "ent.bill",
        "start_index": 12,
        "end_index": 15,
        "canon": "bill",
        "label": null
      }
    ],
    "options": []
  }
]

4.1.2. regression.json

Define your regression.json file in the path: pipeline_eval/ob/[country_code]/resources/[language]/[channel]/regression.json

It may include crucial functionalities that must work in the system or other key checks that are not included in testset.json. The purpose is to verify that modifications do not impact in existing features and to prevent the system from bugs.

Previously executed test cases are re-executed in order to verify the impact of a change.

4.2. Define stage-specific E2E test set files

ℹ️ This is an optional step if you want to include specific E2E tests for the evaluation of an isolated stage in the testing batch.

You can create specific E2E testsets files for the evaluation of an isolated stage. It is done adding phrases that must be solved by this specific stage in order to ensure that the end-to-end evaluation is representative for that stage and avoid tests that do not evaluate it.

Currently, this is only available for the OpenAI embeddings recognizer stage.

For the definition of specific E2E tests for this stage, follow these instructions:

Define specific phrases to be resolved by the OpenAI embeddings recognizer stage.
Execute the script build_local_testset.sh in:
aura-nlpdata-[country_code]/tools/build_local_testset.sh
Once executed, it creates a stage-specific testset.json file in the path: tmp_testsets/[country_code]/resources/[language]/[channel]/
Although the name of the file can never be modified, it is possible to modify its content, as long as its structure is respected, adding new test sentences or eliminating them.
To be able to use these E2E test, copy it in the following path for it to be packaged with the general testset.json file: pipeline_eval/ob/[country_code]/resources/[language]/[channel]/

4.3. Best practices for the definition of E2E test set files

All intents should be represented within all the existing test testset.json files.
Firstly, generate a battery of statements for the use case, taking into account its semantic complexity. After that, divide all the generated statements into three groups in the way that statements in the training set are not included in the test sets and vice versa:
- Training set
- Specific NLP stage test set
- E2E test set
Follow this pre-established ratio between training and testing statements: each intent must satisfy that the number of test statements is, at least, 20% of the total statements (training and test statements).
Depending on the specific NLP stages, the number of recommended testing statements must be representative. In general terms, and only as a guidance, the number of testing statements can be as follows:
- Only CLU: 20% of statements in CLU training
- CLU + Grammar: 20% of statements in CLU training
- Only Grammar: 3 statements
- More than 1 use case on an intent: 30 statements per use case.
The testing statements provided by the Product Team and/or UX Team must be included, as prototypical of a given use case.
The statements must include different variations (for example, with/without entities, etc.).
Keys of the testset.json file should be ordered from generic to specific ones:

        { 
            "phrase": "Search the film Frozen", 
            "domain": "domain.tv", 
            "intent": "intent.tv.search", 
            "entities": [],
            "options": []
        },

The end-to-end test set is specific for each of the potential channels, as some use cases can be implemented in certain channels but not in others.
The field options is optional and only included when disambiguation is considered.
In case that, due to non-satisfactory results during the evaluation process, a re-training is required, linguists should check that all the modifications are included in the E2E tests.
In case roles are defined in entities for their recognition through the Grammar stage, they do not affect to the E2E tests (See more information regarding roles in Grammars in recognition of utterances with several entities in Grammars.

5. Train your understanding model

Once all the resources for each stage of the pipeline have been generated, you have to launch the training process in order to compare the testing batch against the training model.

For this purpose, the aura-nlpdata-[country_code]/tools folder of the NLP repository includes bash scripts, described in the following sections.

It is important to mention that the NLP system can be locally trained in an intelligent way, meaning that only the stages that have been modified (from a last training) are trained again, thus making the process much more efficient.

5.1. Set up configuration properties

Go the the path: aura-nlpdata-[country_code]/tools/build_local_variables.sh.tpl
This file is a template used for configuration purposes, specifically for defining CLU connection parameters. To setup these properties, copy this file to a new one named build_local_variables.sh, removing the .tpl extension.
Fill in the config variables included in this file with the local credentials, as explained below.
This file is automatically ignored by git because it has been included in the .gitignore file, thus it must not be included manually.

The parameters to fill in the build_local_variables.sh script are shown below:

#!/usr/bin/env bash

# BUILD_LOCAL AND RUN_WEB_TRAININGS
export AZURE_NLP_MODELS_URL=""
export OAI_ID_SUBSCRIPTION=""
export OAI_RESOURCE_GROUP=""
export OAI_ACCOUNT_NAME=""
export OAI_AZURE_TOKEN_CLIENT_ID=""
export OAI_AZURE_TOKEN_CLIENT_SECRET=""
export OAI_AZURE_TOKEN_TENANT=""
export OAI_USER=""
export RESOURCE_NAME_OPENAI=""
export QDRANT_URL=""
export QDRANT_API_KEY=""
export CLU_SUBSCRIPTION_KEYS=""
export CLU_RESOURCE_NAME=""
export CLU_USER=""
export CLU_STORAGE_SUBSCRIPTION_KEYS=""
export CLU_STORAGE_RESOURCE_NAME=""

# RUN_WEB_TRAININGS
export GITHUB_TOKEN=""
export GITHUB_USER=""
export REPO_OWNER=""
export TRAINING_WEB_AZURE_BASE_URL=""
export TRAINING_WEB_AZURE_SAS_TOKEN=""

# BUILD_CATALOGS
export LANGUAGE=""
export CHANNEL_LIST=""
export AZURE_CATALOGS_ACCOUNT_NAME=""
export AZURE_CATALOGS_TOKEN=""

export AWS_CATALOGS_ACCESS_KEY=""
export AWS_CATALOGS_SECRET_KEY=""
export CATALOGS_RESOURCES_CONTAINER=""
export CATALOGS_RESOURCES_PROVIDER=""

The required variables are described below:

AZURE_NLP_MODELS_URL: URL for the Azure NLP models container.
GITHUB_TOKEN: Variable only required for ABACUS. Personal token provided by GitHub for secure authentication.
GITHUB_USER: Variable only required for ABACUS. Name of Github user.
REPO_OWNER: Variable only required for ABACUS. Name of the owner of the repository. Value: Telefonica.
TRAINING_WEB_AZURE_BASE_URL: Variable only required for ABACUS. URL base to get web package. It is provided by APE Team.
TRAINING_WEB_AZURE_SAS_TOKEN: Variable only required for ABACUS. SAS token with the required permission granted. It is provided by APE Team.
OAI_ID_SUBSCRIPTION: Azure OpenAI subscription ID. It can be obtained from the subscription website.
OAI_RESOURCE_GROUP: Name of resource group in Azure where the OpenAI applications are created.
OAI_ACCOUNT_NAME: Name of OpenAI resource to be used.
OAI_AZURE_TOKEN_CLIENT_ID: Client ID of Azure Portal – App registration page assigned to your app.
OAI_AZURE_TOKEN_CLIENT_SECRET: Application secret created in the app registration portal.
OAI_AZURE_TOKEN_TENANT: Value that indicates who can sign into the application.
OAI_USER: Parameter to identify the user of OpenAI application. It is unique for each developer in order not to overlap the OpenAI trainings. This value is used to create database collections.
RESOURCE_NAME_OPENAI: Name of resource to be used.
QDRANT_URL: URL of Qdrant service. In the virtual machine, it is http://localhost:6333.
QDRANT_API_KEY: APIkey of Qdrant service. In the virtual machine, it is void.
CLU_SUBSCRIPTION_KEYS: Parameter provided by CLU to create applications.
CLU_RESOURCE_NAME: Name of resource to be used.
CLU_USER: Parameter to identify the user of CLU application. It is unique for each developer in order not to overlap the CLU trainings.
CLU_STORAGE_RESOURCE_NAME: Name of shared resource to be used as library of applications.
CLU_STORAGE_SUBSCRIPTION_KEYS: Parameter provided by CLU to import and copy applications in CLU shared resources.

📄 For detailed information regarding how to obtain Azure credentials for CLU, please check the section Complementary processes.
📄 Information regarding how to get ABACUS variables in ABACUS documentation.
📄 If dictionaries are included in the NLP model, additional variables are required. check them in Set up specific configuration variables for dictionaries.

5.2. Execute the training script

From this point on, linguists or NLP experts have two options to continue with the process:

OPTION A	OPTION B
Use our web tool ABACUS 1.0.0. following the guidelines in ABACUS documentation. (*) After using ABACUS, continue with the process for the NLP model deployment in section Certify NLP model accuracy	Execute the training script, following the guidelines below

Execute the training script: aura-nlpdata-[country_code]/tools/build_local.sh
The script automatically creates a Python virtual environment to ensure the training and evaluation processes are being carried out in an isolated and encapsulated way.
All dependencies included in requirements.txt are installed in the virtual environment.
This script also validates the format of the involved files to ensure they match the specifications.
Once this script is executed, a tmp folder is created in the root repository. In this folder, you can find some temporary files corresponding to the resources, as well as the results and metrics obtained from the training process.
This directory is ignored by the git version control system because it has been included in the .gitignore file and it must not be included manually.

Intelligent training

The NLP system is trained in an intelligent way, so that the training of certain stages can be skipped, if they were previously trained, making the process more agile and efficient. This feature is based on the verification of an internal hash table and hash index generated after training.

The default behavior of the intelligent training is:

On one hand, only the stages that have been modified from a previous retraining in a specific channel are trained again. For this purpose, the system keeps a hash table in the tmp/ folder to detect changes.
On the other hand, if the configuration and the model generated to train a stage are the same as those of a previous stage but of a different channel, the last do not need to be trained and it will use the model trained before, making the process much more efficient. To achieve this, an internal hash index is generated in the tmp/trained_models folder. It is important that the training files in every channel are exactly the same, with similar name and similar content.

However, the hash table and hash index can be managed manually in order to modify this behavior:

Management of the hash index to force the training of a stage in a specific channel

The hash table is included in the tmp/ folder after training. This folder must not be deleted when tests are executed, unless all the stages are to be re-trained again. If you want to force the training of a specific stage, its corresponding file can be deleted in the specific channel. For instance, if there are no modifications on a stage within the mh channel, but you want to force its retraining, then go to the tmp/ folder and delete the file saved_training_hashes.json in the path:
tmp/recognizer/ob/ES/es-es/mh/resources/saved_training_hashes.json

Management of the hash index to force the training of a stage in different channels

The hash index identifies similar training files from stages of the same type that belong to different channels. It is included in the following folder: tmp/trained_models/[stage]/[hash]/

[stage]: name of a stage.
[hash]: it is resulting from the content of the training files used for that stage and its specific configuration.

Each sub-tree contains the necessary files that were used during the training phase for that specific stage.
By default, if the same stage with the same training files exists in different channels, only the first one found is retrained.
If you want to force the training for a specific stage, in addition to eliminating the hash tables seen in the previous section, delete the tmp/trained_models/[stage] for this stage.

5.3. Generation of results from the training process

When the training process is finished, certain temporary files are created in the tmp/ directory in the root repository.
This folder contains the resources of the NLP model and results and metrics from the training process obtained from launching the testing batch against the training model.

Files generated in the tmp/ directory are organized as shown in the following tables:

Input resources for the NLP model

The input resources for the NLP training are placed on:

Input resources	tmp/ folder
Training and test set files in data/	tmp/[stage]/ob/[country_code]/[language]/[channel]/
E2E test files	tmp/pipeline_eval/ob/[country_code]/resources/[language]/[channel]/
`bootstrap.cfg` config file	tmp/pipeline_eval/ob/[country_code]/etc/

Results from the NLP training

Results from the NLP training	tmp/ folder
Result files from each NLP stage training	tmp/results/[stages]/[country_code]/[language]/[channel]/
Result files from the overall pipeline training	tmp/results/pipeline_eval/[country_code]/
Result files from the overall pipeline training: regression tests	tmp/results/pipeline_eval/[country_code]/regression/

Testset files

If you have defined stage-specific E2E testset files, then after the execution of the script build_local_testset.sh, some temporary files are created in the tmp_testsets/ folder:

Testset files	tmp/ folder
Stage-specific E2E testset files	tmp_testsets/[platform]/resources/[language]/[channel]/

Each channel folder contains the end-to-end test files for each stage (currently, only for OpenAI embeddings recognizer). These files are used for the evaluation of the pipeline in future trainings and can be extended with as many tests as desired.

Intelligent training behavior

Intelligent training behavior	tmp/ folder
Hash table including the modified training files	tmp/recognizer/ob/ES/es-es/mh/resources/saved_training_hashes.json
Hash index including the modified training and test set files for a specific stage	tmp/trained_models/[stage]/[hash]/

6. Evaluate E2E accuracy locally

With all the results from the training process, saved in the tmp/results/ folders as explained before, now these results must be analyzed in order to evaluate if the NLP process is accurate enough for the recognition of intents and entities.

✅ If the local analysis of results is satisfactory at this stage, linguists can proceed to create the Pull Request.

⛔ If the analysis shows that the metrics are not good enough, meaning that the recognition is less accurate than required, then linguists must work again on the resources data to increase the performance and repeat the training process to re-calculate the metrics.

The analysis of results can be carried out from two different points of view, as explained in the following sections:

Focusing on each stage composing the pipeline
Or treating the pipeline as a single component to measure the end-to-end performance.

6.1. Evaluate NLP stages accuracy

For this purpose, analyze the following file, generated after training in the tmp/results/ folder:

tmp/results/[stages]/[country_code]/[language]/[channel]/test_results.txt

This file is generated per each pipeline stage, country, language and channel in the above-mentioned path and contains the metrics of the stage performance:

Precision: reflects false positives (false statements recognition)
Recall: reflects false negatives (missed items)
F1-score: combines precision and recall
Generation of average values

However, these metrics depend on the specific stages of the pipeline as, for example, the normalizer stage requires no evaluation and others such as Domain Classifier, NER or the intent recognizers can use all or some specific metrics among the four previously defined.

Moreover, depending on the stage, it is possible to find other files such as:

cv_results.txt that includes metrics regarding cross-validation
fitted-params.txt with information about the algorithm and parameters used to train the model.

Below, an example of test_results.txt file is shown, that corresponds to the Domain Classifier stage evaluation. The values for precision, recall, f1-score and support for each domain classified are calculated, as well as the total average.

	precision	recall	f1-score	support
None	0.40	0.84	0.54	32
intent.tv.search	0.97	0.89	0.93	122
intent.common.greetings	0.99	0.99	0.99	715
intent.billing.check	1.00	1.00	1.00	53
avg / total	0.84	0.93	0.86	922

6.2. Evaluate the overall pipeline accuracy

For the evaluation of the accuracy of the complete pipeline, you should analyze the files generated in the tmp/results/ folder after training:

tmp/results/pipeline_eval/[country_code]/
tmp/results/pipeline_eval/[country_code]/regression/

In both folders, the files are shown below, both generated from launching the testset files testset.json and regression.json:
- results.json
- details_[language]_[channel].csv
- test_results_by_intent_[language]_[channel].json
- test_results_by_intent_[language]_[channel].txt
- test_results_by_entity_[language]_[channel].json
- test_results_by_entity_[language]_[channel].txt

At this point, you are expected to analyse the results of the metrics included in these files in terms of accuracy and precision of intents and entities recognition. The files provide a detailed description about the testing statements that have obtained an unexpected result, as well as useful information for debugging purposes.

⚠️ If you use ABACUS, take into account that, currently, the tool only shows two test files: results.json and details_[language]_[channel].csv

results.json

General file that includes the results of the overall pipeline performance through statistics regarding the number of entries misclassified in the test set and their relative scores.

The metrics that contain this file are defined below:

Accuracy intent: Percentage of successful intents.
Accuracy overall: Percentage of successful inputs.
Accuracy perfect in options: Percentage of successful inputs included when the first option is right.
Entity error: Number of inputs in which entity recognition has failed.
Intent error: Number of inputs in which intent recognition has failed.
Option error: Number of inputs in which options recognition has failed.
Missing entities overall: Ratio of training statements (sentences, phrases or isolated words) in which entity recognition is failed to the total number of statements.
Missing entities right intent: Ratio of training statements in which entity recognition is failed but intent recognition is successful to the total number of statements with the intent recognized.
Missing options overall: Ratio of training statements in which options recognition is failed to the total number of statements.
Missing options right intent: Ratio of training statements in which options recognition is failed but intent recognition is successful to the total number of statements with option recognized.
Perfect: Total number of inputs without errors.
Perfect in options: Total number of inputs without errors where the first option is right.
Test size: Total number of inputs in test set file.

Additionally, each time a Pull Request (PR) is generated, a comment appears automatically in the results.json content in the GitHub repository to ease the reviewing task.

An example of the results.json file is included below:

{
    "date": "2021-08-30T09:27:03Z",
    "language": {
        "es-es": {
            "mp": {
                "accuracy_intent": 0.9681528662420382,
                "accuracy_overall": 0.9585987261146497,
                "accuracy_perfect_in_options": 0,
                "entity_error": 3,
                "intent_error": 10,
                "missing_entities_overall": 0.009554140127388535,
                "missing_entities_right_intent": 0.009868421052631578,
                "missing_options_overall": 0,
                "missing_options_right_intent": 0,
                "option_error": 0,
                "perfect": 301,
                "perfect_in_options": 0,
                "test_size": 314
           }
        }
    },
    "ob": "ES"
}

details_[language]_[channel].csv

One file is generated per each pair language/channel, containing the original training statement, the expected values versus the obtained values for intents, entities and domains after the pipeline execution as well as an additional column with a tag summarizing the error type, with five possible values:

D: error when recognizing the domain.
I: error when recognizing the intent.
E: error when recognizing the entity.
O: error when recognizing the options.
W: special tag used when result expected is the first option in recognized result.

This additional column is able to have more than one of these values. In detail, fields contained in the .csv file are:

phrase: Original statement (sentence, phrase, or isolated word) evaluated.
different: Summary of errors. It could have from one to three letters depending on the errors found. Suitable letters are D (domain), E (entities), and I (intent).
intent_obtained: Intent obtained by the pipeline.
intent_expected: Intent expected as defined in the test set.
entities_obtained: Entities obtained by the pipeline.
entities_expected: Entities expected as defined in the test set.
options_obtained: Options obtained by the pipeline.
options_expected: Options expected as defined in the test set.
domain_obtained: Domain obtained by the pipeline.
domain_expected: Domain expected as defined in the test set.

An example is shown below:

phrase	different	intent_obtained	intent_expected	entities_obtained	entities_expected	domain_obtained	domain_expected	options_obtained	options_expected

test_results by intent and by channel

They are both .txt and .json files containing the results of the pipeline performance per each pair language/channel and per intent, with the following format:

test_results_by_intent_[language]_[channel].json

The metrics that contain this file are defined below:

n: Number of successful statements.
total: Total number of statements by intent.
overall: Total accuracy by intent.
intent: Accuracy of intents by intent.
entities: Accuracy of entities by intent.
options: Accuracy of options by intent.
domain: Accuracy of domains by intent.
perfect_in_options: Number of successful statements recognized in the first option by intent.

Example of test_results_by_intent_[language]_[channel].json:

  "intent.common.greetings": {
    "n": 2,
    "total": 2,
    "overall": 1.0,
    "intent": 1.0,
    "entities": 1.0,
    "options": 1.0,
    "domain": 1.0,
    "perfect_in_options": 0.0
  }
}

test_results_by_intent_[language]_[channel].txt

Same fields as the JSON file but written in legible mode.

Example of test_results_by_intent_[language]_[channel].txt:

PIPELINE RESULTS: 
      intent.common.greetings: n:2, Total: 2, Accuracy (Overall): 1.000000, Accuracy (Intent): 1.000000, Accuracy (Entities): 1.000000, Accuracy (Options): 1.000000, Accuracy (Domain): 1.000000, Accuracy (Perfect in options): 0.000000
------------------------------

      Test Size: 4

test_results by entity and by channel

They are both .txt and .json files containing the results of the pipeline performance per each pair language/channel and per entity, with the following format:

test_results_by_entity_[language]_[channel].json

The metrics that contain this file are defined below:

n: Number of successful statements.
total: Total number of statements by entity.
overall: Total accuracy by entity.
intent: Accuracy of intents by entity.
entities: Accuracy of entities by entity.
options: Accuracy of options by entity.
domain: Accuracy of domains by entity.
perfect_in_options: Number of successful statements recognized in the first option by entity.

An example is shown below:

{ 
  "ent.audiovisual_film_title": { 
    "n": 4, 
    "total": 4, 
    "overall": 1.0, 
    "intent": 1.0, 
    "entities": 1.0, 
    "options": 1.0, 
    "domain": 1.0 
  } 
}

test_results_by_entity_[language]_[channel].txt

Same fields as the .json file but written in legible mode.

Example:

PIPELINE RESULTS BY ENTITIES: 
------------------------------
           ent.audiovisual_film_title: n:4, Total: 4, Accuracy (Overall): 1.000000, Accuracy (Intent): 1.000000, Accuracy (Entities): 1.000000, Accuracy (Options): 1.000000, Accuracy (Domain): 1.000000
------------------------------

      Total uniques Entities: 1

      Total Entities: 4

6.3. Analyze compatibility between global grammars and local grammars

⚠️ The current section only applies if both global and local grammars are implemented in the NLP recognition process.

As explained in Grammars management the two types of grammars defined in Aura NLP recognition process, global and local, must be aligned. For checking the compatibility between both grammars, you must generate two test set files:

data/[language]/[channel]/test_grammar/commons/testset.json
Test set with statements that must be recognized by both grammars (with identical results).
data/[language]/[channel]/test_grammar/disjoints/testset.json
Test set with statements that must be only recognized by the global grammar (as the local grammar is a subset of the global grammar).

Both tests are JSON files including a list of test phrases, as shown in the example:

[
  "push play again",
  "turn on the light"
]

These tests run through an automatic process and, if some error is detected, it is reported. In this scenario, linguists must check the errors and fix them:

Errors in disjoints testset

Local grammar recognizes a global phrase
This error occurs when a disjoint testset statement is recognized by the local grammar. An example of this error message for the language es-es, channel mh and the statement “push play again”:
```
Local grammar recognized a global phrase "push play again" for language es-es and channel mh
```
To resolve this problem, carry out the required modifications over the local grammar in order not to recognize the statement.
Global grammar does not recognize a test statement
This error occurs when a disjoint test set statement is not recognized by the global grammar.
```
Error recognizing phrase: " push play again " by pipeline grammar for language es-es and channel mh
```
To resolve this problem, carry out the required modifications over the global grammar in order to recognize the statement.

Errors in the commons testset

Local grammar does not recognize the statement but global grammar does. The program logs the following error message:
```
Error recognizing phrase: "turn on the light" by local grammar for language es-es and channel mh
```
In order to fix this error, improve the local grammar.
Global and local grammars recognize different intents The program logs the following error message:
```
Recognized phrase "turn on the" by both grammar with different intents.
Pipeline intent: intent.domotics.light_off, 
Local grammar intent: intent.domotics.light_on
```
In order to fix this error, improve both grammars to make them recognize the same intents.
Global grammar does not recognize the statement but local grammar does The program logs the following error message:
```
Error recognizing phrase: "turn on the light" by pipeline grammar for language es-es and channel mh
```
In order to fix this error, improve the global grammar.

6.4. Launch and test your pipeline locally (live mode)

Another useful functionality for a quick a real-time evaluation of the accuracy of the NLP model is running the pipeline in live mode in local environment.

To use this interactive execution approach:

Execute the script:
aura-nlpdata-[country_code]/tools/run_local_pipeline.sh
Once the script is run, select manually both channel and language.
After that, insert testing statements representing potential users’ utterances through the command line in a responsive way.
Evaluate the response in real time to the input statement: the associated intents, entities and score provided by the system.

⚠️ It is important to run this script after the build_local.sh (that is, after training the model) to ensure the system has been trained and all the resources have been generated.

This script neither generates temporary files nor directories and it can be run from the IDE or the OS terminal.

7. Pull Request to release branch

All the steps in previous sections are developed in a local branch, cloning the NLP master branch.

Once the NLP model is validated locally, now you must create a Pull Request (PR) to your release branch in order to upload your files and apply for validation to the NLP Global Team.

Follow the steps explained hereunder to create a Pull Request in the GitHub web application:

Verify current working branch and files to be included in the Pull Request: git status
If, when executing this command, there are files that should not be uploaded, remove them using git checkout and the path of the corresponding file that appears in status.
Add the local files: git add <file_name>

Use git add -A to upload all files in your local branch.
Use git rm <file1> <file2> <file3> if you need to remove certain modified files.

Commit changes with the command git commit -a "[[<feat>]] change description"
Execute the command git pull as an optional step to check if, during the execution of these commands, there are modifications in the same path that can produce further errors.
Push local branch: git push origin <branch_name>
Create a Pull Request to release branch: Access to the corresponding directory: aura-nlpdata-[country_code]/
And create a Pull Request from this branch to master or to the current release branch.
The title of the PR should start with [[feat]], [[fix]], or [[release]] and contain a representative description of the modifications.

Access our best practices for the creation of a Pull Request.

8. Certify NLP model accuracy

⚠️ REMEMBER… If you have used the tool ABACUS for the local training, testing and publication of your NLP model, now you must continue here with the process for its deployment.

When the Pull Request is launched, a validation process starts for the evaluation of the NLP recognition process: the so-named Continuous Integration (CI), defined as a process for the integration of code into a shared repository and its validation.
The validation comprises the execution of the training script build_local.sh by the NLP Global Team, that launches two processes:

An automatic validation process.
A manual review of results by the NLP Global Team.

Automatic generation of the NLP metrics

The system automatically generates certain metrics files for checking:

Accuracy of the whole pipeline
Accuracy of specific intents
Ratio of test set
Valid format of files
Modification without permission or by mistake of certain tasks
Compatibility between local and global grammars

These metrics will be included in the PR conversation using the E2E files testest.json and regression.json in order to provide a summary of the NLP system quality.

Review by the NLP Global Team

Complementary, the NLP Global Team carries out a review of results and report the existing problems.

The setting of an adequate threshold for the NLP system accuracy depends on the use case. Therefore, for a specific use case, the minimum accuracy should be agreed by L-CDO and the NLP Global Team.

After the Pull Request approval by Aura Global Team, the modifications are ready to be merged.

It can be very useful for Local Teams to know the process and criteria used by the NLP Global Team to validate the NLP model in order to focus on the critical points. Discover all this information in Validation process by the NLP Global Team.

9. Merge and generate your understanding package

At this stage, after the Pull Request approval, you are ready to merge the Pull Request in GitHub. Modifications are then included in the NLP release branch.

The system automatically initiates the process for the generation of the new version of the understanding package (artifact): a new Debian package with the version and name of the corresponding Platform release. This process can last a few hours.

When the new understanding package is generated, an e-mail is sent to PMOs, communicating that there is a new version available.

The APE Team is in charge of communicating the OB the name of the new package.

Now, the Local DevOps Team is responsible of the deployment of the understanding package.

10. Deploy the new understanding package

Once the previous stages are completed, the Local DevOps Team should deploy the NLP artifact with the new or updated trainings.

Remember that OBs are able to deploy NLP packages through a hot swapping process.

📄 For both processes, the local DevOps Team should check the document Aura Deployment of NLP packages.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Narugo normalization pipeline

Description and stages

Narugo is a pipeline used for the normalization of the user’s utterance through the execution of the following normalizers:

PunctuationNormalizer
SplitPunctNormalizer
SpaceNormalizer
CurrencyNormalizer
UnicodeNormalizer
LowercaseNormalizer
CardinalityNormalizer

Configuration

This stage requires the following configuration in the nlp.json configuration file:

{
  "es-es": {
    "mp": {
         "nlp": {
         "normalizer_pipeline_class": "auracog_pipelines.pipelines.normalization.narugo.NarugoPipeline"
      }
    }
  }
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Catalog of NLP connectors

NLP connectors to compose the NLP pipeline

Aura Platform Team has implemented different types connectors to join NLP stages in order to configure the pipeline.

Select your intended connector in the left menu. Each of them is characterized by its description, path, files and configuration.

Section	Content	Role in the NLP process
Description	Identification and objective of the stage in the recognition process	Descriptive purpose of the stage in the recognition process
Path	Class path (Python class) of an element (stage or connector)	The path of each stage of the pipeline must be included in the file `pipeline.json` for building up the NLP dynamic pipeline
File	Specific training files and test set files for the NLP stage required to train and validate the NLP model	Linguists must generate these files for the training and the validation of the NLP model during the data resources definition
Configuration	Required configuration for each NLP stage	Configuration of each stage of the NLP model

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP normalizers

What are Aura NLP normalizers

Text normalization is the process of transforming an Aura user’s utterance (expressed in natural language) into a standardized one to be more easily recognized by Aura NLP.

During the normalization process, certain characters are replaced/removed in order to reduce the input diversity that does not provide relevant information to Aura, such as replacing uppercase by lowercase letters, removal of punctuation marks, etc.

Within Aura NLP, there are different normalization stages which are handled as simple stages, taking part of a pipeline. Additionally, it is possible to define pipelines composed only by normalization stages suitable to be nested into another pipeline.

The following sections show the Aura NLP normalizers included in the current catalog.

Cardinality

The cardinality normalizer replaces ordinal or cardinal numbers expressed in text characters by digits. It cannot be used for percentages. For this purpose, the normalizer uses a fork of the library Microsoft.Recognizers.Text.

Example: “Put the second $” –> “Put the 2$”; “$Give me ten results” –> “$ Give me 10 results”.

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.cardinality_normalizer_wrapper.CardinalityNormalizerWrapper

Currency

This normalizer provides an appropriate format to the amount and currency in an utterance, separating the currency symbol from the amount with a single space. The implementation of this normalizer can be consulted in https://github.com/Telefonica/Recognizers-Text.

It is able to read the following currencies: $, € and £.

Example: “1$” –> “1 $”; “$1” –> “$ 1”; “1€” –> “1 €”.

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.currency_normalizer_wrapper.CurrencyNormalizerWrapper

FromConfig

FromConfig normalizer executes the normalization pipeline defined in configuration.

This normalizer does not require any file.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.from_config_normalizer_wrapper.FromConfigNormalizerWrapper

Configuration

This stage requires the following configuration in the nlp.json file.

  "es-es": {
    "mp": {
      "nlp": {
        "normalizer_pipeline_class": "auracog_pipelines.pipelines.normalization.nabro.NabroPipeline",
      }
  }
}

Where:

normalizer_pipeline_class: in this field, the specific normalization pipeline is referred (Nabro, Narugo, Nikko or Noro).

Lowercase

The lowercase normalizer replaces uppercase characters by lowercase ones in the utterance.

Example: “watch the NBA” –> “watch the nba”.

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.lowercase_normalizer_wrapper.LowercaseNormalizerWrapper

Punctuation

The punctuation normalizer removes special punctuation characters (¿?¡!,.;:) from the utterance, inserting blank spaces instead.

Example: “How are you? Fine, thanks!” –> “How are you Fine thanks”.

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.punctuation_normalizer_wrapper.PunctuationNormalizerWrapper

Space

The Space normalizer removes duplicated spaces from the utterance reducing space sequences to a single space.

Example: “How are you Fine” –> “How are you Fine”

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.space_normalizer_wrapper.SpaceNormalizerWrapper

Split Punct

Split Punct normalizer tokenizes the utterance splitting by words and punctuation marks using the NLTK framework. This framework uses NLTK recommended word tokenizer (currently an improved TreebankWordTokenizer that uses regular expressions to tokenize the text, together with PunktSentenceTokenizer that builds a model for abbreviations, collocations and words starting sentences.

The model is used to find sentence boundaries. The result is the utterance split by words separated by single spaces.

Example: “Please!!, get out now… right?” –> “Please ! ! , get out now … right ?”.

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.split_punct_normalizer_wrapper.SplitPunctNormalizerWrapper

Stop words

The stop words normalizer removes stop words, defined as commonly used words such as “the”, “is”, “at”, “which”, or “on” from the user’s utterance. This normalizer is able to recognize stop words from different languages using the NLTK framework.

Example: “its ok, I prefer the first or second option too” –> “ok prefer first second option”

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.stop_words_normalizer_wrapper.StopWordsNormalizerWrapper

Unicode

The Unicode normalizer replaces non-Unicode characters by other characters pre-defined in the Unicode characters list.

Example: “the 1º stop will be in München” –> “the 1. stop will be in munchen”.

This normalizer does not require any file or configuration.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.unicode_normalizer_wrapper.UnicodeNormalizerWrapper

Stop words from file

While the previous normalizer identifies predefined stop words from a database, the current normalizer allows the generation of a customized list of stop words, leading to a more accurate recognition of the user’s utterance.

The stop words from file normalizer requires the edition of the stop_words.json file to define a list of personalized stop words for each language and channel. This file must be placed at:
aura-nlpdata-[country_code]/data/[language]/[channel]/stop_words.json

The stop_words.json file performs the following tasks during the training process:
- Transforms each word to lowercase
- Removes repeated words

This normalized file is saved in a new file normalized_stop_words.json, in a temporary directory.

When Aura receives a request from the user, the behavior of the stop words from file normalizer is shown below for a specific example:

File content of stop_words.json: [“Its”, “its”, “I”, “,”, “the”, “or”, “which”]
Utterance: “its ok, I prefer the first or second option too” –> “ok prefer first second option too”

⚠️ When this normalizer is used, the words to be included in the stop_words.json file must be already normalized.

⚠️ The normalization does not validate if the defined “stop word” in the file is composed by only one word. Therefore, a “stop word” could be composed by more than one word.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.stop_words_from_file_normalizer_wrapper.StopWordsFromFileNormalizerWrapper

Word replacer from file

The Word replacer from file normalizer allows the exchange of words in the utterance.

The word replacer from file normalizer requires the edition of the word_replacer_mapper.json file to define a mapper containing the final words as a key and the list of words to replace as a value for each language and channel. This file must be placed at:
aura-nlpdata-[country_code]/data/[language]/[channel]/word_replacer_mapper.json

The word_replacer_mapper.json file performs the following tasks during the training process:
- Transforms each word to lowercase
- Removes repeated words in word values

This normalized file is saved in a new file normalized_word_replacer_mapper.json, in a temporary directory.

When Aura receives a request from the user, the behavior of the word replacer from file normalizer is shown below for a specific example:

File content of word_replacer_mapper.json:

{
  "hello": ["hi", "greetings", "howdy", "hey"],
  "option": [ "alternative"]
}

Utterance: “howdy, i want the second alternative” –> “hello, i want the second option”

⚠️ When this normalizer is used, the words to be included in the word_replacer_mapper.json file must be already normalized.

⚠️ All values should be only composed by one word. If a value contains more than one word, the normalizer raises an error in the training process. If multiple words are allowed, the normalization process is not idempotent.

Path

auracog_pipelines.stage_wrappers.normalizer_wrapper.word_replacer_from_file_normalizer_wrapper.WordReplacerFromFileNormalizerWrapper

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Recognition of utterances with several entities in Grammars

Specific guidelines in the scenario when the user’s utterance includes several entities to be recognized by Grammars through the use of roles

Use of roles for the recognition of utterances with several entities

If the user’s utterance contains several entities of the same type, it is required to add a role tag to the entity identifier, following the format: [entity_name:rol_value]
The role tag is used by the system to identify that there are two entities of the same type but with different roles.

This process will not affect the final output of the Grammars. That is, if the entity with the role is ent.audiovisual_sports_team:visitor, the system will only retrieve the tag ent.audiovisual_sports_team.

A practical example of Grammars that use this functionality is shown below:

The utterance “I would like to see Madrid against Barsa” contains two entities of the same type: [ent.audiovisual_sports_team]
For both entities to be detected correctly, it will be necessary to add the role when defining the name of the entities. For example, the roles of local and visitor could be assigned, resulting [ent.audiovisual_sports_team:local] and [ent.audiovisual_sports_team:visitor]

Certain considerations must be considered:

In order to recognize two entities of the same type in sequence, there is no need to create two entity graphs with the role tag but one, since the system is capable of discerning between the tagged entity and the untagged one.
Entity values should be the same in both graphs.
For better understanding purposes, we suggest using digits to name the entity graph having the role with the name of the role itself. For example: ent.audiovisual_sports_team:1 -> ent.audiovisual_sports_team_1.grf
This can be also done using non-digit characters. That is, if the entity ent.audiovisual_sports_team has the role “visitor” (ent.audiovisual_sports_team:visitor), the name of the graph should be ent.audiovisual_sports_team_visitor.grf.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Selection connectors

Introduction

Selector connectors allow, when included on a pipeline, to specify which path of the pipeline is applied depending on a certain parameter.

Currently, only one selection connector is developed in Aura NLP: Domain selector connector.

Domain selector connector

Description

The domain selector connector allows specifying which path of the pipeline is applied depending on the recognized domain. Therefore, it has to be preceded by a domain classifier step.

In this example, once the Domain Classifier has recognized the domain, the Domain Selector stage comes into play. In case the recognized domain is “domain1”, the flow continues to “Pipeline1”. Otherwise, if domain is “domain2” or “domain3”, “Pipeline2” or “Pipeline 3” are selected respectively as the following stage.

Path

auracog_pipelines.pipelines.joint.selectors.DomainSelectorPipeline

Configuration

This connector requires a specific configuration with the following fields in the dynamic NLP pipeline pipeline.json

elements: definition of every element composing the pipeline (stages and joints). It must include:
- Element name. In this case,
  .
  - type: It must be set to joint
  - classpath: path to be included in order to use this stage: auracog_pipelines.pipelines.joint.selectors.DomainSelectorPipeline
  - links: links between each specific domain and its corresponding pipeline.
  - args section: dictionary that must be included after the class path with a key named selection_map that defines a correspondence between domains and different pipelines identified by its position as a child of the pipeline in the links field. One of these domains must be default. In this case, if no domain is established, this path within the pipeline is followed.

Note that a non-existent pipeline, or missing default pipeline results in an error.

It is always required to check two items:

The correspondence is right (be aware that the index in the selection_map subsection starts with 0)
The referred pipelines exist in links section of the pipeline.json file.

Example:

"elements": {
   "<DomainSelectorPipelineName>": {
       "type": "joint",
       "classpath": "auracog_pipelines.pipelines.joint.selectors.DomainSelectorPipeline",
       "args": {
          "selection_map": {
                "<domain1>": <Index of pipeline desired for domain 1>,
                "<domain2>": <Index of pipeline desired for domain 2>,
                ...
                "<domainN>": <Index of pipeline desired for domain N>,
                "default": <Index of pipeline desired for other domains>
                               }
                                  }
 }
}

In the example:

If domain is 1, then pipeline continues with element defined in position 0.
If domain is 2, then pipeline continues with element defined in position 1.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Train Aura to understand: Use cases development over Aura NLP

Guidelines for every step in the process for personalized use cases development over Aura NLP, in order to make Aura understand the users’ requests.

Related documents
📄 Aura NLP descriptive documentation

Process at a glance

Previous requisites

. Get sure your system has the required technical resources
. Install the Aura NLP Virtual Machine
. Generate a local branch for the NLP data repository

Generate NLP model

. Build up the dynamic pipeline
. Configure the NLP model
. Generate training files, test set files and dictionaries

Train NLP model

. Train the understanding model in order to make it understand properly the users' requests

Test NLP model

. Evaluate the accuracy of the NLP model locally
. If results are satisfactory, it must be also validated by Aura Global Team

Deploy NLP package

. Merge and generate the NLP package containing the understanding model
. Deploy the new package to make it available

Introduction

This section includes the detailed process for the development of use cases over aura-nlp together with all the complementary stages that linguists and NLP experts need for this purpose.

The following figure schematically shows the workflow for the development of a use case over Aura NLP, where every stage is fully described in succeeding sections.

Firstly, get sure you fulfil all the prerequisites for the configuration of the NLP development environment.
Afterwards, follow the orderly stages in use cases development over Aura NLP.

If you are interested in a specific process, access directly to its documentation here:

Catalog of components for NLP pipelines: catalog of stages, connectors and normalization pipelines that can be used to compose the NLP pipeline.
Aura NLP entities catalogs: Description of entities catalogs, input for Aura NLP dictionaries.
Aura NLP dictionaries: Description of dictionaries, used to recognize entities.
Use of Grammars in Aura NLP: guidelines for using Grammars in an NLP model.
Complementary processes: processes that may be carried out over external software when developing a use case and procedures followed by the Aura NLP Global Team.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Guidelines for the use of Abacus 1.0.0.

After the Abacus set-up, learn how to use the tool with our guidelines.

Open Abacus

Open the NLP Virtual Machine.

Execute the following commands in the console:

cd ${Root_project}/tools
./run_web_trainings.sh

Abacus is now opened.

Discover Abacus main screen

Create a new Pull Request

Enter Abacus and select “New Pull Request” as your working mode.
In the main interface, select the channel where your understanding model is.

Train your understanding model.
When the training is finished, select “Results” in the drop-down menu and access to the results of the accuracy tests.
- End-to-end tests (accuracy of the overall NLP model)
Evaluate the accuracy of your model:

Check in “last test” the overall accuracy related to the last test performed in the tool.
Click on each specific error in “last test report” to access to detailed information.
Use the filters to visualize errors per type, intent obtained or intent expected.

Test your NLP model locally (Abacus simulator).
Once your model is trained, you can use the Simulator to launch and test your pipeline locally and make a real-time evaluation of accuracy in the recognition of a specific statement.
- Open the simulator.
- Type a phrase.
- Check how your model recognizes it.
When the accuracy is satisfactory, publish your model:

Press the “PUBLISH” button.
Create your Pull Request inserting the type, related JIRA issue, title and description and publish it.
If everything goes right, your PR is published. You can also see the details in Github.

Edit an existing Pull Request

Enter Abacus and select “Edit existing Pull Request” as your working mode.
Select your intended Pull Request. You can open it from Abacus or from GitHub to see its details.
Now, you can work over it as explained in Create a new Pull Request section, from step 2 onwards.

Abacus additional information

When Abacus is started, the API is available in: http://127.0.0.1:4000 (*)

(*) The port is variable, with values in the range: 4000 - 4100

The API documentation is included in Aura NLP API definition.

Abacus troubleshooting

Logs files

All the operations carried out with Abacus are recorded in the logs file: ${Root_project}/server.log

If an error occurs, Abacus will show this message:
Clicking on “Download”, you can access to the log report and check the error for fixing it.
If, the problem persists, contact the Global Support Team and provide them with the report.
When it is solved, press “Reload”, to continue using Abacus.

Git conflicts

If the remote branch has changes that have not been updated in the local working branch, Github detects this situation as a conflict when publishing, and is not able to recognize which are the correct changes.

The following pop up will appear after clicking on “publish”.

Clicking on “Download”, you will get the file ${Root_project}/server.log, where you can check the errors that have occurred and try to solve them.

!  [rejected]    <<working-branch>>  ->  <<working-branch>> (fetch first)
error: failed to push some refs to 'git@github.com:Telefonica/aura-nlpdata-global.git'

Conflicts in server.log must be resolved locally using the editor:

Open a terminal in code editor. Make sure the branch in terminal is the correct working branch.
Move changes from the remote branch to local:
git pull origin <<working-branch>>
A message with a list of files with conflicts is shown.
Open each file and right-click to access the conflicts resolver.
Resolve the conflict. You can include texts from the local branch (left) or from the remote one (right) or discard both and copy what is needed.
Once conflicts are resolved, launch the following command to review the status of the local branch and check that all the conflicts are fixed in the response message:
git status
When all the conflicts are resolved, commit changes:
git commit –m "[[feat]] resolve conflicts: <<description of changes>>"
Push to update changes to the remote branch:
git push origin <<working-branch>>
A message with all the details will appear.
If everything is solved, now restart the web server and continue with the existing Pull Request.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Disambiguation connector

Description

The disambiguation connector is a joint stage that allows disambiguation between different pipelines (therefore, between different recognizers).

The general behavior of this connector is shown as follows:

It executes in parallel the different pipelines.
When the execution of all the pipelines is finished, the connector will carry out a disambiguation by intents, comparing the top results from the execution of the pipelines.
However, take into account that, if there is a blacklist of intents, this behavior changes, as explained in the following section.

Disambiguation connector with a blacklist of intents

Aura NLP allows the integration of configurable blacklists of intents for a custom behavior of disambiguation. In this case, the disambiguation mechanisms will not apply for the intents included in the blacklist.

The use case constructors can edit a blacklist of intents in the nlp.json configuration file, filling the parameter intent_blacklist.

When there is a blacklist of intents, the disambiguation connector behaves as explained below:

It executes in parallel the different pipelines, with their corresponding stages.
The recognized intents from each pipeline are extracted (unless they have a None intent).
If the top scored intent of these pipelines is included in the intent_blacklist or its score is greater than the exact_match threshold, then this intent is returned.
If the top intent is not included in the intent_blacklist, then the predefined values of the configuration parameters come into play:
- All the intents between the disambiguation_margin and the top score, and not present in the intent_blacklist, are selected.
- If there is only one intent, it will be returned in a pipeline message.
- If there is more than one intent, a pipeline message with the intent intent_template and a score of 1.0 is assigned. This pipeline message will contain nor entities, neither domains, but it will contain all the selected intents in pipeline messages as options.

Path

auracog_pipelines.pipelines.joint.disambiguation.DisambiguationPipeline

Configuration

This stage requires a specific configuration in the dynamic NLP pipeline pipeline.json. The following parameters are required for this stage:

elements: definition of every element composing the pipeline (stages and joints). It must include:
- Element name. In this case, JointDisambiguation
- type: It must be set to joint
classpath: path to be included in order to use this stage: auracog_pipelines.pipelines.joint.disambiguation.DisambiguationPipeline
args section: dictionary with the following fields:
- exact_match: If the intent with the highest score is greater than this value, the result is this intent. Float number.
- disambiguation_margin: Margin between the highest score and the lower score considered for the response. Float number.
- intent_template: Name of the intent that the stage returns when there are multiple options as response. String.
- intent_blacklist: list of intents that will be removed in case there are other options. If there are no blacklisted intents it will have to be an empty list. List of strings.

See two examples of configuration for the disambiguation connector:

Disambiguation connector - Example 1

{
  "name": "Example",
  "initial_node_id": "JointDisambiguation",
  "elements": {
    "JointDisambiguation": {
      "type": "joint",
      "classpath": "auracog_pipelines.pipelines.joint.disambiguation.DisambiguationPipeline",
      "args": {
        "exact_match": 0.95,
        "disambiguation_margin": 0.2,
        "intent_template": "intent.disambiguation",
        "intent_blacklist": ["intent.intent1", "intent.intent1", ...]
      }
    },
    "OtherStage1": {
      "type": "stage",
      "classpath": "..."
    },
    "OtherStage2": {
      "type": "stage",
      "classpath": "..."
    },
    ...
  },
  "links": {
    "JointDisambiguation": [
      "OtherStage1",
      "OtherStage2",
      ...
    ]
  }
}

Disambiguation connector - Example 2

Example for a disambiguation margin equal to 0.2

Input data model

[
    {
        'query': 'original Phrase',
        'channel': 'mp',
        'intent_result': {
            'entities': [
                {
                    'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
                },
                {
                    'entity': 'name2', 'type': 'type2', 'score': 0.78, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label2'
                }
            ],
            'top_result': {'intent': 'intent.tv.search', 'score': 0.96},
            'intents': [
                {'intent': 'intent.tv.search', 'score': 0.96},
                {'intent': 'intent.tv.display', 'score': 0.95},
                {'intent': 'intent.tv.launch', 'score': 0.60}
            ]
        },
        'domain_result': {}
    },
    {
        'query': 'original Phrase',
        'channel': 'mp',
        'intent_result': {
            'entities': [
                {
                    'entity': 'name3', 'type': 'type3', 'score': 0.89, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label3'
                },
                {
                    'entity': 'name4', 'type': 'type4', 'score': 0.76, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label4'
                }
            ],
            'top_result': {'intent': 'intent.tv.display', 'score': 0.94},
            'intents': [
                {'intent': 'intent.tv.display', 'score': 0.94},
                {'intent': 'intent.tv.launch', 'score': 0.70}
            ]
        },
        'domain_result': {}
    },
    {
        'query': 'original Phrase',
        'channel': 'mp',
        'intent_result': {
            'entities': [
                {
                    'entity': 'name5', 'type': 'type5', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label5'
                },
                {
                    'entity': 'name6', 'type': 'type6', 'score': 0.78, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label6'
                }
            ],
            'top_result': {'intent': 'intent.tv.search', 'score': 0.81},
            'intents': [
                {'intent': 'intent.tv.search', 'score': 0.81},
                {'intent': 'intent.tv.display', 'score': 0.75}
            ]
        },
        'domain_result': {}
    },
    {
        'query': 'original Phrase',
        'channel': 'mp',
        'intent_result': {
            'entities': [
                {
                    'entity': 'name7', 'type': 'type7', 'score': 0.99, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label7'
                }
            ],
            'top_result': {'intent': 'intent.tv.launch', 'score': 0.60},
            'intents': [
                {'intent': 'intent.tv.launch', 'score': 0.60}
            ]
        },
        'domain_result': {}
    }
}
]

Output data model (2 intents fulfil the predefined criteria)

{
    'query': 'original Phrase',
    'channel': 'mp',
    'intent_result': {
        'entities': [],
        'top_result': {'intent': 'intent.disambiguation', 'score': 1.0},
        'intents': [
            {'intent': 'intent.disambiguation', 'score': 1.0}
        ]
    },
    'domain_result': {},
    'options': [
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': 'name1', 'type': 'type1', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label1'
                    },
                    {
                        'entity': 'name2', 'type': 'type2', 'score': 0.78, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label2'
                    }
                ],
                'top_result': {'intent': 'intent.tv.search', 'score': 0.96},
                'intents': [
                    {'intent': 'intent.tv.search', 'score': 0.96},
                    {'intent': 'intent.tv.display', 'score': 0.95},
                    {'intent': 'intent.tv.launch', 'score': 0.60}
                ]
            },
            'domain_result': {}
        },
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': 'name3', 'type': 'type3', 'score': 0.89, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label3'
                    },
                    {
                        'entity': 'name4', 'type': 'type4', 'score': 0.76, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label4'
                    }
                ],
                'top_result': {'intent': 'intent.tv.display', 'score': 0.94},
                'intents': [
                    {'intent': 'intent.tv.display', 'score': 0.94},
                    {'intent': 'intent.tv.launch', 'score': 0.70}
                ]
            },
            'domain_result': {}
        },
        {
            'query': 'original Phrase',
            'channel': 'mp',
            'intent_result': {
                'entities': [
                    {
                        'entity': 'name5', 'type': 'type5', 'score': 0.88, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label5'
                    },
                    {
                        'entity': 'name6', 'type': 'type6', 'score': 0.78, 'start_index': 1, 'end_index': 2, 'canon': 'canon1', 'label': 'label6'
                    }
                ],
                'top_result': {'intent': 'intent.tv.search', 'score': 0.81},
                'intents': [
                    {'intent': 'intent.tv.search', 'score': 0.81},
                    {'intent': 'intent.tv.display', 'score': 0.75}
                ]
            },
            'domain_result': {}
        }
    ]
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Domain Classifier stage

What is Domain Classifier?

Aura NLP can include the Domain Classifier stage preceding CLU.

The Domain Classifier stage has the objective of providing a coarse and probabilistic classification of intents per pre-defined service domains (TV services, telecom services, etc.).

Including a Domain Classifier just before the CLU stage allows to have several apps, each of them expert on a specific domain (domain-restricted CLU). Once the user’s query is classified in its corresponding domain, it will be finely recognized by the CLU app pointed out by the Domain Classifier.

Path

auracog_pipelines.stage_wrappers.domain_classifier_wrapper.domain_classifier_wrapper.DomainClassifierWrapper

Files

dispatcher.tef.json

The Domain Classifier requires one training file called dispatcher.tef.json. This file has the following fields:

metadata: metainformation such as name, modification date, domain or country of the linguistic model under consideration.
intents: dictionary, where:
- Keys: domain name
- Values: list of all the training statements (sentences, phrases or isolated words) under that particular domain.

The defined domains and statements must be the same as the ones used to train CLU in different instances. However, whereas each domain is trained in a different CLU app, the training for the Domain Classifier consists of all the training examples condensed in a single file and, instead of having the intent names as dictionary keys, it will have the domain names as dictionary keys.

To add a new domain, it is necessary to append it in the instance_map property of CLU configuration.

In addition, the training and test set files for the CLU stage must be generated including the new domain and this domain must be included, together with the statements, in the dispatcher.tef.json file.

Example

{
    "metadata": {
        "name": "Aura Dispatcher",
        "version": "dev"
        "date": "2019-01-21"
    },
    "intents": {
	      "domain.xxx": [
	          "training statement 1",
            "training statement 2",
            "training statement 3"
	]
    }
}

Best practices

It is recommendable to add comments (using double hash ‘## intent_name ##’) with the intent name, instead of removing it. In this way, it would be easier to know where the training statements of a given intent start from.
Put intents and utterances in the same order as in the CLU training. In that way, it would be easier to control changes.
Update the date of the file in order to know when the last modification was made.

Configuration

This stage requires the following configuration in the nlp.json file:

{
  "es-es": {
    "mp": {
      "training-dc": {
        "model_name": "RandomForest",
        "apply_cv": true,
        "n_cv_folds": 2,
        "fit_params": false,
        "model_params": {},
        "tv_ratio": 0.2,
        "pseudo_seed": 42,
        "ngram_min": 1,
        "ngram_max": 3
      }
    }
  }
}

The fields are explained below:

model_name: name of the algorithm used to train the model. NBayes, Rlogistica and RandomForest are the only values allowed.
apply_cv: this field indicates if the training uses cross-validation or not through true/false values.
n_cv_folds: number of folds for cross-validation.
fit_params: this field can have true/false values. If true, at the end of the training a file is created with the params used.
model_params: used as optional arguments for the algorithm selected.
tv_ratio: value between 0 and 1 indicating the percentage of test statements (sentences, phrases or isolated words) that composes the test set file.
pseudo_seed: value to initialize the seed in order to split training/test sets.
ngram_min: minimum ngrams used for internal term frequency.
ngram_max: maximum ngrams used for internal term frequency.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Generate a local branch for the NLP data repository

Discover the structure of Aura NLP data repository and learn how to clone it for working purposes in local environment

Introduction to Aura NLP data repository

The GitHub Aura NLP data repositories, for uses cases, are defined below for every country:

Use cases: aura-nlpdata-[country_code]

Both have the same specific structure of folders and files, as shown in the section Aura NLP data repository structure

Local NLP experts must work over a local branch, thus cloning the intended global repository, following the steps in section Generate a local branch.

In the continuous process for Aura NLP optimization, Aura Global Team offers the possibility of splitting the NLP repository into different repos, for a more efficient way of working. Find the details in section Split Aura NLP repository.

The following sections show the content of each folder and file in the Aura NLP repository, for use cases.

As an example, access https://github.com/Telefonica/aura-nlpdata-es

.github

GitHub config files

config/etc

This folder includes files for the configuration of the Aura NLP model:

config/etc/	Description	Modifiable for use cases development?	Detailed information
`bootstrap.cfg`	General purpose config file.	No	NLP system configuration
`nlp_config/nlp.json`	File that contains the configuration by language and channel for each stage of the pipeline.	Yes	Configure your NLP model
`build_catalogs.cfg.tpl`	File to configure source data for dictionaries. Only required if the NLP model includes stages using dictionaries.	Yes	Guidelines for the generation of dictionaries in Aura NLP
`api_trainings.cfg.tpl`	File only used in ABACUS tool. It is a configuration template that will be filled automatically with the values defined in `build_local_variables.sh`.	No	ABACUS documentation
`env.js.tpl`	File only used in ABACUS tool. This template will be filled automatically.	No	ABACUS documentation

data/

This folder includes the resources and files required for the generation of the Aura NLP pipeline and for the training of every NLP stage:

data/	Description	Modifiable for use cases development?	Detailed information
`pipeline.json`	File for building up the NLP dynamic pipeline	Yes	Build the NLP dynamic pipeline
Training files	Specific training files for each NLP stage	Yes	Define your data resources
`sdict_items.json` `sdict_aliases.json`	Dictionary files automatically generated per language and channel	Yes	Guidelines for the generation of dictionaries in Aura NLP

delivery

Internal folder containing scripts and resources related with Continuous Integration.

⚠️ Do not to modify this folder when developing new use cases.

pipeline_eval

pipeline_eval/	Description	Modifiable for use cases development?	Detailed information
pipeline_eval/ob/[country_code]/resources/[language]/[channel]/	end-to-end tests for evaluation of the pipeline accuracy per country, language and channel	Yes	Define your E2E tests

tools

Scripts for local training and testing of the Aura NLP model:

tools/	Description	Used for use cases development?	Detailed information
`build_local_variables.sh.tpl`	File for configuration purposes, specifically for the definition of CLU and other connection parameters.	yes	Set up configuration properties
`build_local.sh`	Script that automatically generates the local training environment and results files.	yes	Execute the training script
`build_local_testset.sh`	Script for the definition of specific E2E testsets files for an isolated stage. Currently, available for the OpenAI embeddings stage.	yes	Define stage-specific E2E testset files
`run_local_pipeline.sh`	Script used to test the system in a live mode during the pipeline launching stage.	yes	Launch and test your pipeline locally
`build_local_catalogs.sh`	Script used to generate dictionaries using local catalogs data.	yes	Guidelines for the generation of dictionaries in Aura NLP
`run_web_training.sh`	Script used to run ABACUS tool.	yes	ABACUS documentation
`import_nlpdata_tools.sh`	Auxiliar script used by other scripts. This script must not be executed by the user.	no	…

ℹ️ Now, all the scripts need to connect with the centralized repository in Github aura-nlp-tools, so it is necessary that your Github user have read access to it. Ask the APE Team to get this permission.

catalogs

Folder required just in case the Aura NLP uses manual catalogs.

catalogs/	Description	Used for use cases development?	Detailed information
catalogs/[language]/[channel]/	Files for the manual update of catalogs	yes	Guidelines for the generation or update of entities catalogs

validation

Configuration files for different validators.

⚠️ These files must not be modified.

gitignore

Config file containing files to be ignored by the version control system.

CODEOWNERS

Config file indicating which user or group is the code owner responsible for merging the code.

⚠️ This file must not be modified.

config.txt

File containing branch name of current working release, used in different scripts.

⚠️ This file must not be modified.

requirements.txt

File containing Python module dependencies. These dependencies are installed automatically during the training process.

⚠️ This file must not be modified.

Generate a local branch

The GitHub interaction allows the generation of local branches from the master branch.

Local NLP experts must carry out the NLP customization over the local branch, that is a clone of the NLP GitHub repository and, afterwards, create a Pull Request (PR) to push the local branch to master or release branch of the corresponding Aura release.

For this purpose, follow these steps:

Create the working directory:
```
mkdir -p ~/Telefonica
cd ~/Telefonica 
```
In order to clone the Aura NLP data project (Step 3), generate an SSH key and add it to your Github account.
For this purpose, follow the instructions in Github documentation or access to the document SSH configuration guidelines.
Clone the Aura NLP data project of your country. The repository URL follows the next pattern: https://github.com/Telefonica/aura-nlpdata-[country_code]-[optional:channelName).git

Where [country_code] is the acronym of a specific country, for example: es, br, de, gb

In order to clone the repository, it is possible to use some git client as GitKraken or it can be done directly from a console running the command:
git clone <url_repo>

The project should be cloned in the folder where the above command was executed and the folder should have the same name as the repository:
git clone git@github.com:Telefonica/aura-nlpdata-[country_code].git
Once the repository is cloned in the local machine, create a new git branch every time modifications need to be made concerning new use cases implementation, bug fixing, etc.

The name of the branch should start with one of the next reserved words, depending on the modification purpose, followed by a slash and a brief description:
- feat/: new functionalities (for example, feat/weather_forecast_UC_#56624)
- fix/: bugs or non-relevant modifications (for example: fix/balance_light_on_#117076)
- release/: release synchronization
The command to create this new branch must follow this pattern:
```
cd ~/Telefonica/aura-nlpdata-gb
git checkout -b "[feat|fix|release]/<change_description>"
```
Find here detailed information regarding Semantic Commit Messages.

Split Aura NLP repository

As a recommendation, the OB’s aura-nlpdata repository can be split by groups of channels with similar uses cases. This provides a greater flexibility and independence to constructors.

At the same time, this functionality allows optimizing the training times, as only the pipelines of the repositories that undergo modifications will be retrained.

In this scenario, the format of the repository name must be: aura-nlpdata-[country_code]-[repo_name]

If OBs want to organize their NLP repo in this way, they must contact with Aura Global Team.

Finally, it is possible to allocate dedicated processing capacity of the C.I, system, if necessary, but only after a joint analysis with Aura Global Team.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Hot swapping processes in Aura

Key processes that can be done over Aura without service outage

1. Update of locales: Hot swapping of POEditor texts

If locales belong to aura-bot (Generation or update of locales through POEditor when developing a use case):
Hot swapping is done through the aura-configuration-updater.
If locales belong to aura-groot, aura-authentication-api, aura-bridge, aura-bridge-outbound: Hot swapping is done through the guidelines Manage locales in Aura.

By doing so, the update of locales will be available without the need to wait for the next Aura Platform release.

Be aware of the impact of both processes over certain Aura components: Available hot swapping processes and associated impact - Item 1 in table.

2. Update of NLP packages

When developing a use case over Aura NLP, the understanding model including the trainings and test set files must be deployed.

If OBs are interested in the update of an NLP package through a hot swapping process, the local DevOps Team can execute two different procedures, which are equivalent in terms of outcome and impact on the system. Therefore, OBs have the freedom to choose whichever process they prefer.

Through aurak8s installer: Local NLP deployment
Through the use of Aura Operator: NlpTraining

By doing so, the updated NLP package will be available without the need to wait for the next Aura Platform release.

Be aware of the impact of both processes over certain Aura components: Available hot swapping processes and associated impact - Item 2 in table.

3. Modify TV use cases

Certain modifications in TV use cases (channels mapping, specific experiences, etc.) can be executed through a hot swapping process. They correspond to experiences that are managed by the User_helper component and, specifically, by the M+ Resolution module, that resolves TV use cases calling different APIs.

If OBs are interested in the update of a TV use case through a hot swapping process, the local DevOps Team can execute two different procedures, which are equivalent in terms of outcome and impact on the system. Therefore, OBs have the freedom to choose whichever process they prefer.

Through aurak8s installer: Local ResolutionResources deployment
Through the use of Aura Operator: ResolutionResources

Be aware of the impact of both processes over certain Aura components: Available hot swapping processes and associated impact - Item 3 in table.

4. Channels handling

The execution of changes in the configuration of channels can be done through a hot swapping process using the aura-configuration-api component.

This process is fully described in the document Update channels configuration.

Be aware of the impact of this process over certain Aura components: Available hot swapping processes and associated impact - Item 4 in table.

5. Skills handling

The hot swapping processes that can be executed in the framework of skills management are included in the document Hot swapping processes in Aura distributed architecture and include adding a new skill to aura-root, modifying it or deleting an existing one.

Be aware of the impact of this process over certain Aura components: Available hot swapping processes and associated impact - Item 5 in table.

6. Applications handling

The execution of changes in the configuration of applications, the components in charge of the communitation of a channel, service or skill with aura-gateway-api for the connection with an external service, can be done through a hot swapping process using the aura-configuration-api component. This process is fully described in the document Hot swapping processes in Aura Applications Configurations.

Be aware of the impact of this process over certain Aura components: Available hot swapping processes and associated impact - Item 7 in table.

7. Add new local dialog in aura-bot

The deployment of a new local dialog over a specific Aura Platform release can be done through a hot swapping process, following the guidelines Local modules deployment.

Be aware of the impact of this process over certain Aura components: Available hot swapping processes and associated impact - Item 8 in table.

8. Update of libraries and dialogs configuration

Aura allows making changes in certain aura-bot configuration related to libraries, dialogs and channels through hot swapping processes:

contextFilters of the intents in aura-bot dialogs
Scopes and purposes in aura-bot libraries and dialogs

These processes can be done using the aura-configuration-updater.

Be aware of the impact of this process over certain Aura components: Available hot swapping processes and associated impact - Item 9 in table.

9. Add static resources in a use case

When developing a use cases, aura response can include static resources such as visual components (cards), such as icons, images, diagrams, etc.

If required, the update of these resources can be done through a hot swapping process, following the guidelines included here.

These processes can also be done using the aura-configuration-updater.

Doing that, the updated static resources will be available in the period between one release and the consecutive one.

Be aware of the impact of this process over certain Aura components: Available hot swapping processes and associated impact - Item 10 in table.

10. Review indexes definitions

Due to the continuous evolution of the Aura Platform, it is possible that the indexes definitions of the different components need to be reviewed and updated. This process can be done through a make-up process.

If required, the update of these indexes can be done through a hot swapping process, uploading a new index definition file to Azure Blob Storage, in the container aura-configuration/AURA-VERSION and then executing the make-up process of the corresponding component.

Follow the guidelines Manage MongoDB Indexes to understand how to define a new index file.

Be aware of the impact of this process over certain Aura components: Available hot swapping processes and associated impact - Item 11 in table.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

NLP system configuration

Internal configuration of Aura NLP system: operational configuration and configuration of NLP stages

Introduction

The configuration of the NLP system is organized in two different purposes, each of them supported by one configuration file:

NLP operational configuration:
- Internal configuration for NLP system
- Not modifiable by OBs
- Based on the file bootstrap.cfg
- For descriptive purposes, it is included below
NLP stages configuration:
- Configuration of each stage composing the NLP pipeline
- Configurable when developing an NLP model for a specific use case
- Based on the file nlp.json
- The practical process for editing the nlp.json pipeline when developing a use case is included in Configure your NLP model

NLP operational configuration: bootstrap.cfg

The bootstrap.cfg file contains operational config sections for Aura NLP (ports, URIs, usernames, passwords, etc.).

This file can be found in the path:
aura-nlpdata-[country_code]/config/etc/bootstrap.cfg

⚠️ When developing a use case, NLP engineers or linguists should not modify this file. If any modification is required, it must be approved by the Aura Platform Team.

The file follows the general structure shown hereunder:

[working_directory]
stages_folder =./tmp/

[logging]
handlers = { . . . }
loggers = { . . . }
root = { . . . }

[country-langs]
country_mapper = { . . . }

[channels]
channel_list = [
        {
             'prefix': 'fb',
	         'name': 'whatsapp',
	         'id': '269d6-f052-4d2e-8f66-f59a9f31eff9'
	    },
. . . ]

[platform]
platform = 'ES'

[azure_models]
container_url = ${AZURE_NLP_MODELS_URL}

Moreover, it is required to include in this file other different sections belonging to specific stages or databases used. The fields included in each section are described below.

Working directory

[working_directory]
stages_folder = ./tmp/

The main fields are explained below:

stages_folder: Main directory for the different stages.

Logging

[logging]
handlers = {
      'hdl1': {
         'class':'logging.StreamHandler',
         'formatter':'console',
         'level':'INFO'
      }
   }
loggers = {
      'nlp': {
         'level': 'INFO',
         'handlers': [
            'hdl1'
         ],
    'filters': []
      }
    }
root = {
      'level':'INFO',
      'handlers': [
         'hdl1'
      ]
    }

The main fields are explained below. However, for more details, developers are kindly requested to read the General Python logging documentation

handlers: This field configures a dictionary with different logging handlers. Each key is the name of a handler, and it is composed by the next parameters:
- class: It is configured with Python logging handlers (See Python documentation).
- formatter: It configures the format of logs. It must be filled with the labels json, string, console or simple.
- level: Level of the logging event. It must be filled with the labels INFO, ERROR, WARN or DEBUG.
loggers: The corresponding value is a Python dictionary in which each key is a logger name and each value is a dictionary describing how to configure the corresponding Logger instance:
- level (optional parameter): Level of the logger.
- handlers (optional parameter): List of IDs of the handlers for this logger.
- filters (optional parameter): List of IDs of the filters for this logger.
root: Configuration for the root logger.
- level (optional parameter): Level of the logger.
- handlers (optional parameter): List of IDs of the handlers for this logger.

Country-langs / channels / platform

[country-langs]
country_mapper = { 
  'es-es': {
        'country_name': 'Spain',
        'language_name': 'Spanish',
        'alpha2': 'es',
        'alpha3': 'esp',
        'culture': 'es-es'
    }
  }

[channels]
channel_list = [
        {
            'prefix': 'mp',
            'name': 'movistar-plus',
            'id': '60f0ffda-e58a-4a96-aad9-d42be70b7b42'
        },

  ]

[platform]
platform = 'ES'

The main fields are explained below:

country_mapper: Mapper with a list of fields that specifies the allowed languages based on the ISO-639 code.
channel_list: List of available channels. This field must contain three parameters for each channel. This information is already configured for every OB.
- prefix: Prefix of the channel.
- name: Name of the channel.
- id: ID of the channel.
platform: Allowed platform.

CLU

The CLU stage requires a specific operational configuration:

[CLU]
base_url = https://${RESOURCE_NAME_CLU}.cognitiveservices.azure.com
base_url_api = https://${RESOURCE_NAME_CLU}.cognitiveservices.azure.com
api_version = 2023-04-01
http_retry_codes = {429, 500}
http_max_attempts = 10
http_sleep_time = 5
http_time_out = 60
http_time_out_recognizer = 60
http_retry_codes_recognizer = {429, 500}
http_max_attempts_recognizer = 5
http_sleep_time_recognizer = 0.5
http_raise_when_retry_limit_exceeded_recognizer = True

The main fields are explained below:

base_url: Base URL for CLU service.
base_url_api: Base URL for CLU API service.
api_version: CLU API version.
http_retry_codes: Response status code, if more requests than the limit have been sent.
http_max_attemps: Maximum number of HTTP requests allowed.
http_sleep_time: Timeout between HTTP requests.
http_time_out: Time in seconds for raising a timeout exception when HTTP request does not return a response for training API requests.
http_time_out_recognizer: Time in seconds for raising a timeout exception when HTTP request does not return a response for CLU recognizer.
http_retry_codes_recognizer: Set of response status codes that will retry CLU recognizer request.
http_max_attemps_recognizer: Maximum number of attempts that will be performed in CLU recognizer request when there is an exception by timeout or connection error or a request code defined in http_retry_codes_recognizer is set.
http_sleep_time_recognizer: Time to wait between HTTP CLU recognizer requests.
http_raise_when_retry_limit_exceeded_recognizer: Boolean (true/false) value to inform if an exception must be re-raised when it happens and the maximum number of retries is exceeded.

OpenAI Embeddings

The OpenAI Embeddings stage configuration allows to have different databases per each combination of language and channel.

Some of these values will be configured by the installer aurak8s, such as base_url_api.
It is also necessary to enable its configuration in aurak8s installer, following the instructions in the Enable OpenAI deployment section.

[openai_embeddings_recognizer]
azure_token_base_url = https://login.microsoftonline.com
management_url = https://management.azure.com
management_api_version = 2023-05-01
http_retry_codes = {429,500}
http_max_attempts = 10
http_sleep_time = 5
http_time_out = 30
base_url_api = https://test.openai.azure.com/openai
base_api_version = 2023-05-15
http_time_out_recognizer = 20
http_retry_codes_recognizer = {429,500}
http_max_attempts_recognizer = 10
http_sleep_time_recognizer = 10
http_raise_when_retry_limit_exceeded_recognizer = True
sku_name = Standard
sku_capacity = 120

[qdrant:instance]
url = http://hotname:6333
api_key = api-test
shard_number = 1
replication_factor = 1
chunk_size = 30
exponential_sleep = True
max_exponential_sleep_time = 120

The associated fields are defined below:

azure_token_base_url: Base URL to get oauth token.
management_url: Azure URL where the embedding OpenAI model will be deployed.
management_api_version: Version of the embedding OpenAI model in Azure.
http_retry_codes: Response status code to retry request.
http_max_attemps: Maximum number of HTTP requests allowed.
http_sleep_time: Timeout for each attempt when we retry any HTTP request.
http_time_out: Time in seconds for raising a timeout exception when HTTP request does not return a response for OpenAI embeddings training API requests.
base_url_api: Base URL for OpenAI embeddings service.
base_api_version: OpenAI embeddings version.
http_time_out_recognizer: Time in seconds for raising a timeout exception when HTTP request does not return a response for OpenAI embeddings recognizer.
http_retry_codes_recognizer: Set of response status codes that will retry OpenAI embeddings recognizer request.
http_max_attemps_recognizer: Maximum number of attempts that will be performed in OpenAI embeddings recognizer request when there is an exception by timeout or connection error or a request code defined in http_retry_codes_recognizer.
http_sleep_time_recognizer: Time to wait between HTTP OpenAI embeddings recognizer requests.
http_raise_when_retry_limit_exceeded_recognizer: Boolean (true/false) value to inform if an exception must be re-raised when it happens and the maximum number of retries is exceeded.
sku_name: Name of the resource model representing the SKU.
sku_capacity: Capacity of Tokens per Minute Rate Limit (Thousands).
url: URL for Qdrant service.
api_key: Key needed to connect with Qdrant service.
shard_number: Number of shards for Qdrant service.
replication_factor: Replication factor for Qdrant service.
chunk_size: Number of embeddings to be sent in each request to the Qdrant service.
exponential_sleep: Boolean (true/false) value to inform if the exponential sleep is enabled. By default, it is False.
max_exponential_sleep_time: Maximum time in seconds for the exponential sleep. By default, it is 120 seconds.

base_url_api = https://internal.com/
http_retry_codes = {429, 500}
http_max_attempts = 10
http_sleep_time = 5
http_time_out = 30

Where:

http_retry_codes: Response status code to retry request.
http_max_attempts: Maximum number of HTTP requests allowed.
http_sleep_time: Timeout for each attempt when we retry any HTTP request.

Azure models

The azure_models configuration is detailed below:

[azure_models]
container_url = ${AZURE_NLP_MODELS_URL}

Where:

container_url: URL for the Azure NLP models container.

Embeddings Domain Classifier

The Embeddings Domain Classifier stage configuration allows the use of different databases per each combination of language and channel.

Some of these values will be configured by the installer aurak8s, such as base_url_api.

It is also necessary to enable its configuration in aurak8s installer, following the instructions in the Enable OpenAI deployment section.

[openai_embeddings_domain_classifier]
azure_token_base_url = https://login.microsoftonline.com
management_url = https://management.azure.com
management_api_version = 2023-05-01
http_retry_codes = {429,500}
http_max_attempts = 10
http_sleep_time = 5
http_time_out = 30
base_url_api = https://test.openai.azure.com/openai
base_api_version = 2023-05-15
http_time_out_domain_classifier = 20
http_retry_codes_domain_classifier = {429,500}
http_max_attempts_domain_classifier = 10
http_sleep_time_domain_classifier = 10
http_raise_when_retry_limit_exceeded_domain_classifier = True
sku_name = Standard
sku_capacity = 120


[qdrant:instance]
url = http://hotname:6333
api_key = api-test
shard_number = 1
replication_factor = 1
chunk_size = 30
exponential_sleep = True
max_exponential_sleep_time = 120

The associated fields are defined below:

azure_token_base_url: Base URL to get oauth token.
management_url: Azure URL where the embedding OpenAI model will be deployed.
management_api_version: Version of the embedding OpenAI model in Azure.
http_retry_codes: Response status code to retry request.
http_max_attemps: Maximum number of HTTP requests allowed.
http_sleep_time: Timeout for each attempt when we retry any HTTP request.
http_time_out: Time in seconds for raising a timeout exception when HTTP request does not return a response for OpenAI embeddings training API requests.
base_url_api: Base URL for OpenAI embeddings service.
base_api_version: OpenAI embeddings version.
http_time_out_domain_classifier: Time in seconds for raising a timeout exception when HTTP request does not return a response for embeddings domain classifier.
http_retry_codes_domain_classifier: Set of response status codes that will retry embeddings domain classifier request.
http_max_attempts_domain_classifier: Maximum number of attempts that will be performed in embeddings domain classifier request when there is an exception by timeout or connection error or a request code defined in http_retry_codes_domain_classifier.
http_sleep_time_domain_classifier: Time to wait between HTTP embeddings domain classifier requests.
http_raise_when_retry_limit_exceeded_domain_classifier: Boolean (true/false) value to inform if an exception must be re-raised when it happens and the maximum number of retries is exceeded.
sku_name: Name of the resource model representing the SKU.
sku_capacity: Capacity of Tokens per Minute Rate Limit (Thousands).
url: URL for Qdrant service.
api_key: Key needed to connect with Qdrant service.
shard_number: Number of shards for Qdrant service.
replication_factor: Replication factor for Qdrant service.
chunk_size: Number of embeddings to be sent in each request to the Qdrant service.
exponential_sleep: Boolean (true/false) value to inform if the exponential sleep is enabled. By default, it is False.
max_exponential_sleep_time: Maximum time in seconds for the exponential sleep. By default, it is 120 seconds.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Naeba

Description and stages

Naeba is a pipeline used for the normalization of the user’s utterance through the execution of the following normalizers:

PunctuationNormalizer
SplitPunctNormalizer
SpaceNormalizer
CurrencyNormalizer
LowercaseNormalizer

Configuration

This stage requires the following configuration in the nlp.json configuration file:

{
  "es-es": {
    "mp": {
         "nlp": {
         "normalizer_pipeline_class": "auracog_pipelines.pipelines.normalization.naeba.NaebaPipeline"
      }
    }
  }
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Components for NLP pipelines

Current catalog of stages, connectors and normalization pipelines existing in the Aura Platform release that can be used to compose the NLP pipeline

Aura NLP pipelines are the basis for the generation of an understanding model.

Linguists must design their pipeline through the most appropriate combination of stages for the recognition of intents and entities in the use case and join these stages through different types of connectors in order to set a specific behavior in the pipeline flow. They can also use nested normalization pipelines in order to homogenize the input request.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Catalog of NLP normalization pipelines

Catalog of NLP normalization pipelines to compose the NLP pipeline

Aura Platform Team has implemented a set of normalization pipelines in order to be nested in the NLP model pipeline. They are built joining different normalization stages (normalizers).

In every use case, it is necessary to choose the most adequate normalization pipeline.

For example, if numbers are expected to be expressed with text characters (i.e., “one”), it is useful to include the normalization stage CardinalityNormalizer to turn them into digits (“1”).

Another example refers to the fact that written requests are required. In this situation, it can be important to include a normalization stage that reduces transcription mistakes.

Select your intended normalization pipeline in the left menu. Each of them is characterized by its description and configuration.

Section	Content	Role in the NLP process
Description	Identification and objective of the stage in the recognition process	Descriptive purpose of the stage in the recognition process
Configuration	Required configuration for each NLP stage	Configuration of each stage of the NLP model

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Best practices for the generation of a Pull Request

This process is required once the NLP model is fully developed and tested in local environment and it’s time to create a Pull Request to the corresponding release branch : Pull Request to release branch.

Best practices

When creating a Pull Request, include the NLP Global Team as reviewers of the process and, likewise, notify the APE Team.
It is mandatory to create reduced PRs (per use case, per bug, etc.) in order to speed up the validation process.
Do not modify configuration files during the Pull Request, excepting in case the pipeline has been changed or if any configuration adjustment is required for the system’s proper performance. If configuration files have been modified locally for testing purposes, get sure that they are not uploaded in the PR in order to avoid conflicts.
It is recommended to specify different tasks in the PR, so the review progress can be marked:

It is recommended to make a backup for those PRs modifying files that may conflict with other ones, or for large Pull Requests.
If the use case is going to be available in different channels, check that the content and order of the training files is the same.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Abacus 1.0.0. user guide

Learn how to use Abacus, the tool for the training, testing and certification of the NLP model in Aura through an agile and efficient process

What’s Abacus?

Abacus is a web tool that eases the management of understanding models in Aura. Using Abacus, you can get a higher efficiency and autonomy, leading to a more agile certification of your NLP trainings.

Linguists or NLP experts can use Abacus to train, test and certify a full NLP understanding model in a specific channel and verify that Aura is able to understand properly.

With Abacus, you can certify your NLP model in four steps:

Upload my NLP training
Train & test the full understanding model
Live mode: Iterate and test
Check your accuracy and iterate your model
Publish your model
Publish your branch in Github transparently

Limitations of Abacus version 1.0.0.

Abacus journey map

The following image shows the main steps in your process for using Abacus. Access the guidelines for each step in the following documents:

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP

Aura NLP is the component in charge of processing, analyzing and understanding human natural language. Discover throughout these documents key descriptive documentation regarding this component.

Shared component between Aura Virtual Assistant and ATRIA

Related documents
Use cases development over Aura NLP

What is Aura NLP?

Aura NLP (Natural Language Processing) is the module of Aura Cognitive Services in charge of processing and understanding human natural language in simplified use cases.

Aura’s interaction with users is based on the intent & entity model: a user’s request expressed in natural language is understood by Aura in terms of identifying the user’s intent and the associated entities.

An NLP model contains three basic components: stages, connectors and pipelines. Stages provide different methods for the recognition of intents and entities in the user’s utterance. They are linked through different types of connectors composing an NLP pipeline.

When developing a use case, linguists or NLP experts must build up the NLP model and train it, that is, teach Aura to understand. Afterwards, the model is tested through an ongoing and cyclical process until its accuracy is good enough in terms of recognition of the use case intent and entities.

Throughout this section, you can access to detailed information, both descriptive and practical, regarding Aura NLP:

📄 Aura NLP basic concepts and components. Key concepts that must be known by linguists in order to manage Aura NLP.
📄 Configuration of the NLP system. Description of NLP operational configuration (internal) and introduction to the configuration of NLP stages.
📄 API definition
📄 Moreover, access our practical guidelines for NLP experts and linguists: Train Aura to understand: Use cases development over Aura NLP.

Overview of intent and entities recognition

Aura’s conversational process with the user is composed of three overall stages: the user makes a request to Aura; Aura recognizes the user’s intent and associated entities; Aura provides the user with the requested answer or service.

Two are the main actors in the process: while aura-bot is the component in charge of handling the conversational flow with the user, Aura NLP is responsible for the understanding process, which is schematically shown below.

Aura user asks for a service/request (utterance) through a specific channel.
aura-bot receives the request and handles it. For its understanding, aura-bot summons Aura NLP.
Aura NLP recognizes the intents and associated entities in the user request and sends the information back to aura-bot.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Generation of Aura NLP catalogs

Entities catalogs are the input for the Aura NLP dictionaries, used to recognize entities from the users’ utterances.

Introduction

Catalogs in Aura are knowledge bases of entities. These catalogs are the input for the generation of Aura NLP dictionaries to be included in an NLP model.

Discover in the current documents:

Types of catalogs in Aura NLP

There are two types of catalogs, at least one of them is required:

Automatic catalogs: data from Kernel URM
Manual catalogs: data in catalogs/ folder

Automatic catalogs

Telefonica Kernel URM is a database that includes data from different key content such as film title, documental title, TV series title, TV shows, actors’ name, directors’ name, etc.

Aura can connect to the URM and automatically download the URM content when the NLP dictionaries (sdict files) are generated. You can indicate in the configuration whether to take data from Azure or AWS.

Data that can be downloaded from the URM correspond to the section urm_type_entities in the nlp.json configuration file:

audiovisual_director
audiovisual_actor
audiovisual_documental_title
audiovisual_film_title
audiovisual_tvshow_title
audiovisual_tvseries_title

The URM database should be continuously updated, in order to show the most recent content and scheduled programs (for instance, new films or series in Movistar + catalog).

As NLP dictionaries automatically include the data from the URM database, two situations are found that can lead to the generation of manual catalogs:

The URM must be completed with the very latest content that can be offered to the user and must be recognized by Aura. In case a relevant entity is missing, the catalog must be updated manually.
Linguists can detect mistakes in URM data: wrong formats, typos, missing aliases, etc. To overcome this problem, the manual updating of catalogs is required.

Manual catalogs

Catalogs can be updated manually in the catalogs/ folder, included in Aura NLP data directory: aura-nlpdata-[country_code]

This folder contains, categorized by language and channel, all the files required for the manual updating of entities.

The final goal is to complete the dictionaries with entities that should be recognized by the NLP system (when a NER stage is used) and to complete and/or refine data from URM (in case this source is used).

The detailed process to update manual catalogs is included in Guidelines for the generation or update of entities catalogs.

Guidelines for the generation or update of manual catalogs

As explained before, apart from automatic catalogs, that provides data from Kernel URM database, manual catalogs can be also generated to complete the automatic ones with new entities or correct mistakes.

The following sections include the orderly guidelines for the generation or update of manual entities catalogs.

1. Identify content

Identify content to be updated in dictionaries: very latest content that must be included in dictionaries and recognized by Aura (for instance, new films or series in Movistar+ catalog).
Check if this content (entities) are included in the URM database:
- These specific entities are missing
- Any mistake is detected in URM data regarding these entities (wrong formats, typos, missing aliases, etc.)

2. Access the catalogs/ folder and edit it

Access the catalogs/ folder in: aura-nlpdata-[country_code]/catalogs

Now, you should edit the different files, each one with its corresponding data as shown in the following sections.

2.1. auth/ folder

Working directory: aura-nlpdata-[country_code]/catalogs/[language]/[channel]/auth/

auth/ folder contains multiple JSON files including prioritized content that are added to the sdict_item.json and sdict_aliases.json dictionaries.

Follow these steps to edit the auth/ folder:

Organize data into different JSON files by entity types (for example, one file for time entities and another for tv entities).
It is mandatory that files names have the format: <file_name>.ent.json
Add a file named most_relevant_content.ent.json for those key entities that must be recognized with 100% accuracy related to these fields:
- Film title > ent.audiovisual_film_title
- Documental title > ent.audiovisual_documental_title
- TV series title > ent.audiovisual_tvseries_title
- TV shows > ent.audiovisual_tvshows_title
Add a JSON file for organizing any other entity type or topic (for example, movistar+_sports.ent.json.
Edit each JSON file:
- metadata field should include the following fields:
  - format: specification of format used in file.
  - name: representative name to identify the content of the file.
  - version: this should be updated when changing the file.
- Keys: entity types
- Values: list of entities
If the item is a string, it is considered a canon and deleted from the rest of the entity types where it is found.
If it is a list, the first element of the list is considered a canon and the rest of values are aliases for this canon. The canon is deleted from the rest of the entity types where it is found and aliases are removed from the sdict_aliases.json dictionary.

Example

{
    "metadata": {
        "format": "tef:dict:entity",
        "name": "AURA Movistar XXX",
        "version": "1.0"
    },
    "ent.audiovisual_sports_team": [
       [
           "Real Madrid",
           "el Real Madrid|comment",
           "##el Real Madrid",
           "Madrid"
       ],
       [
           "Sevilla|comment",
           "el Sevilla",
           "Sevilla club de fútbol",
           "Sevilla futbol club"
       ]
    ]
}

Best practices

Comments can be added, since the script ignores them:
1. Adding “##” before a value. ("## Spanish Football Teams")
2. Adding “|” in a value or entity type, the text after this symbol is not considered as part of the entity (“el Real Madrid|comment”)
Maintain correct indentation to ease catalogs reading.
Declared entities, canons and aliases should be ordered alphabetically.
Capitalize: first letter for proper nouns, titles, teams, companies, etc. (“The Wedding Date”); acronyms (“Chelsea FC”).
Write punctuation correctly within values. For example, “Chelsea F C” could be written also as “Chelsea F.C.”. Do not include both forms because it could cause a duplicate due to normalization process.
If the language includes words with diacritical marks, write values correctly.
Check that the canon is the expected in case the API expects a specific one.
Compare canon/alias included in catalogs to avoid overlaps and conflicts.
Avoid duplicates.

2.2. add/ folder

Working directory: aura-nlpdata-[country_code]/catalogs/[language]/[channel]/add/

add/ folder contains multiple JSON files including additional or non-prioritized content to be added to the sdict_item.json and sdict_aliases.json dictionaries. It is used to complement information in dictionaries. In case there is non-prioritized content, this folder will be empty.

Follow these steps to edit the add/ folder:

Organize data into different JSON files by entity types (for example, one file for time entities and another for tv entities).
It is mandatory that files names have the format: <file_name>.ent.json
Edit each JSON file:
- metadata field should include the following fields:
  - format: specification of format used in file.
  - name: representative name to identify the content of the file.
  - version: this should be updated when changing the file.
- Keys: entity types
- Values: list of entities
If the item is a string, it is considered a canon and added to sdict_items.
If it is a list:
1. The first element of the list is considered a canon and added to sdict_items.json.
2. The rest of values are aliases and are included in sdict_aliases.

Example

{
    "metadata": {
       "format": "tef:dict:entity",
       "name": "AURA Movistar XXX",
       "version": "1.0"
    },
    "ent.audiovisual_sports|comment": [
       [
           "GOLF",
           "Golf"
       ],
       [
           "tennis|comment",
           "##tenis",
           "tenis",
           "ten"
       ]
    ]
}

Best practices

Best practices for the auth/ folder also apply to add/ folder.

2.3. precedence.json file

Working directory: aura-nlpdata-[country_code]/catalogs/[language]/[channel]/precedence.json

precedence.json file establishes the priority of an entity type over the rest in the sdict_items.json dictionary.

Follow these steps to edit the precedence.json file:

Edit the file including:
- Keys: entity type
- Values: list of entity types over which the key prevails.

Example

If the entity “Real Madrid” is present in both ent.audiovisual_documental_title and ent.audiovisual_sports_team, and we want soccer teams to have priority over documentaries, it has to be defined in precedence.json like this:

{
     "ent.audiovisual_sports_team": [
         "ent.audiovisual_documental_title"
     ]
}

Doing this way, “Real Madrid” of the entity type ent.audiovisual_documental_title would be eliminated.

Best practices

Entities declared should be ordered alphabetically.
Be careful to maintain the required JSON format.

2.4. skip.json file

Working directory: aura-nlpdata-[country_code]/catalogs/[language]/[channel]/skip.json

skip.json file defines conflicting items that must be eliminated from sdict_items.json and sdict_aliases.json dictionaries.

Follow these steps to edit the skip.json file:

skip_items_in_entity: dictionary, where:
- Keys: entity type
- Values: list with entities to be deleted from that type of entity. Values defined here affect just canons.
skip_items_in_all_entities: list of values which will be removed from all types of entities where included. It affects to canons and aliases.

Example

{
    "skip_items_in_entity": {
        "ent.audiovisual_film_title": [            
            "telefono",
            "the movie",
            "la resistencia"
        ],
        "ent.audiovisual_tvseries_title": [            
            "cine",
            "director",
            "pelicula"
        ],
        "skip_items_in_all_entities": [            
            "El peliculon",
            "dummy alias",
            "dummy del"
        ]
    }
}

Best practices

Entities declared should be ordered alphabetically.
Values inside entities should be ordered alphabetically.
Be careful to maintain the required JSON format.
Include values as they are found in dictionaries, respecting capitalization, diacritical marks, etc. The system deletes not only these values but also their normalized version.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Grammars stage

Description of Grammars

Grammars provide an exact and lightweight utterance’s recognition method that offers a deterministic approach: specific utterances from the users are recognized if they are included in Grammars.

This approach makes Grammars interesting for Aura NLP, due to the existence of specific utterances from Aura users that must be recognized by Aura (such as common utterances from users or difficult ones that are hardly recognized by an intent recognition stage such as CLU).

Find detailed information regarding Grammars in Use of Grammars in Aura NLP model:

Path

auracog_pipelines.stage_wrappers.grammar_wrapper.pygrape_grammar_wrapper.PygrapeGrammarWrapper

Files

This stage needs the following training files for each language and channel:

Dico: .dic files. These files include standardized content and must not be modified.
Grammar: .grf files, generated by Unitex.
[entity_extraction_mapper.json]

In addition, if local grammars are used, you must generate two additional files in order to evaluate the compatibility between the global and the local grammars. These two files are placed in the test_grammar folder:

commons/testset.json. This file is used for checking that both grammars, global and local, recognize the same test set statements. You must fill in the test set with key statements, as shown in the following example:

[
  "call 600586375",
  "turn on the light"
]

disjoints/testset.json. This file is used for checking that the test set statements are only recognized by the global grammar (if the statements do not apply to the local grammar scope). You must fill in the test set with key statements, as shown in the following example:

[
  "watch coco on tv"
]

Configuration

This stage requires one of the following configurations per channel in the nlp.json file:

Use this configuration to define a single intent prefix with a pre-defined string. In this example, the string intent is defined as the intent prefix in the mp channel.
```
{
  "es-es": {
    "mp": {
      "grammar": {
        "intent_prefix": "intent"
      }
    }
  }
}
```
Use this configuration to define a list of possible intent prefixes for the intent name. The items passed inside the list intent_matches can be explicit strings or regular expressions written in string format that the intent name must start with, according to the pattern passed in the regex. In order to define a regex for the intent prefix, start the string with the keyword regex: and then add the regular expression.
In this example, the strings tef.int. or intent. are the two possible intent prefixes that the intent name must start with in the mp channel.
```
 {
   "es-es": {
     "mp": {
       "grammar": {
         "intent_matches": ["regex:^tef\\.int\\.[a-z]+$", "intent."]

       }
     }
   }
 }
```

There are two mutually exclusive allowed parameters per channel in the configuration file. They are defined below:

intent_prefix: prefix to be added to the intent determined by the grammar.
intent_matches: a list of strings with possible prefixes to be added to the intent determined by the grammar. The strings passed can be explicit strings or regex written in string format. If a regex is passed, it must contain the keyword regex: at the beginning of the string to be processed as a regular expression.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Nikko normalization pipeline

Description and stages

Nikko is a pipeline used for the normalization of the user’s utterance through the execution of the following normalizers:

PunctuationNormalizer
SplitPunctNormalizer
SpaceNormalizer
CurrencyNormalizer
UnicodeNormalizer
LowercaseNormalizer
CardinalityNormalizer
PunctuationNormalizer
SpaceNormalizer

Configuration

This stage requires the following configuration in the nlp.json configuration file:

{
  "es-es": {
    "mp": {
         "nlp": {
         "normalizer_pipeline_class": "auracog_pipelines.pipelines.normalization.nikko.NikkoPipeline"
      }
    }
  }
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Review of a Pull Request by NLP Global Team

Procedure followed by the NLP Global Team in order to validate the Pull Request including the NLP model

This process is done once the Pull Request is launched, for the evaluation of the NLP model by the NLP Global Team: Certify NLP model accuracy: review by the NLP Global Team.

Introduction

The review of the Pull Request including the NLP model carried out by the NLP Global Team includes the processes explained in the following sections.

It can be very useful for Local Teams to know these processes and criteria used by the NLP Global Team in order to focus on the critical points.

Categories of errors and problems

Detected errors are classified into three categories:

BLOCK: Blocking task. It must be resolved in order to approve and merge the PR. In case there are certain blockers to be modified, the system dismisses the GitHub Pull Request and publishes a comment describing the problem and indicating the procedure to resolve it. This case requires re-training the NLP model.
NON-BLOCK: Mandatory but non-blocking task. It must be resolved following the guidelines and best practices in the current or in further PRs.
SUGG: Not mandatory but recommended modifications that should be taken into account even for subsequent PRs. For them, it is recommended to inform the NLP Global Team whether the suggestion is taken into account or not.

The setting of an adequate threshold for the NLP system accuracy depends on the use case. Therefore, for a specific use case, the minimum accuracy should be agreed by L-CDO and the NLP Global Team.

Best practices for the Pull Request validation

These best practices should be followed both by the NLP Global Team and the local linguists, if they participate in the validation process.

Take into account the following icons that indicate different status to reviewers:
- 👍 It indicates that the reported problem has been visualized and will be included in further commits.
- 👀 It indicates that linguists have gone over the comment but it is not resolved yet. In this situation, linguists must include an explanatory text with the justification of this status (for example, to be resolved later; disagreement; etc.)
Comments should be launched from the corresponding file or from the general screen (conversation). For its resolution, click Resolve conversation or select Hide from the drop-down menu. Afterwards, select the option Resolved.
If the comment cannot be resolved, it is edited and substituted by “OK”.
In general, reviewers are in charge of changing the comment status to Resolved.
Comments should be as clarifying as possible by including screenshots or other references.
In case a comment resolution is pending, the local developer must be informed and it is recommended to change the status to still pending.
If the answer to a comment by the reviewer is not clearly understood, the local team can contact him.
If modifications affect to several channels, changes can be uploaded to one channel and, afterwards, copied to the other channels.
Comparation of branches:
- In case of merging of a large PR, it is recommended to compare the corresponding branches to avoid information to be lost. For this purpose, Pycharm can be useful.
- The tool compare allows this comparison, just by selecting the folder/file with the right bottom, selecting the option Git and compare with branch and then clicking on the branch to be compared.
- The different files appear in different colors: existing files in blue, added files in green and deleted ones in grey. By clicking on a file, a new window is opened showing the differences between branches.
- It is also possible to compare branches and versions from Github: https://github.com/Telefonica/[REPO] /compare/
For the PR review, it is recommended the use of REGEX. Some examples are included below:
- Finding duplicates: ^(.?)$\s+?^(?=.^\1$)
- No space after an entity: [ent.[a-z_]+][a-z]+
- No space before an entity: [a-z]+[ent.[a-z_]+]
- No extra spaces after values: \h+$
- Sentences missing: "\¿[a-záéíóúñ _[].]+"
- Sentences missing: "[a-záéíóúñ _[].]+?"
The PR is reviewed by different members of the team, within an ongoing process.

Most frequent comments in the review process

The following table includes some of the most frequent comments that are reported during the review of the Pull Requests by the NLP Global Team, organized by category.

⚠️ Please, take the following tables as merely indicative in terms of the category where each comment is included as, depending on the specific scenario and the use case specifications, a comment can be moved from one category (“block”, “non-block” or “sugg”) to another.

Review of CLU training and testset

The following best practices are valid for the CLU intent recognition stage.

Entities

Block	non-block	Sugg
Ill-formed (incorrect name, missing ‘[‘, blank space missing before/after the entity; blank space before ‘:’ in the entity name)	Alphabetic order missing (by type and by value)	Structuring of training and test set files in blocks (for example, verbs, use cases, entities, etc.)
Value declared in phraselist but not tagged in training set	“Cosmetic changes”: uppercase letters, question marks, unnecessary blank spaces, accents	New values for entities
Values with an incorrect entity	Indentation	Suggestions on phrases for training and test set files
Repeated values in two entities		Suggestions on new entities
Repeated values for a specific entity		Suggestions on patterns for the test set file
Value tagged but not declared in a phraselist
Typographical errors (if not on purpose), missing words
Values representativeness: as far as possible, the training set must contain all the different values of entities. At least, it must include a representative list of them

Intents

Block	non-block	Sugg
Intent name not agreed by the Global Team	Alphabetic order missing (by type and by value)	Structuring of training and test set files in blocks (for example, verbs, use cases, entities, etc.)
All intents not represented in the training set and testset files	“Cosmetic changes”: uppercase letters, question marks, unnecessary blank spaces, accents	New values for entities
Overlap between intents	Indentation	Suggestions on phrases for training and test set files
Phrases with out-of-scope intent		Suggestions on new entities
Typographical errors, missing words		Suggestions on patterns for the test set file
Repeated phrases
Illogical phrases
Unfulfillment of ratio 80%-20% for training-test statements

Files

Block	non-block	Sugg
Ill-formed json files
Not updated date
Different information between channels (between shared intents)
Modification on configuration files (except to agreed changes)

Review of E2E testset

Block	non-block	Sugg
Ill-formed json files	“Cosmetic changes”: uppercase letters, question marks, unnecessary blank spaces, accents	Structuring of training and test set files in blocks (for example, verbs, use cases, entities, etc.)
Wrong position of entities	Lack of representativeness of the different structures	New values for entities
Incorrect tags	Alphabetic order missing (by domain, intent & utterance)	Suggestions on phrases for training and test set files
Not represented intent	“Default” domain	Suggestions on new entities
Wrong order for keys: phrase, domain, intent, entities		Suggestions on patterns for the test set file
Typographical errors (if not on purpose), missing words
Accuracy lower than 80% (by default value set by Aura Global Team)
Result validation: Review of results from the PR, identification of errors and improvement suggestions
Regression file: Bugs or specific phrases not included in the `testset.json` file that must be recognized
Canonical phrase not included in E2E testset
Unfulfillment for recommended number of testing statements in the E2E test set: - 20 statements (CLU); - 30 statements (CLU + Grammar); - 3 statements (Grammar)

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Generation of Aura NLP dictionaries

Aura NLP dictionaries are knowledge bases used to recognize entities from the users’ utterances.

Process at a glance

Update
catalogs

. Firstly, check if catalogs must be updated to include the latest content.
. If required, update catalogs manually.

Generate
dictionaries

. Check that your NLP model is configured to use dictionaries
. Run the script and generate both items and aliases dictionaries

Entities
in Grammars

. Add new entities in dictionaries to the Grammars model to get sure that these entities are recognized with 100% accuracy.

Retrain
NLP model

. Retrain the understanding model
. Validate it
. Merge and generate the NLP package
. Deploy the updated package

Introduction

The recognition of entities in the Aura NLP model is based on dictionaries: knowledge bases of entities that are included in the NLP model as part of stages for the recognition of entities in the user’s utterance.

Currently, these stages are Standard NER, Gazetteer NER and Entity Tagger Adapter.

Dictionaries are generated automatically from catalogs, during the NLP flow, when developing a use case.

Discover in the current documents:

Types of dictionaries in Aura NLP

There are two types of dictionaries defined in Aura:

Items dictionary: it includes all the different values in its canonical form for each entity type. The canonical question is defined as the most common way to mention a specific entity. This file distinguishes by entity types.
Alias dictionary: it includes the canonical value of a given concept (those found in items dictionary) and its list of aliases, that is, the most significant alternative names of an entity canon. This file does not distinguish by entity types.

For example, a TV use case can include the following dictionaries:

Items dictionary: ent.audiovisual_actor: [Robert de Niro, Dustin Hoffman; Al Pacino, …]
Alias dictionary: Robert de Niro: [De Niro, Robert Niro, Robert Deniro, …]

Aura NLP uses two dictionaries for entities recognition:

Items dictionary: sdict_items.json
Alias dictionary: sdict_aliases.json

Items dictionary

sdict_items.json consists of a dictionary whose keys are the names of all the entity types and the value of each key includes a list with the canonical values of those entities. All canonical forms should be contemplated in this file.
This file is automatically generated based on the data from manual catalogs and data from URM.

An example of sdict_items.json dictionary is shown below:

{
  "ent.audiovisual_actor": [
    "Angelina Jolie",
    "Brad Pitt",
    "Cate Blanchett",
    "Jennifer Anniston",
    "Jennifer Lawrence",
    "Morgan Freeman"
	]
}

Alias dictionary

sdict_aliases.json contains all the possible values (aliases) for an entity. These aliases are different ways to refer to the same value.
The dictionary keys are the canonical value of a given concept (those found in the sdict_items.json file) and their value is a list of aliases, meaning all the potential ways of referring to that concept. This file does not distinguish by entity types.
The alias dictionary is automatically generated based on the data from manual catalogs and data from URM.

Examples of the sdict_aliases.json dictionary are shown below:

{
  "#0": [
     "0",
     "zero",
     "the zero"
  ]
}

{
  "The Mandalorian": [
     "De Mandalorian",
     "De Mandaloriano",
     "El Mandalorian",
     "El Mandaloriano",
     "Mandalorian",
     "Mandaloriano",
     "el mandalorian",    
     "el mandaloriano",
     "te mandalorian",   
     "the mandalorian" 
…  ]
}

Generation of Aura NLP dictionaries

When developing a use case in Aura, if it requires the recognition of entities, the NLP model must include any of the entities recognition stages:

In these stages, as part of the step for defining data resources, where all the training files required for every specific stage must be generated, the sdict dictionaries must be included.

For this purpose, follow these steps:

1. Check if content in catalogs is updated and complete

Manual catalogs are one of the inputs for NLP dictionaries.

At this stage, you have to check if their content is totally updated or if it is required to generate a newer version to include the very latest content (for instance, new films or series in Movistar+).

Discover how to generate or update content in manual catalogs in Aura.

⚠️ If the catalogs content is identical in different channels, the dictionaries can be generated just for one channel and then copied to the rest of them.

2. Configure the NLP model to use dictionaries

Dictionaries require a specific configuration, that must be set during the configuration of the NLP model, with two differentiated stages:

2.1. Dictionaries configuration in nlp.json file

If dictionaries are used, specific sections must be included in the nlp.json file, placed in the path: aura-nlpdata-[country_code]/config/etc/nlp_config/nlp.json

urm_type_entities: from all the URM entities in the catalogs, it indicates which ones must be downloaded.
headers_ignore: list with all the headers to be ignored.
ner: this section is required as the StandardNer class is instantiated when building the catalogs:
- n_context_words: number of context words used in the BILOU algorithm.
- phone_number_entity_type: type of entity to be assigned to an entity recognizer as phone number.

Example:

{
  "test-test": {
     "test_channel": {
        "training-sner": {
           "urm_type_entities": [
              "ent.audiovisual_director",
              "ent.audiovisual_actor",
              "ent.audiovisual_documental_title",
              "ent.audiovisual_film_title",
              "ent.audiovisual_tvshow_title",
              "ent.audiovisual_tvseries_title"
           ],
           "headers_ignore": [
              "metadata"
           ],
        },    
        "ner": {
           "n_context_words": 3,                    
           "phone_number_entity_type": "ent.phonenumber"
        }
      }
   }
}

2.2. Dictionaries configuration in build_catalogs.cfg.tpl

The file build_catalogs_cfg.tpl is only required if the dictionaries sdict_item.json and sdict_aliases.json are generated from the manual catalogs in three specific stages: Standard NER, Gazetteer NER, and Entity Tagger Adapter.

It is placed on the path:
aura-nlpdata-[country_code]/config/etc/build_catalogs.cfg.tpl

Edit this file to indicate, for each language and channel, if URM data is to be downloaded and used as source for the generation of dictionaries.

For this purpose, the following fields must be filled, depending on the script used for the generation of dictionaries:

If the new global script build_local_catalogs_etl.sh is used, the following parameters must be filled:
Recommended method
- urm_mapper: dictionary that indicates, for each language and channel, if it has to download the URM.
- To connect to API URM:
  - $API_URM_ENDPOINT
  - $USER_KERNEL_ACCESS_TOKEN
  - $PASSWORD_KERNEL_ACCESS_TOKEN
If the original script build_local_catalogs.sh is used, the following parameters must be filled:
- urm_mapper: dictionary that indicates, for each language and channel, if it has to download the URM.
- resources_provider: provider, that can be aws or azure.
- container: folder that includes the data to be downloaded. It can be $AWS_S3_BUCKET or $AZURE_CONTAINER.
- key and secret: these fields correspond to provider credentials.
  - To connect to AWS, you need:
    - $AWS_ACCESS_KEY
    - $AWS_SECRET_KEY
  - To connect to Azure, you need:
    - $AZURE_ACCOUNT_NAME
    - $AZURE_SAS_TOKEN

Example:

[catalogs]
resources_provider = aws
container = $AWS_S3_BUCKET or $AZURE_CONTAINER
urm_mapper = {
        'es-es': {
            'mp': {
                'urm': True
            },
            'stb': {
                'urm': True
            },
            'stbh': {
                'urm': True
            },
            'la_global': {
                'urm': True
            }
        }
    }
 
[aws]
key = $AWS_ACCESS_KEY
secret = $AWS_SECRET_KEY
 
[azure]
account_name = $AZURE_ACCOUNT_NAME
sas_token = $AZURE_SAS_TOKEN

[direct_sql:instance]
base_url = ${API_URM_ENDPOINT}
user = ${USER_KERNEL_ACCESS_TOKEN}
password = ${PASSWORD_KERNEL_ACCESS_TOKEN}

3. Set up specific configuration variables for dictionaries

Before training your understanding model, it is required to set up the configuration properties. Check the general process in the previous link.

If dictionaries are included in the model, there are certain additional variables required for the execution of the dictionaries script, which are enumerated below.

Moreover, the last six variables must only be defined when data from URM is included for the generation of the dictionaries.

Remember that you need to indicate the name of the CATALOGS_RESOURCES_PROVIDER provider and the container where the data is. Then, you only need the credentials of the chosen provider:

export CHANNEL_LIST: list of channels where dictionaries are generated. For example: export CHANNEL_LIST="la_global mh mp"
export LANGUAGE: language for the generation of files. For example: export LANGUAGE=“es-es”
export AZURE_CATALOGS_ACCOUNT_NAME: Azure account name where the data is.
export AZURE_CATALOGS_TOKEN: Azure SAS token.
export AWS_CATALOGS_ACCESS_KEY: AWS Access Key credential.
export AWS_CATALOGS_SECRET_KEY: AWS Secret Key credential.
export CATALOGS_RESOURCES_CONTAINER: Container or bucket name.
export CATALOGS_RESOURCES_PROVIDER: Provider name, aws or azure.
export API_URM_ENDPOINT: Endpoint of URM API.
export USER_KERNEL_ACCESS_TOKEN: Username Kernel Access Token.
export PASSWORD_KERNEL_ACCESS_TOKEN: Password Kernel Access Token.

4. Run the script for the generation of dictionaries

There are two alternatives to generate dictionaries:

Use the new global script that makes use of the URM content datasets uploaded to the Kernel platform:
Recommended method
- Run the global script build_local_catalogs_etl.sh, located at:
  aura-nlpdata-[country_code]/tools/build_local_catalogs_etl.sh
Use the original script that downloaded the information from the previously chosen URM containers:
- Run the original script build_local_catalogs.sh, located at:
  aura-nlpdata-[country_code]/tools/build_local_catalogs.sh

After the script execution, the NLP dictionaries sdict_items.json and sdict_aliases.json are automatically generated in:
/aura-nlpdata-[country_code]/data/[language]/[channel]

You can create a Pull Request directly and see changes in comparison with the previous files.

Complementary, they are also placed in the temporary folder tmp_catalogs:

You can also check the downloaded data from URM in the urm_bucket folder inside tmp_catalogs.

5. Best practices for checking dictionaries

Once the dictionaries are generated, there are certain checks that should be done:

Check that values that have been added and removed from catalogs are updated in sdict_items.json and sdict_aliases.json.
Check that all the canons that have been included in the catalogs appear in sdict_items.json and all the aliases appear in sdict_aliases.json with its corresponding canon.
Check that, at least, all the aliases of a canon that have been included in the catalogs appear in sdict_aliases.json under the expected canon.
Check that there are no unwanted duplicates. It is highly recommendable to check that the same canon (or normalized one) does not appear in different entities to avoid possible overlaps. For these situations, use the catalogs’ skip.json file for skipping values from dicts and use the precedence.json file to prioritize an entity type.
Once both sdict_items.json and sdict_aliases.json have been generated, all the values that were added to the catalogs should be tested in your local environment to check that they retrieve their corresponding canon, entity type and label. In case there is an error, check what it is due to and make the necessary modifications. This step should be repeated until the result is the expected one.

5. Add new entities in dictionaries to the Grammars stage

⚠️ Of application just in case Grammars stage is included in the NLP model.

If you want to assure that new entities included in dictionaries are recognized with 100% accuracy, they must be included in the Grammar stage.

The NLP stage Grammars has specific guidelines for the generation of the required files through the software Unitex: Guidelines for the generation of Grammars in Unitex.

Take into account that input data for the Grammars stage should be normalized first.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP tutorials

Tutorials for the development of a use case over aura-nlp

Index of tutorials

COMING SOON

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

NLP configuration by OB

NLP configuration by OB for the TV content get info use case

Introduction

The current document includes the configuration of the NLP model for the TV content get info use case in the OBs where it is available.

Brazil

The aura-nlpdata repository structure in Brazil is organized by channel: aura-nlpdata-br-[channel]. Each repository will contain the files for the available use cases in this channel.

The following configuration corresponds to the STB channel, which is included in the repository: https://github.com/Telefonica/aura-nlpdata-br-stb

Intent

intent.tv.content_get_info

Entities

ent.audiovisual_actor
ent.audiovisual_actor
ent.audiovisual_best
ent.audiovisual_channel
ent.audiovisual_director
ent.audiovisual_documental_title
ent.audiovisual_film_title
ent.audiovisual_genre
ent.audiovisual_most_watched
ent.audiovisual_releases
ent.audiovisual_sports
ent.audiovisual_sports_circuit
ent.audiovisual_sports_player_driver
ent.audiovisual_sports_player_rider
ent.audiovisual_sports_season
ent.audiovisual_sports_season_motor
ent.audiovisual_sports_team
ent.audiovisual_sports_unit
ent.audiovisual_subgenre
ent.audiovisual_tv_episode_number
ent.audiovisual_tv_season_number
ent.audiovisual_tvseries_title
ent.audiovisual_tvshow_title
ent.device_mobile
ent.device_tv
ent.time_instant
ent.time_interval
ent.time_interval_future
ent.time_interval_past

Pipeline stages

The following stages of STB’s pipeline have been used to develop this UC:

PygrapeGrammarWrapper
EntityTaggerAdapterWrapper

The whole pipeline.jsonfile is available at: https://github.com/Telefonica/aura-nlpdata-br-stb/blob/master/data/pt-br/stb/pipeline.json

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

NLP configuration by OB

NLP configuration by OB for the TV move backwards use case

Introduction

The current document includes the configuration of the NLP model for the TV move backwards use case in the OBs where it is available.

Brazil

The aura-nlpdata repository structure in Brazil is organized by channel: aura-nlpdata-br-[channel]. Each repository will contain the files for the available use cases in this channel.

The following configuration corresponds to the STB channel, which is included in the repository: https://github.com/Telefonica/aura-nlpdata-br-stb

Intent

intent.tv.move_backwards

Entities

ent.audiovisual_genre
ent.time_length_sec
ent.time_length_min
ent.time_length_hour
ent.time_instant

Pipeline stages

The following stages of STB’s pipeline have been used to develop this UC:

PygrapeGrammarWrapper
EntityTaggerAdapterWrapper

The whole pipeline.jsonfile is available at: https://github.com/Telefonica/aura-nlpdata-br-stb/blob/master/data/pt-br/stb/pipeline.json

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

NLP configuration by OB

NLP configuration by OB for the TV move forwards use case

Introduction

The current document includes the configuration of the NLP model for the TV move forwards use case in the OBs where it is available.

Brazil

The aura-nlpdata repository structure in Brazil is organized by channel: aura-nlpdata-br-[channel]. Each repository will contain the files for the available use cases in this channel.

The following configuration corresponds to the STB channel, which is included in the repository: https://github.com/Telefonica/aura-nlpdata-br-stb

Intent

intent.tv.move_forwards

Entities

ent.audiovisual_genre
ent.time_length_sec
ent.time_length_min
ent.time_length_hour
ent.time_instant

Pipeline stages

The following stages of STB’s pipeline have been used to develop this UC:

PygrapeGrammarWrapper
EntityTaggerAdapterWrapper

The whole pipeline.jsonfile is available at: https://github.com/Telefonica/aura-nlpdata-br-stb/blob/master/data/pt-br/stb/pipeline.json

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

NLP configuration by OB

NLP configuration by OB for the TV None use case

Introduction

The current document includes the configuration of the NLP model for the TV None use case in the OBs where it is available.

Brazil

The aura-nlpdata repository structure in Brazil is organized by channel: aura-nlpdata-br-[channel]. Each repository will contain the files for the available use cases in this channel.

The following configuration corresponds to the STB channel, which is included in the repository: https://github.com/Telefonica/aura-nlpdata-br-stb

Intent

intent.tv.none

Entities

No use specific entities to obtain results.

Pipeline stages

The following stages of STB’s pipeline have been used to develop this UC:

CluRecognizerWrapper

The whole pipeline.jsonfile is available at: https://github.com/Telefonica/aura-nlpdata-br-stb/blob/master/data/pt-br/stb/pipeline.json

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

NLP configuration by OB

NLP configuration by OB for the TV VOD EPG information use case

Introduction

The current document includes the configuration of the NLP model for the TV VOD EPG information use case in the OBs where it is available.

Brazil

The aura-nlpdata repository structure in Brazil is organized by channel: aura-nlpdata-br-[channel]. Each repository will contain the files for the available use cases in this channel.

The following configuration corresponds to the STB channel, which is included in the repository: https://github.com/Telefonica/aura-nlpdata-br-stb

Intent

intent.tv.vod_epg_information

Entities

ent.device_tv
ent.device_mobile
ent.device_phone

Pipeline stages

The following stages of STB’s pipeline have been used to develop this UC:

“ExactMatchRecognizerWrapper”
“PygrapeGrammarWrapper”
“EntityTaggerAdapterWrapper”

The whole pipeline.jsonfile is available at: https://github.com/Telefonica/aura-nlpdata-br-stb/blob/master/data/pt-br/stb/pipeline.json

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Standard NER stage

What is Standard NER?

Standard Named Entity Recognition (Standard NER) is a process based on machine learning for information extraction that seeks to locate and classify named entities in a text into pre-defined categories.

The input for Standard NER is the normalized user’s utterance. It searches for entities in the utterance and categorizes the recognized words in pre-defined categories (labelling).

The first step when using Standard NER is the creation of dictionaries of entities that are knowledge bases (KB) used to train the NER to recognize, extract and label entities from the user’s utterance. Once the NER is properly trained, it will act as an intelligent system able to think by itself and recognize entities not previously existing in the dictionaries.

Moreover, Standard NER takes into account the entity context (considering not only the analysis of the isolated word but also the left and right words).

Which movies do you have with Clint Eastwood as actor?
Which movies do you have with Clint Eastwood as director?

Standard NER training flow

The training process for Standard NER is schematically shown in the figure below.

Aura Standard NER uses the BILOU tagging scheme for encoding information in a set of labels. BILOU encodes the Beginning, Inside and Last token of multi-token chunks while differentiates them from unit-length chunks.

The feature extraction phase extracts features from tokens, therefore helping their characterization and recognition. This process uses diverse ways to discriminate tokens with the purpose of extracting named entities: Cases; Numbers; Part of speech (PoS); Dictionary entries; Word itself. The feature extraction can also use features from adjacent words in order to take into account the entity context in the decision-making.

When the tokens are recognized as pre-defined entities, Standard NER replaces these tokens by labels. Therefore, the output generated is the user’s utterance tagged in the following way:

Standard NER input	Standard NER output
I want to watch the movie The Matrix	I want to watch the [ent.audiovisual_genre] [ent.audiovisual_film_title]

Standard NER is also capable of recognizing multi-token entities. (i.e., “Out of Africa”). However, Standard NER has a limitation: It can recognize an entity composed of a maximum of 6 tokens.

Path

auracog_pipelines.stage_wrappers.ner_wrapper.standard_ner_wrapper.StandardNerWrapper

Files

The Standard NER stage requires the generation of the following training files:

dispatcher.tef.json
sdict_items: Items dictionary including the canonical values of entities.
sdict_aliases: Alias dictionary including the aliases for an entity.
entity_extraction_mapper.json

entity_extraction_mapper.json

This file allows the selection and configuration of which stage is in charge of the entities’ extraction.

For this purpose, the file entity_extraction_mapper.json must be created in the folder: aura-nlpdata-[country_code]/data

It has the following format:

Keys: entity types
Values: stages in charge of the entity extraction

{
  "default": [
    "standard_ner",
    "grammar",
    "clu",
    "gazetteer_ner"
  ],
  "ent.audiovisual_film_title": [
    "clu"
  ]
}

In the previous example the format indicates that, in the NLP recognition process, four stages are in charge of the entity extraction: Standard NER, Grammar, CLU and Gazetteer NER. But for a specific entity type, ent.audiovisual_film_title, the entity extraction is only done by CLU, and the stages Standard NER, Gazetteer NER and Grammar ignore it.

The name of the corresponding stage must be defined as shown in the example above. The default key is not mandatory. If a specific entity type is not declared specifically or there is no default key within the entity_extraction_mapper.json file, then every entity of this type is discarded.

Configuration

This stage requires the following configuration in the nlp.json file.

{
  "es-es": {
      "ner": {
        "n_context_words": 3,
        "phone_number_entity_type": "ent.phonenumber"
      }
    }
  }
}

The Standard NER config is distributed between the training-sner section (config fields for the training stage) and the ner section (fields for the production phase), with the following fields:

apply_cv: this field indicates if the training uses cross-validation or not through (true/false).
n_cv_folds: number of folds for cross-validation.
fit_params: this field can have true/false values. If true, at the end of the training a file is created with the params used.
model_params: used as optional arguments for the algorithm selected.
algorithm: name of the training algorithm, with the next allowed values:
lbfgs: gradient descent using the L-BFGS method
l2sgd: stochastic Gradient Descent with L2 regularization term
ap: averaged Perceptron
pa: passive Aggressive (PA)
arow: adaptive Regularization of Weight Vector (AROW)
verbose: boolean value to enable trainer verbose mode.
max_iterations: integer value with the maximum number of iterations for optimization algorithms.
tv_ratio: value between 0 and 1 indicating the percentage of statements (sentences, phrases or isolated words) that composes the test set file.
pseudo_seed: value to initialize the seed in order to split training/test sets.
explore_n_features: parameter used for the model evaluation.
repeat: parameter of BILOU algorithm that defines the number of repetitions for each value.
n_context_words: number of context words used in the BILOU algorithm.
phone_number_entity_type: type of entity to be assigned to an entity recognizer as phone number.

Additionally, for the configuration of dictionaries, two aditional fields can be included optionally:

urm_type_entities: from all the URM entities, in this section developers should indicate which ones they want to be downloaded.
headers_ignore: list with all the headers to be ignored.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Gazetteer NER stage

What is Gazetteer NER?

Gazetteer NER is a stage defined in the NLP recognition process as an alternative engine to NER for entities recognition. This stage is based on deterministic entity detection: it recognizes entities only based on their presence in the dictionaries, matching terms in the dictionaries with a user’s utterance.

Moreover, Gazetteer NER has been designed with entity-level discrimination capabilities, therefore enhancing its selectiveness by allowing it to detect only instances for a given entity type.

Gazetteer NER stage can appear in a pipeline in parallel to Standard NER (merging both results according to a fixed criteria) or sequentially (letting one engine detect entities not covered by the previous NER engine).

This stage is also capable of recognizing multi-token entities. (i.e “Out of Africa”). However, it has a limitation, as Gazetteer NER can recognize an entity composed of a maximum of 6 tokens.

Path

auracog_pipelines.stage_wrappers.ner_wrapper.gazetteer_ner_wrapper. GazetteerNerWrapper

Files

The Gazetteer NER stage requires the generation of the following training files:

sdict_items: Items dictionary including the canonical values of entities.
sdict_aliases: Alias dictionary including the aliases for an entity.
entity_extraction_mapper.json

Configuration

No configuration is required for the Gazetteer NER in the nlp.json file.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Use of Grammars in Aura NLP

This section includes the description of Grammars, a deterministic recognition method used in Aura NLP for the recognition of the users’ utterances, their role in the NLP model and practical processes regarding how to use this stage in the understanding process

What are Grammars?

Grammars are a tool that provides an exact and lightweight utterance’s recognition method through a deterministic approach. Grammar uses probabilistic formalisms to recognize specific utterances from the users and to identify how to interpret them.

Aura NLP include Grammars as a stage that can be included in the NLP pipeline. It use has key limitations due to the large burden of building the language model, as Grammars are only able to recognize exact utterances. However, because of it, they constitute an interesting segment within Aura NLP, due to the existence of specific utterances produced by Aura’s users that must be recognized by Aura (such as common utterances from users or difficult ones that are hardly recognized by CLU).

Discover in the documents:

Detailed description of Grammars: Grammars engines, types of Grammars
Guidelines for the generation of Grammars in Unitex
Recognition of utterances with several entities in Grammars

Grammars engines: GrapeNLP and Unitex/GramLab

GrapeNLP is used by Aura NLP for intent recognition and entity extraction using grammars. This grammar engine is based on handcrafted grammars which describe in an exact manner the sentences that are to be recognized and the output information that is to be generated for each one, in our case, the intent the sentence corresponds to and the entities to extract.

Linguists should develop by hand the grammar that exactly recognizes the required sentences. Just in the case of ambiguity (multiple interpretations defined in the grammar for the same sentence), GrapeNLP uses a heuristic approach in order to choose one of the interpretations: the one that in the grammar uses more restrictive linguistic conditions.

The core of GrapeNLP is implemented in C++ and includes a Python module to facilitate its integration with Python programs. It can analyse around 2700 sentences per second in an average computer and can be run in Ubuntu, Alpine, MacOS and Android. Moreover, it is open source and LPGL licensed, thus it can be used in commercial products.

GrapeNLP does not include a grammar editor. Instead, we use the editor included in the Unitex / GramLab platform. Unitex / GramLab is also LGPL licensed and can be installed in Windows, Linux and MacOs machines. The grammars created with Unitex are represented with graphs organized in connected boxes that linguists can easily create and update manually. Each box contains a set of possibilities for each token from the user’s utterance. The combination of different connected boxes provides a full variability of sentences to be recognized. The system also allows the generation of sub-grammars for specific Aura domains or for certain intents.

Once the grammars have been developed in Unitex, the grammar engine goes through all the graph paths from the beginning (left side) and compares box by box the user’s utterance with the grammar to evaluate the matching.

At the end of each path, a score is specified corresponding to the highest score among all the feasible paths. The output is a set of labels together with a start and end index and a score. The output is presented as a .json format.

It is important to bear in mind that, currently, grammars are used in Aura mainly for intents recognition. The grammar engine only provides recognized entities that have previously been labelled in the graphs. As another example of Grammars, the utterance “I would like to watch the film Frozen” provides the following output:

PipelineMessage:
       -OriginalMessage:
         -phrase: 'I would like to watch the film Frozen'
       -normalized_phrase: 'i would like to watch the film frozen'
       -normalized_presentable_phrase: 'I would like to watch the film Frozen'
       -annotated_phrase: ' i would like to watch the film frozen'
       -intent:  intent.tv.search'
       -score: 1.0
       -entities:
               Entity: Frozen, Type: ent.audiovisual_film_title, Score: 1.0, Start index: 31, End index: 37, Canon: frozen, Label: None, Deep Links: None

Note that, even though GrapeNLP does not make use of statistical methods or probabilities, the resulting .json includes a score field. This has been added for homogeneity with the machine learning workflow, but it is always hardcoded to 1.0 (since GrapeNLP performs exact matching, the probability is 100%). The machine learning pipeline never returns a score of 1.0, thus this field can be used for knowing whether the sentence was recognized by GrapeNLP or by an intent recognition stage (CLU, etc.).

📄 For more information regarding the use of Grammars for language recognition, please check the Unitex User Manual.

Global and local grammars

There are two types of grammars defined in Aura NLP recognition process, both based on the Grammars engine that offer a different performance depending on the location where they are executed:

Global grammars: defined and executed in Aura back-end.
Local grammars: they are a subset of the Global grammar.

The understanding process is carried out locally, in the channel side, for an agile resolution of the process, therefore allowing a significant latency reduction. It is available for a selected set of use cases.

Global and local grammars must be aligned, so there are no differences in the E2E understanding process (for instance, the same user input must provide the same result in terms of NLP recognition both global and local grammars).

Channels can automatically update their local grammars based on the grammar backend information. Moreover, the channel needs to be able to share the information with the global backend in terms of logs and KPIs.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Kernel configuration for URM Global script

Guidelines for the configuration of the script URM Global in Kernel

Introduction

Aura NLP dictionaries can now be generated and configured using the URM Global script build_local_catalogs_etl.sh.

In order to have a correct behavior for the URM data extraction used in the global script, it is necessary to execute the tasks defined in the following sections, that are a particularization of the general guidelines “Kernel configuration: General steps”.

1. Check APIs publication in Kernel

Check that the directsql:query API is published in Kernel: List of available APIS on Telefónica Kernel.
If not, follow the guidelines in the document Publish an API in Kernel.

2. Check datasets publication in Kernel

Check that the required datasets for the configuration of the URM Global script are published in Kernel: List of available datasets on Telefónica Kernel.
- D_Gbl_Video_Content_Type
- Video_Content
- D_Gbl_Video_Staff_Role
- Video_Content_Staff_Rel
- D_Video_Staff_Role
- D_Video_Staff

3. Create a Kernel application

A Kernel application with the name aura-cognitive-trainings must be created and configured with specific scopes.

Ask the Kernel Team to create the new application in Kernel: "id": "aura-cognitive-trainings"

Once the app is created, two parameters will be provided for securely accessing: - client_id: unique identifier of the consuming app acting as Kernel API client. - client_secret: password.

4. Assign purpose/scopes to the application

No purpose is required, as datasets do not include personal information.
Ask the Kernel Team to assign the following scopes to the application:
- data:Video_Content:read
- data:Video_Content_Staff_Rel:read
- data:D_Video_Staff:read
- data:D_Video_Content_Category:read
- data:D_Video_Staff_Role:read
- data:D_Video_Age_Rating:read
- data:D_Gbl_Video_Content_Category:read
- data:D_Gbl_Video_Staff_Role:read
- data:D_Gbl_Video_Content_Type:read
- data:D_Gbl_Video_Age_Rating:read
- directsql:query

5. Add other required fields

Provide the Kernel Team with other necessary fields, as shown in the code snippet.

The final file for the configuration of the application, including all the above-mentioned parameters, is shown below:

{
  "name": "Data consumption for Aura Cognitive Trainings",
  "grant_types": [
    {
        "authentication": "client_credentials"
        "scopes": [
               "data:Video_Content:read",
               "data:Video_Content_Staff_Rel:read",
               "data:D_Video_Staff:read",
               "data:D_Video_Content_Category:read",
               "data:D_Video_Staff_Role:read",
               "data:D_Video_Age_Rating:read",
               "data:D_Gbl_Video_Content_Category:read",
               "data:D_Gbl_Video_Staff_Role:read",
               "data:D_Gbl_Video_Content_Type:read",
               "data:D_Gbl_Video_Age_Rating:read",
               "directsql:query"
             ]
        "purposes": []
        "api": "directsql:query"
    }
  ],
  "description": "Aura cognitive application to consumption data of the kernel",
  "raw_dataset_read": true,
  "tags": {},
  "encrypt_access_tokens": true,
  "id": "aura-cognitive-trainings",
  "requires_authorization_id": true,
  "client_type": "CONFIDENTIAL",
  "legal_entity_id": "telefonica",
  "redirect_uris": []
}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura NLP metrics

List of metrics available in Aura NLP

These metrics are stored since Heroes (7.0.0.) release

http_request_duration_seconds

This Prometheus metric is modelled as a summary where its value is the spent time until the remote host responds to an HTTP request.

Note that the value is a float number rounded to its third decimal. It is stored as a Summary in Prometheus.

This metric is intended to store the duration of outgoing requests in seconds.

Labels:

All label values are strings.

method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.).
path: HTTP path of the incoming request.
status_code: the responded HTTP status code (as a string).

Value:

Request duration in seconds.

outgoing_request_duration_seconds

This Prometheus metric is a modelled as a summary where the value is the spent time until the remote host responds to an HTTP request.

Note the value is a float number rounded to its third decimal. It is stored as a Summary in Prometheus.

This metric is intended to store the duration of outgoing requests in seconds.

Labels:

All label values are strings.

method: HTTP method (GET, POST; etc.), a string in uppercase.
host: remote host that will receive the outgoing request.
path: HTTP path of the outgoing request.
status: the responded HTTP status code (as a string).

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Complementary processes in the development process

Processes over external software that may be required when developing a use case over Aura NLP and best practices

Introduction

This section includes certain processes that may be carried out over external software when developing a use case in order to obtain credentials from these software, best practices for the generation of Pull Requests and procedures followed by the Aura NLP Global Team.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Noro normalization pipeline

Description and stages

Noro is a pipeline used for the normalization of the user’s utterance through the execution of the following normalizers:

PunctuationNormalizer
SplitPunctNormalizer
SpaceNormalizer
CurrencyNormalizer
UnicodeNormalizer
LowercaseNormalizer
WordReplacerFromFileNormalizer
CardinalityNormalizer
PunctuationNormalizer
SpaceNormalizer

Configuration

This stage requires the following configuration in the nlp.json configuration file:

For the specific language and channel, in the nlp field of this JSON file, the key normalizer_pipeline_class must be filled in with the value:
auracog_pipelines.pipelines.normalization.noro.NoroPipeline

{
  "es-es": {
    "mp": {
         "nlp": {
         "normalizer_pipeline_class": "auracog_pipelines.pipelines.normalization.noro.NoroPipeline"
      }
    }
  }
}

Aura – aura-nlp

Docs:

Aura NLP Adapters

What are Aura NLP adapters

Description

Path

Files

Configuration

Intent Entity Mapper

Description

Path

Files

Best practices to edit intent_entity_mapper.json

Configuration

None Handler

Description

Path

Files

Configuration

Standard Threshold

Description

Path

Files

Configuration

Entity Tagger Adapter

Description

Path

Files

ner_entity_translation.json

ner_aliases.json

Best practices for the edition of ner_aliases.json

Configuration

Description

Path

Files

Configuration

Intent Disambiguation Adapter

Description

Intent disambiguation with a blacklist of intents

Path

Files

Configuration

Examples of intent disambiguation adapter

Length Adapter

Description

Path

Files

Configuration

Docs:

How to obtain Azure credentials for OpenAI

Prerequisites

Guidelines

Docs:

Guidelines for the generation of Grammars in Unitex

General guidelines

Intents and entities tagging

Tag an intent in the grammar interface

Tag an entity in the grammar interface

Best practices for graphs generation

Best practices for the generation of .grf files

Graphs

Domain folder

Intent folder

Entity graphs

Verbal structure and nomenclature

Introduction

Auxiliary verbs

aux_W: auxiliary verb + infinitive tense

aux_Y: auxiliary expressions/verbs + imperative tense

aux_SQT: auxiliary verb + present/past imperfect subjunctive tense

Main verbs

Verbal graphs

[verb]_main_graph structure

Guidelines for testing Grammars in Unitex

1. Testing grammars using the Unitex interface

2. Testing grammars through run_local_pipeline.sh scripts

Docs:

Logical connectors

Introduction

Competitive connectors

Best practices to edit `intent_entity_mapper.json`