Aura – kernel

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Kernel configuration: General steps

General guidelines for the execution over Kernel of tasks required for Aura features

Introduction

Certain Aura features requires the execution of preliminary tasks over Kernel to access its integrated resources and capabilities, such as APIs, datasets, etc.

The following sections outline the tasks that are common to all Aura features. Additionally, each of them will have their own specific requirements.

1. Check APIs publication in Kernel

If the Aura feature uses a Kernel API, check if the API is published in Kernel: List of available APIS on Telefónica Kernel.
If not, follow the guidelines in the document Publish an API in Kernel.

2. Check datasets publication in Kernel

Only necessary if the Aura feature requires datasets

Identify the required datasets for the configuration of the feature:
- Aura entities
Check that these datasets are published in Kernel: List of available datasets on Telefónica Kernel.
If not, ask the Kernel Team to publish the datasets in Kernel productive environment with the latest version.

3. Create a Kernel application

Accessing Kernel data requires the previous generation of an application (Kernel client), which must be configured with permissions to access specific datasets.

For certain Aura features, a specific Kernel application must be created from scratch. Other ones require the use of already existing Kernel applications, such as aura-bot-[environment], that must be specifically configured for this feature (Step 4).

Ask the Kernel Team to create a new application with a specific name (id) in Kernel for your intended environment.
Once the app is created, two parameters will be provided for securely accessing:
- client_id: unique identifier of the consuming app acting as Kernel API client.
- client_secret: password.

4. Assign purpose/scopes to the application

Only in the case of data that contain personal information (PI-scopes), it is necessary to create a purpose in Kernel, that defines the reason for accessing information-related data. In this scenario, ask the Kernel Team to generate a purpose for the new application or the existing one required by the Aura feature.
Ask the Kernel Team to generate the scopes required for the Aura feature, that define the access level to datasets (writing/reading permissions to/from Kernel datasets).

Take into account the following considerations:
- If a purpose is required, the scopes must be associated to it.
- There are global admin scopes that are always mandatory for every app in order to write/read datasets:
  - admin:datasets:read
  - data:read
  - data:write
- Additionally, each Aura feature requires its specific scopes.
- The version number is not needed in the scopes.

Guidelines for Kernel configuration in specific Aura features

List of published guidelines that include specific Kernel configuration:

Billing module operation

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Publish an API in Kernel

Guidelines for the publication of an API in Kernel

Guidelines

As a prerequisite for building an experience in ATRIA, the aura-gateway-api API must be published in Kernel.

For this purpose, follow the instructions below:

Request Kernel team to configure in the corresponding Kernel environment the scopes needed to call this API:
- aura-ai-services:messaging:write: Permission to send Generative / RAG and feedback messages to Aura.
- aura-ai-services:nlp-messaging:write: Permission to send NLP as a Service messages to Aura.
Access from Kernel to aura-gateway-api will be done by APIKey. It is necessary to create this APIKey in each environment following these instructions Generate an APIKey.
Request Kernel team to configure in the corresponding Kernel environment this API Aura AI Services.
- For that, the following settings are needed:
  - URL: https://{{aura-services-environment}}.auracognitive.com/aura-services/v2/aiservices/
  - Authorization header: APIKEY {{api-key}}
    
    Where:
    - {{aura-services-environment}} should look like svc-[country]-[environment], for instance svc-es-pre
    - {{api-key}} is a specific APIKey created for Kernel to access this endpoint. This APIKey must be requested to the team in charge of operating the corresponding Aura environment:
      - Aura Global team for development and staging environments
      - GES for certifications, pre-production and production environments
This process must be executed in each Kernel deployment.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Postman Collection - Video Contents Kernel API

Postman collection with sample requests to Video Contents Kernel API

Introduction

This Postman Collection is based in the Video Contents Kernel API.

It includes requests to get token and example requests of TV Search plugin.

Environment and collection files

Download environment and collection and import both files in Postman.

Import collection

To import the collection and environment, you can see the reference info about this process here.

Once the collections are imported, the environment’s values should be completed. Below, a short description about the enviroment’s variables is listed:

administrative_number_es: administrative number of Spanish user.
user_id_es: identifier of Spanish user.
channel_id_es: identifier of Spain channel authentication.
administrative_number_br: administrative number of Brazilian user.
user_id_br: identifier of Brazil user.
channel_id_br: identifier of Brazil channel authentication.
auth_url: authentication URL.
issuer_url: issuer URL.
expiration_token: time to expire the token in seconds.
scope: list of scopes separated with spaces.
purpose: list of purposes separated with spaces.
private_key: private key to get assertion token.
header_key_id: key id for headers to get assertion token.
client_id_es: client id to authenticate a Spanish user.
client_secret_es: client secret password to authenticate a Spanish user.
client_id_br: client id to authenticate a Brazilian user.
client_secret_br: client secret password to authenticate a Brazilian user.
authorization_id_es: authorization id for Spanish user.
authorization_id_br: authorization id for Brazilian user.

The majority of variables imported for this environment have assigned a default value, only the following variables must be completed: private_key, header_key_id, client_secret_es and client_secret_br.

The collection is composed by:

Get token ES request: to get the access token for Spanish users.
Get token BR request: to get the access token for Brazilian users.
ES folder: requests to search in the Spanish video content.
BR folder: requests to search in the Brazilian video content.

How to use this collection

First, select the environment imported, see the process here.

Then, to send a request in the folder ES or BR, it is previously required to send the request Get token ES or Get token BR respectively.

And finally, try to send any request included in the folder ES or BR.

⚠️ When the request response is UNAUTHENTICATED, send the request Get token ES or Get token BR according to each case and try again the previous request.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Kernel datasets upload handling

Guidelines for the enabling and disabling of Kernel datasets upload in non-productive environments.

Introduction

After the deployment of Aura in any environment, all its components will generate KPI entities files that will be uploaded into Kernel in CSV or Avro format, as datasets. These procedures increment the cost, both in Aura and in Kernel instances:

More consumption of Azure Storage
More time of execution of the Databricks cluster of Aura
Need for more storage in Kernel, both in Azure for the CSVs and for Avro datasets

Moreover, the data generated in these environments is almost never analyzed nor used.

Because of this, the proposal is to disable the uploading and to minimize the storage of these files, to minimize the costs, once the sanity test set was executed and the process has been validated.

If, eventually, there is a need to test the process again or to upload some data to validate algorithms or to use the Aura billing module, everything can be enabled again.

Prerequisites

A kubeconfig of the Aura environment must be configured.
az client installed in your PC.
Credentials to access the Azure subscription.
Substitute <YOUR-ENV> with the corresponding pre-production environment: es-pre, es-cert, br-pre, de-pre, de-int, etc.
The installation output file (output_install/<YOUR_ENV>_info.json) to get:
- The token and the URL of the Databricks cluster.
  - Substitute <DATABRICKS_TOKEN> with Databricks cluster token.
  - Substitute <DATABRICKS_URL> with the domain of the Databricks cluster URL.
- The job_id of the databricks job in charge of uploading the datasets to Kernel
  - Substitute <DATABRICKS_JOB_ID> with the job_id.
- The Azure Storage account name and the blob container where the KPI entities files are stored.
  - Substitute <AZURE_COMMON_STORAGE> with STORAGE_ACCOUNT_NAME and <KPI_BLOB_CONTAINER_NAME> with its value.

PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE=output_install/<YOUR_ENV>_info.json
STORAGE_ACCOUNT_NAME=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .common_azure_storage_account_name)
STORAGE_ACCESS_KEY=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .common_azure_storage_access_key)
KPI_BLOB_CONTAINER_NAME=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .kpi_blob_container_name)
DATABRICKS_JOB_ID=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .databricks.job_id)
DATABRICKS_URL=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .databricks.url)
DATABRICKS_TOKEN=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .databricks.token)

Disable data uploading

Disable aura-kpis-uploader CSV files upload

Suspend aura-kpi-uploader job:

kubectl -n <YOUR-ENV> patch cronjobs kpi-uploader -p '{"spec" : {"suspend" : true }}'

Disable aura-databricks-job Avro files upload

Pause aura-databricks-job job:
- Substitute <DATABRICKS_JOB_ID> with the DATABRICKS_JOB_ID obtained from the installation output file.

curl -XPOST --header 'Authorization: <DATABRICKS_TOKEN>' https://<DATABRICKS_URL>/api/2.1/jobs/update --data '{
   "job_id": <DATABRICKS_JOB_ID>,
   "new_settings":{
      "schedule":{
         "pause_status":"PAUSED"
      }
   }
}'

Remove old KPI entity files generated by Aura and ATRIA components

This step will be fulfilled by applying a removal policy on the Azure blob container where the components write the KPIs.

There are two ways of applying this change:

Apply the policy from Azure portal
Apply the policy using az client

Apply the policy from Azure portal

Access Azure portal
Look for <AZURE_COMMON_STORAGE> account and <KPI_BLOB_CONTAINER_NAME>
Apply management-policy to <KPI_BLOB_CONTAINER_NAME> and to <KPI_BLOB_CONTAINER_NAME>/processed

Apply the policy using az client

To execute this step, first log in to the Azure subscription with az login.

PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE=output_install/<YOUR_ENV>_info.json
STORAGE_ACCOUNT_NAME=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .common_azure_storage_account_name)
STORAGE_ACCESS_KEY=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .common_azure_storage_access_key)
RESOURCE_GROUP=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .common_resource_group)

az storage account management-policy show -g ${RESOURCE_GROUP} --account-name ${STORAGE_ACCOUNT_NAME} -o json > policy.json

KPI_BLOB_CONTAINER_NAME=$(cat ${PATH_TO_YOUR_OUTPUT_INSTALL_ENV_FILE}|jq -r .kpi_blob_container_name)

sed -i "s|${KPI_BLOB_CONTAINER_NAME}/proccesed||g" policy.json

az storage account management-policy create -g ${RESOURCE_GROUP} --account-name ${STORAGE_ACCOUNT_NAME} --policy policy.json

Enable data uploading

Execute Aura installer and everything will be reconfigured again, running the deploy_core stage.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Work with Kernel

Guidelines that include different tasks required for working with Kernel

Index

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Get Kernel access token for Aura Gateway API

Guidelines to get a Kernel access token for working with aura-gateway-api

Steps in the process

To use the Kernel aura-aiservice API, first authenticate with the client credentials specifying the required scopes, that depend on the specific ATRIA capability to be used:
- NLP as a Service: aura-ai-services:nlp-messaging:write
- Generative AI and RAG: aura-ai-services:messaging:write
Afterwards, refresh the token following Kernel instructions.
To obtain the real secret of your app, just run the following command, as an example of using the app “aura-bot” in Kernel “global-int-current” with a fake password.
```
$ kubectl -n $AURA_ENVIRONMENT get secret aura-bot -o json | jq -r ".data.AURA_FP_CLIENT_SECRET|@base64d"
```

Now you can request the access_token:

# generate a valid UUID as correlator
# substitute {{correlator}} with the generated UUID
export CORRELATOR={{correlator}}
# substitute aura-bot:secret with the specific information for your Kernel client.

$ curl -i -X POST -u aura-bot:secret -H 'Content-Type: application/x-www-form-urlencoded' -H 'Cache-Control: no-cache' -H 'x-correlator: $CORRELATOR' 'https://auth.global-int-current.baikalplatform.com/token' -d 'scope=aura-ai-services:messaging:write&grant_type=client_credentials'

HTTP/2 200
{"access_token":"<token>","token_type":"Bearer","expires_in":3599,"scope":"aura-ai-services:messaging:write","purpose":""}

This token expires after a certain time, so it is required to repeat the steps above to obtain a new one.

Access here to more information about Kernel authentication.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Kernel configuration for URM Global script

Guidelines for the configuration of the script URM Global in Kernel

Introduction

Aura NLP dictionaries can now be generated and configured using the URM Global script build_local_catalogs_etl.sh.

In order to have a correct behavior for the URM data extraction used in the global script, it is necessary to execute the tasks defined in the following sections, that are a particularization of the general guidelines “Kernel configuration: General steps”.

1. Check APIs publication in Kernel

Check that the directsql:query API is published in Kernel: List of available APIS on Telefónica Kernel.
If not, follow the guidelines in the document Publish an API in Kernel.

2. Check datasets publication in Kernel

Check that the required datasets for the configuration of the URM Global script are published in Kernel: List of available datasets on Telefónica Kernel.
- D_Gbl_Video_Content_Type
- Video_Content
- D_Gbl_Video_Staff_Role
- Video_Content_Staff_Rel
- D_Video_Staff_Role
- D_Video_Staff

3. Create a Kernel application

A Kernel application with the name aura-cognitive-trainings must be created and configured with specific scopes.

Ask the Kernel Team to create the new application in Kernel: "id": "aura-cognitive-trainings"

Once the app is created, two parameters will be provided for securely accessing: - client_id: unique identifier of the consuming app acting as Kernel API client. - client_secret: password.

4. Assign purpose/scopes to the application

No purpose is required, as datasets do not include personal information.
Ask the Kernel Team to assign the following scopes to the application:
- data:Video_Content:read
- data:Video_Content_Staff_Rel:read
- data:D_Video_Staff:read
- data:D_Video_Content_Category:read
- data:D_Video_Staff_Role:read
- data:D_Video_Age_Rating:read
- data:D_Gbl_Video_Content_Category:read
- data:D_Gbl_Video_Staff_Role:read
- data:D_Gbl_Video_Content_Type:read
- data:D_Gbl_Video_Age_Rating:read
- directsql:query

5. Add other required fields

Provide the Kernel Team with other necessary fields, as shown in the code snippet.

The final file for the configuration of the application, including all the above-mentioned parameters, is shown below:

{
  "name": "Data consumption for Aura Cognitive Trainings",
  "grant_types": [
    {
        "authentication": "client_credentials"
        "scopes": [
               "data:Video_Content:read",
               "data:Video_Content_Staff_Rel:read",
               "data:D_Video_Staff:read",
               "data:D_Video_Content_Category:read",
               "data:D_Video_Staff_Role:read",
               "data:D_Video_Age_Rating:read",
               "data:D_Gbl_Video_Content_Category:read",
               "data:D_Gbl_Video_Staff_Role:read",
               "data:D_Gbl_Video_Content_Type:read",
               "data:D_Gbl_Video_Age_Rating:read",
               "directsql:query"
             ]
        "purposes": []
        "api": "directsql:query"
    }
  ],
  "description": "Aura cognitive application to consumption data of the kernel",
  "raw_dataset_read": true,
  "tags": {},
  "encrypt_access_tokens": true,
  "id": "aura-cognitive-trainings",
  "requires_authorization_id": true,
  "client_type": "CONFIDENTIAL",
  "legal_entity_id": "telefonica",
  "redirect_uris": []
}