Aura – aura-file-manager

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura file manager processes flowchart

Sequential flowchart for Aura file manager processes and API functionalities

Initialization sequence

sequenceDiagram
    participant A as Orchestrator
    participant B as HttpMonkeyPatcher
    participant C as AuraChannelsConfiguration
    participant D as AuraMongoDb.Connector
    participant E as FileQueueDao
    participant F as FileService
    participant G as FileManagerServer
    participant H as ClientOAuthToken4P
    participant I as StorageFileManager
    participant J as 4P Whatsapp API
    participant K as Aura Global Azure storage
    participant L as QueueManager
    participant M as QueueAgent
    A ->>+ B: init
    B -->>- A: instance
    A ->>+ C: init
    C ->>+ K: loadChannelsConfiguration
    K -->>- C: Channels configuration
    C -->>- A: instance
    A ->>+ D: init
    D -->>- A: instance
    A ->>+ E: init
    E -->>- A: instance
    A ->>+ F: init
    F ->>+ H: init
    loop 4P oauth tokens
        H ->>+ J: grantClient
        J -->>- H: TokenSet
    end
    H -->>- F :initialized
    F ->>+ I: new
    I -->>- F: instance
    F -->>- A: instance
    A ->>+ G: init
    G -->>- A: instance
    A ->>+ L: init
    L -->>- A: instance
    A ->>+ M: init
    M -->>- A: instance

Sequence explanation

HttpMonkeyPatcher utility is loaded, mostly to monitor HTTP traffic through Prometheus.
AuraChannelsConfiguration is loaded, which retrieves channels configuration from Aura Global Azure storage. Aura file manager needs it in order to retrieve validation configuration for the selected channel.
AuraMongoDb.Connector and FileQueueDao are loaded to access MongoDB.
FileService initializes other different services.

ClientOauthToken4P: It connects to Kernel WhatsApp API and retrieves tokens for different channels (WhatsApp).
StorageFileManager: It initializes and configures the connection against Aura Global Azure storage.

Queue components are started: QueueManager and QueueAgent, in charge of message management.
FileManagerServer is started, which exposes aura-file-manager API.

API functionalities

In this section, aura-file-manager functionalities are documented using sequence diagrams.

Detailed information about these endpoints can be found in the aura-file-manager API definition.

Upload and validate files

GET /aura-services/v1/files

Upload and validate operation. It allows to upload and validate a file based on previously defined validations.

sequenceDiagram
    actor User as Channel user
    participant K as Groot
    participant A as Bot (FileManagerRecognizer)
    participant B as Bridge (AsyncCallback plugin)
    participant C as FileManagerServer
    participant D as FileController
    participant E as FileService
    participant I as FileQueueDao
    participant J as QueueManager
    participant F as ClientOauthToken4P
    participant G as Aura Global Azure storage
    participant H as FileValidator

    User->>+K: Unprocessed attachment
    K->>-User: 204
    K->>+A: Unprocessed attachment
    A-->>-K: 204
    A->>+C: POST /files
    C-->>-A: 204
    C->>+D: uploadAndValidateFiles
    D->>+E: storeFiles
    E->>+I: insertIncomingFileRequest
    I-->>-E: Ok
    E->>+J: addTask
    J-->>-E: Ok
    D-->>-C: 204    
        alt Have queue space?
        J->>+E: processFile
        loop multiple attachments
            E->>+F: getTokenByClientId
            F-->>-E: Token
            E->>+G: uploadFile
            G-->>-E: Upload result
            E->>+G: getFileContents
            G-->>-E: File contents
            E->>+H: validateFile
            H-->>-E: Validation
            E->>+G: getSasUrl
            G-->>-E: Url
            E-->>-E: Attachment data
        end
        D-->>C: Attachments data
        C-->>B: Attachments data
        B-->>K: Attachments data
        K-->>A: Attachments data
        A-->>K: Attachments data
        K-->>User: Attachments data
    end

Actors

Channel user: User that is interacting with the dialog.
Groot: aura-groot main bot over distributed architecture .
Bot: aura-bot responsible of handling the attachments processing. It is done through the aura-file-manager recognizer.
Bridge: aura-bridge responsible of handling attachments once processed and sending data back to aura-bot.
FileManager microservice: Microservice responsible of processing attachments.
- FileManagerServer: Server responsible of handling request to the microservice.
- File: Controller that handles requests of attachments processing.
- FileService: Service where the attachments files are validated and uploaded to Aura Global Azure storage.
- ClientOauthToken4P: Small utility to retrieve and maintain authorization Kernel tokens.
- FileQueueDao: Decentralized autonomous organization (DAO) to manage BD operations.
- QueueManager: Component in charge of managing queue messages.
Aura Global Azure storage: Repository where the files are going to be stored temporarily. By default, they are accessible for one hour.
aura-file-validator utility: Utility responsible for validating files content.

Sequence explanation

A channel user has uploaded a file/s as requested by a dialog that supports attachments.
When the attachments are detected by aura-bot (through the aura-file-manager recognizer), the attachments are sent to aura-file-manager microservice to be processed.
Once the attachments are received by Aura file manager, basic checks are done synchronously (all parameters needed are present and have correct format), the request added in queue is stored and a ACK is sent to the aura-file-manager recognizer.
After sending the ACK, the real attachments processing starts asynchronously.
If queue is not full, the attachments begin to process.
The direct link is sent directly to Aura Global Azure storage, delegating on it the responsibility of its uploading.
If the repository uploads the file correctly, then aura-file-manager downloads it and, through the use of the aura-file-validator utility, validates its content.
Once the file is validated, a direct download link is retrieved from the repository, and the data is sent to aura_bridge asyncCallback endpoint.
aura-bridge translates the message and sends it to aura-groot.
aura-groot translates the message and sends it to aura-bot, where aura-file-manager recognizer will forward it to the originating dialog.
If queue is full, when one aura-file-manager pod has space enough, the attachments will be processed by QueueAgent.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura file manager environment variables

List of environment variables used by aura-file-manager

Properties marked in bold are mandatory.
Properties marked in italics are optional.

Property	Type	Description	Modifiable by OB
AURA_4P_TOKEN_DELAY_RETRY	number	When retrieving a Kernel token, if an error occurs, delay time in milliseconds between retries. Default value: `100`.	NO
AURA_AUTHORIZATION_HEADER	string	Complete authorization header to be sent to aura_bridge, with the following format: `APIKEY xxxxxx`.	YES, but only if the previous APIKey was deprecated.
AURA_BRIDGE_ASYNC_CALLBACK_PATH	string	Path of async-callback. By default: `aura-services/v1/async-callback/notifications`.	NO
AURA_BRIDGE_APIKEY	string	APIKey needed to call aura-bridge.	NO
AURA_BRIDGE_CALLBACK_RETRIES	number	Number of retries made by aura-bridge in case of error in an HTTP request. By default: `3`.	NO, only if checked and validated with Aura Global Team.
AURA_BRIDGE_CALLBACK_RETRY_DELAY	number	Delay between retries in case of error. By default: `100`.	NO, only if checked and validated with Aura Global Team.
AURA_BRIDGE_CALLBACK_RETRY_FACTOR	number	Factor to multiply delay for every HTTP request retried. By default: `10`.	NO, only if checked and validated with Aura Global Team.
AURA_BRIDGE_ENDPOINT	string	Endpoint for aura-bridge.	NO
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT	string	Endpoint for aura-configuration-api.	NO
AURA_ENCRYPTION_ALGORITHM	string	Encryption algorithm that will be used to encrypt async-callback parameters. By default: `aes-256-cbc`	NO. It would break APIKey validation.
AURA_ENCRYPTION_IV_LENGTH	number	Size for the initialization vector used by the encryption algorithm. By default: `16`	NO. It would break APIKey validation.
AURA_ENCRYPTION_IV_POSITION	number	Position where to insert the initialization vector in the final string with the encrypted payload. By default: `35`.	NO. It would break APIKey validation.
AURA_ENCRYPTION_KEY	string	Encryption key or comma-separated list of encryption keys to be used in the environment. It is mainly used to encrypt sensible data, such as async-callback parameters.	NO. It would break database encrypted data and APIKey validation.
AURA_ENVIRONMENT_NAME	string	Name of the environment where the aura-authentication-api is deployed. For example: `ap-next`, `es-dev` or `de-pre`.	NO
AURA_ENVIRONMENT_PREFIX	string	Prefix that will be used by all Redis keys when using redis-connector. This allows mixing in a single Redis service messages coming from different environments in the same Azure subscription. By default: ``. (empty)	YES
AURA_FILE_MANAGER_SERVER_PORT	number	Port where the API is listening. By default, `8015`	NO
AURA_FILE_MANAGER_TYPES	string[]	Array with the valid default file types, it can be overriden by the channel configuration. Default value: `['pdf', 'docx', 'pptx', 'xlsx', 'aac', 'm4a', 'amr', 'mp3', 'ogg', 'mp4', '3gpp', 'jpg', 'jpeg', 'png']`	NO, it must be done through channel config.
AURA_FILE_MANAGER_MONGODB_COLLECTION_FILES_QUEUE	string	Collection where the files objects will be stored. By default: `files-queue`.	NO
AURA_FILE_MANAGER_MONGODB_DATABASE_NAME	string	Database name where will be included all the aura-file-manager collections. By default: `aura-file-manager`.	NO
AURA_FILE_MANAGER_MONGODB_MAX_RETRIES_QUEUE	number	Number of file processing attempts. Not to be confused with message sending retries. By default: `3`.	NO
AURA_FILE_MANAGER_MONGODB_MAX_PROCESSING_TIME	number	Maximum time in milliseconds that the file is in the `PROCESSING` state before trying to process it again. By default: `300000` milliseconds.	NO
AURA_FP_AUTHSERVER_ENDPOINT	string	URL of the Kernel authentication server. It MUST end with `/`	NO
AURA_HTTP_KEEP_ALIVE	boolean	Flag to indicate whether or not the outgoing HTTP connections should be established with keep alive or not. By default, `true`.	NO
AURA_HTTP_KEEP_ALIVE_MSECS	number	Number of milliseconds to keep HTTP connections open. By default, `100000`.	NO
AURA_HTTP_KEEP_MAX_SOCKETS	number	Number of sockets to keep HTTP connections open. By default, `200`.	NO
AURA_HTTP_MAX_REQUEST_SIZE	string	Maximum size in bytes of body request. Allowed values must indicate the units: 10 mb, 200 kb, etc. By default, `50mb`.	NO
AURA_HTTP_PATHS_LOG_DISABLED	string	Comma-separated string with the outgoing path that should not be logged. By default, `.well-known/openid-configuration`, `aura-configuration`, `metrics,` `healthz`. The paths in this variable will be added to the default ones.	YES, add the values to be filtered as a comma-separated list of paths.
AURA_INTERNAL_RETRIES	number	Number of retries for internal calls. By default, `1`	NO, except if requested
AURA_LOGGING_FORMAT	string	Format to be used in monitoring logs: `json` or `dev`, the latter a more visual format. By default: `json`.	NO. Only for development, set it to `dev`.
AURA_LOGGING_LEVEL	string	Level to be used in monitoring logs, from more to less verbose: `'TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL', 'OFF'`. By default: `INFO`.	YES, for development set it to `DEBUG`. In pre/production should be `INFO` or `ERROR`. For analysis of an issue in pre/production it may be changed to `DEBUG`.
AURA_MAKEUP_MODE	string	It allows dev mode of the make-up with the value `local`. By default: `full`.	NO, only for development, set it to `local`.
AURA_MAKEUP_MONGODB_INDEX_FILE	string	Name of the file that contains the definition of Mongo indexes. By default: `aura-file-manager-mongodb-indexes.json`	NO
AURA_MAX_FILE_SIZE_BYTES	number	Maximum file size in bytes by default. It can be overridden by channel configuration. Default value: `16384000`.	NO, it must be done through channel config.
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY	string	Microsoft Storage password of the deployment.	NO. Only if operations team changes it.
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT	string	Microsoft Storage account of the environment.	NO. Only if operations team changes it.
AURA_MICROSOFT_AZURE_STORAGE_CONFIGURATION_CONTAINER	string	Name of the container where basic aura-bot configuration files are stored: Mongo indexes descriptor, behavior manager, etc. By default, `aura-configuration`.	NO
AURA_MICROSOFT_AZURE_STORAGE_SAS_URL_VALIDITY	number	Number of milliseconds of validity for generated SAS tokens. By default: `3600000`	YES, if Operations Team prefers another duration time.
AURA_MICROSOFT_AZURE_STORAGE_STATIC_CONTAINER_NAME	string	Name of Azure blob container to be used as repository.	NO. Only if Operations Team changes it.
AURA_MIN_FILE_SIZE_BYTES	number	Minimum file size in bytes by default. It can be overridden by channel configuration. Default value: `256`.	NO, it must be done through channel config.
AURA_MONGODB_PASSWORD	string	MongoDB password to access `AURA_MONGODB_BOT_DATABASE` in `AURA_MONGODB_URI`.	NO
AURA_MONGODB_POOL_SIZE	number	Number of connections available in MongoDB pool. By default, `60`.	NO
AURA_MONGODB_SSL	boolean	`true` if access to `AURA_MONGODB_URI` is `SSL`. By default, `false`.	NO
AURA_MONGODB_SUFFIX	string	Suffix of Mongo database name. Its value at run-time will be the value from AURA_ENVIRONMENT_NAME-AURA_VERSION.replace(/./g,’-’)`.	NO
AURA_MONGODB_URI	string	MongoDB URI of the server handling `AURA_MONGODB_BOT_DATABASE`.	NO
AURA_MONGODB_USERNAME	string	MongoDB username to access `AURA_MONGODB_BOT_DATABASE` in `AURA_MONGODB_URI`.	NO
AURA_QUEUE_MANAGER_CHECK_QUEUE_INTERVAL_TTL	number	In milliseconds. The time interval used by the queue manager to check if the queue system has pending messages. By default, `1000`.	NO
AURA_QUEUE_MANAGER_CONCURRENT_TASK	number	Number of concurrently processed tasks. By default, `5`.	NO
AURA_QUEUE_MANAGER_INTERVAL	number	In milliseconds. Time interval for processing messages in the queue. By default: `200`..	NO
AURA_QUEUE_MANAGER_MAX_ELEMENTS	number	Maximum number of elements in the queue. By default: `4`.	NO
AURA_QUEUE_MANAGER_AGENT_INTERVAL	number	Time interval in milliseconds for the queue agent to check if the queue system has pending messages. By default: `20000`.	NO
AURA_QUEUE_TYPE	string	Type of queue system to be used (MONGO/REDIS). By default: `REDIS`.	NO
AURA_REDIS_COMPONENT_CONFIG_PREFIX	string	Prefix for Redis configuration keys. By default: `aura-config:`.	NO
AURA_REDIS_MODE	string	Mode of Redis distribution. Values: `CLUSTER`, `SENTINEL`, `SINGLE`. By default: `SENTINEL`.	NO
AURA_REDIS_PREFIX	string	Redis prefix.	NO
AURA_REDIS_SENTINEL_INSTANCE_NAME	string	Name of the Redis instance. Used in `SENTINEL` mode.	NO
AURA_REDIS_HOSTS	string	String with a list of nodes separated by ‘,’, including host and port separated by ‘:’. For example: “localhost:port,localhost2:port2”.	NO
AURA_REDIS_DATABASE	number	Database number for `SINGLE` or `SENTINEL` mode. By default: `0`.	YES
AURA_REDIS_PASSWORD	string	String with Redis password.	YES
AURA_REDIS_USE_CONNECTION_POOL	boolean	Use pool connections for Redis. By default: `true`.	YES
AURA_REDIS_CONNECTION_POOL_MIN	number	Minimum number of connections in the pool. By default: `2`.	YES
AURA_REDIS_CONNECTION_POOL_MAX	number	Maximum number of connections in the pool. By default: `100`.	YES
AURA_REDIS_MAX_RECONNECT_RETRIES	number	Number of retries to connect to Redis. By default: `25`	YES
AURA_REDIS_MAX_RECONNECT_INTERVAL	number	Time in milliseconds to wait before reconnecting to Redis. By default: `5000`.	YES
AURA_REDIS_QUEUE_PREFIX	number	Prefix for Redis keys used by the queue system. By default: `file_manager:queue:`.	YES
AURA_REDIS_QUEUE_SHARD_COUNT	number	Number of shards to generate lists for store unprocessed context. By default: `5`	YES
AURA_SHUTDOWN_GRACEFUL_TTL	string	Number of milliseconds to wait until all authentication-api tasks are done before shutting each pod down. By default `25000` (25 seconds)	NO
AURA_VERSION	string	Number of the Aura release being executed.	NO
AURA_KPIS_BLOB_TIME_WAIT_IF_EXISTS	number	Time to wait in milliseconds if the KPIs blob exists to avoid duplicate headers. By default: `1000`.	YES

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura file manager

aura-file-manager is the microservice in charge of uploading a file to a temporary repository, validating it and returning a link as validation result.
Find in the current documents the description of this component, its architecture, components and processes.

Aura Virtual Assistant component

Introduction

aura-file-manager is a microservice based on a REST API with the capability of uploading a file to a temporary repository, validating it, and returning a direct download link as validation result.

aura-file-manager uses an utility, aura-file-validator for the validation of files.

Find detailed information regarding Aura file manager in the following documents:
. Architecture and main components
. Communication protocol
. Executed process flowchart and API functionalities
. Environment variables
. API definition

⚠️ In order to use aura-file-manager, it is necessary to activate it previously. Learn how in Aura File Manager activation.

Aura file manager architecture

In the following diagram, the architecture and main components of aura-file-manager are shown:

Internal components

ConfigurationManager

ConfigurationManager is a handler for configuration, obtained through a configuration file or environment variables.

FileService

FileService is where the main functionality of the microservice resides. It is where file upload and validation is done.

HTTP server

Microservice is implemented as an HTTP server (FileManagerServer) that exposes an API to receive the files to be validated.

Middlewares

The route published in the API definition file is handled by a controller but, before a request lands on its controller, it goes through a series of middlewares that provides some common steps needed by all the controllers of the server such as: request validation, common parameters extraction, logging, metrics initialization, etc.

Queue messages processing

Queuing system that allows tolerance to server failures, storing the received files in a DB and processing them later.

Communication protocol

aura-file-manager communication protocol is completely asynchronous, this means that the answer of a request is not included in the HTTP response related to the request, but it is sent as a new HTTP request to the callback aura-bridge.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura file manager API definition

Description of aura-file-manager API, microservice responsible for handling attached files storage and validations at channel level

Download swagger file

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura File Manager metrics

List of metrics available in Aura File Manager

http_request_duration_seconds

This metric is intended to store the information related to all the incoming HTTP requests received by aura-file-manager.

It is stored as a Summary in Prometheus. So every sample, besides the defined labels, also includes its duration.

It measures the duration since the request lands in aura-file-manager until its HTTP response is returned, indicating to the client that Aura is processing the request to obtain a proper answer.

The metric allows measuring the behavior of the requests from any given endpoint:

The number of requests during a time
The average/min/max duration of these requests

Labels:

method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
host: host and domain where the request is being sent
path: specific endpoint of the request
status_code: HTTP status code returned in the response

outgoing_request_duration_seconds

This metric is intended to store the processing time related to all the outgoing HTTP requests made by aura-file-manager.

It is stored as a Summary in Prometheus so every sample, besides the defined labels, also includes its duration.

This metric allows measuring the behavior of the requests to any given endpoint:

The number of requests during a time
The average/min/max duration of these requests

Labels:

method: HTTP method used by the request being stored (GET, POST, PUT, DELETE, etc.)
host: host and domain where the request is being sent
path: specific endpoint of the request
status: HTTP status code returned in the response

outgoing_message_duration_seconds

This metric is intended to store the processing time of aura-bot requests arriving to aura-file-manager.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-file-manager is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the user is sent back to the client callback. This metric measures the duration since the request lands in aura-file-manager until the last message of its answer is sent to the client callback.

Labels:

path: specific endpoint of the request.
httpStatus: HTTP status code returned in the response.
origin: aura-bot

incoming_message_duration_seconds

This metric is intended to store the processing time of aura-bot requests arriving to aura-file-manager.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

As aura-file-manage is an asynchronous server, the processing of a request does not end when the HTTP response is returned, but when the proper answer for the channel or skill is sent back to the client callback. This metric measures the duration from when the request arrives at aura-file-manager until it is processed to send the response.

Labels:

path: specific endpoint of the request.
httpStatus: HTTP status code returned in the response.
originStatus: status sent in the body of the response in the happening of an error.
origin: aura-bot

aura_component_version

This metric is intended to store the number of aura-file-manager instances (pods) running each version of the code. It is stored as a Gauge in Prometheus.

Labels:

version: version field in the package.json file included in the running docker container.
component: name of the component that is writing the metric.

aura_server_unhandled_error

This metric is intended to store the number of unhandled errors happening in aura-file-manager.

It is stored as a Counter in Prometheus.

Labels:

error: exception message that forced the unhandled error.

aura_token_generate

This metric is intended to store the processing time of aura-file-manger to get/refresh kernel token.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

Labels:

path: specific endpoint of the request.
httpStatus: HTTP status code returned in the response.
originStatus: status sent by Direct Line in the body of the response in the happening of an error.
origin: kernel client identifier

file_validation_duration_seconds

This metric is intended to store the validation time of a file.

It is stored as a Summary in Prometheus, so every sample, besides the defined labels, also includes its duration.

Labels:

path: specific endpoint of the request.
code: OK when file is valid.
origin: specific endpoint of the request.

services_status

This metric is intended to store the number of success or errored checks of modules of the server. It is stored as a Counter in Prometheus.

Labels:

moduleId: Id of the module.
status: OK or ERROR