Aura file manager

• 1: Processes
• 2: Environment variables
• 3: API definition

1 - Processes

Aura file manager processes flowchart

Initialization sequence

sequenceDiagram
    participant A as Orchestrator
    participant B as HttpMonkeyPatcher
    participant C as AuraChannelsConfiguration
    participant D as AuraMongoDb.Connector
    participant E as FileQueueDao
    participant F as FileService
    participant G as FileManagerServer
    participant H as ClientOAuthToken4P
    participant I as StorageFileManager
    participant J as 4P Whatsapp API
    participant K as Aura Global Azure storage
    participant L as QueueManager
    participant M as QueueAgent
    A ->>+ B: init
    B -->>- A: instance
    A ->>+ C: init
    C ->>+ K: loadChannelsConfiguration
    K -->>- C: Channels configuration
    C -->>- A: instance
    A ->>+ D: init
    D -->>- A: instance
    A ->>+ E: init
    E -->>- A: instance
    A ->>+ F: init
    F ->>+ H: init
    loop 4P oauth tokens
        H ->>+ J: grantClient
        J -->>- H: TokenSet
    end
    H -->>- F :initialized
    F ->>+ I: new
    I -->>- F: instance
    F -->>- A: instance
    A ->>+ G: init
    G -->>- A: instance
    A ->>+ L: init
    L -->>- A: instance
    A ->>+ M: init
    M -->>- A: instance

Sequence explanation

1. HttpMonkeyPatcher utility is loaded, mostly to monitor HTTP traffic through Prometheus.
2. AuraChannelsConfiguration is loaded, which retrieves channels configuration from Aura Global Azure storage. Aura file manager needs it in order to retrieve validation configuration for the selected channel.
3. AuraMongoDb.Connector and FileQueueDao are loaded to access MongoDB.
4. FileService initializes other different services.

• ClientOauthToken4P: It connects to Kernel WhatsApp API and retrieves tokens for different channels (WhatsApp).
• StorageFileManager: It initializes and configures the connection against Aura Global Azure storage.

3. Queue components are started: QueueManager and QueueAgent, in charge of message management.
4. FileManagerServer is started, which exposes aura-file-manager API.

API functionalities

In this section, aura-file-manager functionalities are documented using sequence diagrams.

Detailed information about these endpoints can be found in the aura-file-manager API definition.

Upload and validate files

• GET /aura-services/v1/files

Upload and validate operation. It allows to upload and validate a file based on previously defined validations.

sequenceDiagram
    actor User as Channel user
    participant K as Groot
    participant A as Bot (FileManagerRecognizer)
    participant B as Bridge (AsyncCallback plugin)
    participant C as FileManagerServer
    participant D as FileController
    participant E as FileService
    participant I as FileQueueDao
    participant J as QueueManager
    participant F as ClientOauthToken4P
    participant G as Aura Global Azure storage
    participant H as FileValidator

    User->>+K: Unprocessed attachment
    K->>-User: 204
    K->>+A: Unprocessed attachment
    A-->>-K: 204
    A->>+C: POST /files
    C-->>-A: 204
    C->>+D: uploadAndValidateFiles
    D->>+E: storeFiles
    E->>+I: insertIncomingFileRequest
    I-->>-E: Ok
    E->>+J: addTask
    J-->>-E: Ok
    D-->>-C: 204    
        alt Have queue space?
        J->>+E: processFile
        loop multiple attachments
            E->>+F: getTokenByClientId
            F-->>-E: Token
            E->>+G: uploadFile
            G-->>-E: Upload result
            E->>+G: getFileContents
            G-->>-E: File contents
            E->>+H: validateFile
            H-->>-E: Validation
            E->>+G: getSasUrl
            G-->>-E: Url
            E-->>-E: Attachment data
        end
        D-->>C: Attachments data
        C-->>B: Attachments data
        B-->>K: Attachments data
        K-->>A: Attachments data
        A-->>K: Attachments data
        K-->>User: Attachments data
    end

Actors

• Channel user: User that is interacting with the dialog.
• Groot: aura-groot main bot over distributed architecture .
• Bot: aura-bot responsible of handling the attachments processing. It is done through the aura-file-manager recognizer.
• Bridge: aura-bridge responsible of handling attachments once processed and sending data back to aura-bot.
• FileManager microservice: Microservice responsible of processing attachments.
  • FileManagerServer: Server responsible of handling request to the microservice.
  • File: Controller that handles requests of attachments processing.
  • FileService: Service where the attachments files are validated and uploaded to Aura Global Azure storage.
  • ClientOauthToken4P: Small utility to retrieve and maintain authorization Kernel tokens.
  • FileQueueDao: Decentralized autonomous organization (DAO) to manage BD operations.
  • QueueManager: Component in charge of managing queue messages.
• Aura Global Azure storage: Repository where the files are going to be stored temporarily. By default, they are accessible for one hour.
• aura-file-validator utility: Utility responsible for validating files content.

Sequence explanation

• A channel user has uploaded a file/s as requested by a dialog that supports attachments.
• When the attachments are detected by aura-bot (through the aura-file-manager recognizer), the attachments are sent to aura-file-manager microservice to be processed.
• Once the attachments are received by Aura file manager, basic checks are done synchronously (all parameters needed are present and have correct format), the request added in queue is stored and a ACK is sent to the aura-file-manager recognizer.
• After sending the ACK, the real attachments processing starts asynchronously.
• If queue is not full, the attachments begin to process.
• The direct link is sent directly to Aura Global Azure storage, delegating on it the responsibility of its uploading.
• If the repository uploads the file correctly, then aura-file-manager downloads it and, through the use of the aura-file-validator utility, validates its content.
• Once the file is validated, a direct download link is retrieved from the repository, and the data is sent to aura_bridge asyncCallback endpoint.
• aura-bridge translates the message and sends it to aura-groot.
• aura-groot translates the message and sends it to aura-bot, where aura-file-manager recognizer will forward it to the originating dialog.
• If queue is full, when one aura-file-manager pod has space enough, the attachments will be processed by QueueAgent.

2 - Environment variables

Aura file manager environment variables

Property	Type	Description	Modifiable by OB
AURA_4P_TOKEN_DELAY_RETRY	number	When retrieving a Kernel token, if an error occurs, delay time in milliseconds between retries. Default value: 100.	NO
AURA_AUTHORIZATION_HEADER	string	Complete authorization header to be sent to aura_bridge, with the following format: APIKEY xxxxxx.	YES, but only if the previous APIKey was deprecated.
AURA_BRIDGE_ASYNC_CALLBACK_PATH	string	Path of async-callback. By default: aura-services/v1/async-callback/notifications.	NO
AURA_BRIDGE_APIKEY	string	APIKey needed to call aura-bridge.	NO
AURA_BRIDGE_CALLBACK_RETRIES	number	Number of retries made by aura-bridge in case of error in an HTTP request. By default: 3.	NO, only if checked and validated with Aura Global Team.
AURA_BRIDGE_CALLBACK_RETRY_DELAY	number	Delay between retries in case of error. By default: 100.	NO, only if checked and validated with Aura Global Team.
AURA_BRIDGE_CALLBACK_RETRY_FACTOR	number	Factor to multiply delay for every HTTP request retried. By default: 10.	NO, only if checked and validated with Aura Global Team.
AURA_BRIDGE_ENDPOINT	string	Endpoint for aura-bridge.	NO
AURA_CHANNELS_CONFIGURATION_API_ENDPOINT	string	Endpoint for aura-configuration-api.	NO
AURA_ENCRYPTION_ALGORITHM	string	Encryption algorithm that will be used to encrypt async-callback parameters. By default: aes-256-cbc	NO. It would break APIKey validation.
AURA_ENCRYPTION_IV_LENGTH	number	Size for the initialization vector used by the encryption algorithm. By default: 16	NO. It would break APIKey validation.
AURA_ENCRYPTION_IV_POSITION	number	Position where to insert the initialization vector in the final string with the encrypted payload. By default: 35.	NO. It would break APIKey validation.
AURA_ENCRYPTION_KEY	string	Encryption key or comma-separated list of encryption keys to be used in the environment. It is mainly used to encrypt sensible data, such as async-callback parameters.	NO. It would break database encrypted data and APIKey validation.
AURA_ENVIRONMENT_NAME	string	Name of the environment where the aura-authentication-api is deployed. For example: ap-next, es-dev or de-pre.	NO
AURA_ENVIRONMENT_PREFIX	string	Prefix that will be used by all Redis keys when using redis-connector. This allows mixing in a single Redis service messages coming from different environments in the same Azure subscription. By default: ``. (empty)	YES
AURA_FILE_MANAGER_SERVER_PORT	number	Port where the API is listening. By default, 8015	NO
AURA_FILE_MANAGER_TYPES	string[]	Array with the valid default file types, it can be overriden by the channel configuration. Default value: ['pdf', 'docx', 'pptx', 'xlsx', 'aac', 'm4a', 'amr', 'mp3', 'ogg', 'mp4', '3gpp', 'jpg', 'jpeg', 'png']	NO, it must be done through channel config.
AURA_FILE_MANAGER_MONGODB_COLLECTION_FILES_QUEUE	string	Collection where the files objects will be stored. By default: files-queue.	NO
AURA_FILE_MANAGER_MONGODB_DATABASE_NAME	string	Database name where will be included all the aura-file-manager collections. By default: aura-file-manager.	NO
AURA_FILE_MANAGER_MONGODB_MAX_RETRIES_QUEUE	number	Number of file processing attempts. Not to be confused with message sending retries. By default: 3.	NO
AURA_FILE_MANAGER_MONGODB_MAX_PROCESSING_TIME	number	Maximum time in milliseconds that the file is in the PROCESSING state before trying to process it again. By default: 300000 milliseconds.	NO
AURA_FP_AUTHSERVER_ENDPOINT	string	URL of the Kernel authentication server. It MUST end with /	NO
AURA_HTTP_KEEP_ALIVE	boolean	Flag to indicate whether or not the outgoing HTTP connections should be established with keep alive or not. By default, true.	NO
AURA_HTTP_KEEP_ALIVE_MSECS	number	Number of milliseconds to keep HTTP connections open. By default, 100000.	NO
AURA_HTTP_KEEP_MAX_SOCKETS	number	Number of sockets to keep HTTP connections open. By default, 200.	NO
AURA_HTTP_MAX_REQUEST_SIZE	string	Maximum size in bytes of body request. Allowed values must indicate the units: 10 mb, 200 kb, etc. By default, 50mb.	NO
AURA_HTTP_PATHS_LOG_DISABLED	string	Comma-separated string with the outgoing path that should not be logged. By default, .well-known/openid-configuration, aura-configuration, metrics, healthz. The paths in this variable will be added to the default ones.	YES, add the values to be filtered as a comma-separated list of paths.
AURA_INTERNAL_RETRIES	number	Number of retries for internal calls. By default, 1	NO, except if requested
AURA_LOGGING_FORMAT	string	Format to be used in monitoring logs: json or dev, the latter a more visual format. By default: json.	NO. Only for development, set it to dev.
AURA_LOGGING_LEVEL	string	Level to be used in monitoring logs, from more to less verbose: 'TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL', 'OFF'. By default: INFO.	YES, for development set it to DEBUG. In pre/production should be INFO or ERROR. For analysis of an issue in pre/production it may be changed to DEBUG.
AURA_MAKEUP_MODE	string	It allows dev mode of the make-up with the value local. By default: full.	NO, only for development, set it to local.
AURA_MAKEUP_MONGODB_INDEX_FILE	string	Name of the file that contains the definition of Mongo indexes. By default: aura-file-manager-mongodb-indexes.json	NO
AURA_MAX_FILE_SIZE_BYTES	number	Maximum file size in bytes by default. It can be overridden by channel configuration. Default value: 16384000.	NO, it must be done through channel config.
AURA_MICROSOFT_AZURE_STORAGE_ACCESS_KEY	string	Microsoft Storage password of the deployment.	NO. Only if operations team changes it.
AURA_MICROSOFT_AZURE_STORAGE_ACCOUNT	string	Microsoft Storage account of the environment.	NO. Only if operations team changes it.
AURA_MICROSOFT_AZURE_STORAGE_CONFIGURATION_CONTAINER	string	Name of the container where basic aura-bot configuration files are stored: Mongo indexes descriptor, behavior manager, etc. By default, aura-configuration.	NO
AURA_MICROSOFT_AZURE_STORAGE_SAS_URL_VALIDITY	number	Number of milliseconds of validity for generated SAS tokens. By default: 3600000	YES, if Operations Team prefers another duration time.
AURA_MICROSOFT_AZURE_STORAGE_STATIC_CONTAINER_NAME	string	Name of Azure blob container to be used as repository.	NO. Only if Operations Team changes it.
AURA_MIN_FILE_SIZE_BYTES	number	Minimum file size in bytes by default. It can be overridden by channel configuration. Default value: 256.	NO, it must be done through channel config.
AURA_MONGODB_PASSWORD	string	MongoDB password to access AURA_MONGODB_BOT_DATABASE in AURA_MONGODB_URI.	NO
AURA_MONGODB_POOL_SIZE	number	Number of connections available in MongoDB pool. By default, 60.	NO
AURA_MONGODB_SSL	boolean	true if access to AURA_MONGODB_URI is SSL. By default, false.	NO
AURA_MONGODB_SUFFIX	string	Suffix of Mongo database name. Its value at run-time will be the value from AURA_ENVIRONMENT_NAME-AURA_VERSION.replace(/./g,’-’)`.	NO
AURA_MONGODB_URI	string	MongoDB URI of the server handling AURA_MONGODB_BOT_DATABASE.	NO
AURA_MONGODB_USERNAME	string	MongoDB username to access AURA_MONGODB_BOT_DATABASE in AURA_MONGODB_URI.	NO
AURA_QUEUE_MANAGER_CHECK_QUEUE_INTERVAL_TTL	number	In milliseconds. The time interval used by the queue manager to check if the queue system has pending messages. By default, 1000.	NO
AURA_QUEUE_MANAGER_CONCURRENT_TASK	number	Number of concurrently processed tasks. By default, 5.	NO
AURA_QUEUE_MANAGER_INTERVAL	number	In milliseconds. Time interval for processing messages in the queue. By default: 200..	NO
AURA_QUEUE_MANAGER_MAX_ELEMENTS	number	Maximum number of elements in the queue. By default: 4.	NO
AURA_QUEUE_MANAGER_AGENT_INTERVAL	number	Time interval in milliseconds for the queue agent to check if the queue system has pending messages. By default: 20000.	NO
AURA_QUEUE_TYPE	string	Type of queue system to be used (MONGO/REDIS). By default: REDIS.	NO
AURA_REDIS_COMPONENT_CONFIG_PREFIX	string	Prefix for Redis configuration keys. By default: aura-config:.	NO
AURA_REDIS_MODE	string	Mode of Redis distribution. Values: CLUSTER, SENTINEL, SINGLE. By default: SENTINEL.	NO
AURA_REDIS_PREFIX	string	Redis prefix.	NO
AURA_REDIS_SENTINEL_INSTANCE_NAME	string	Name of the Redis instance. Used in SENTINEL mode.	NO
AURA_REDIS_HOSTS	string	String with a list of nodes separated by ‘,’, including host and port separated by ‘:’. For example: “localhost:port,localhost2:port2”.	NO
AURA_REDIS_DATABASE	number	Database number for SINGLE or SENTINEL mode. By default: 0.	YES
AURA_REDIS_PASSWORD	string	String with Redis password.	YES
AURA_REDIS_USE_CONNECTION_POOL	boolean	Use pool connections for Redis. By default: true.	YES
AURA_REDIS_CONNECTION_POOL_MIN	number	Minimum number of connections in the pool. By default: 2.	YES
AURA_REDIS_CONNECTION_POOL_MAX	number	Maximum number of connections in the pool. By default: 100.	YES
AURA_REDIS_MAX_RECONNECT_RETRIES	number	Number of retries to connect to Redis. By default: 25	YES
AURA_REDIS_MAX_RECONNECT_INTERVAL	number	Time in milliseconds to wait before reconnecting to Redis. By default: 5000.	YES
AURA_REDIS_QUEUE_PREFIX	number	Prefix for Redis keys used by the queue system. By default: file_manager:queue:.	YES
AURA_REDIS_QUEUE_SHARD_COUNT	number	Number of shards to generate lists for store unprocessed context. By default: 5	YES
AURA_SHUTDOWN_GRACEFUL_TTL	string	Number of milliseconds to wait until all authentication-api tasks are done before shutting each pod down. By default 25000 (25 seconds)	NO
AURA_VERSION	string	Number of the Aura release being executed.	NO
AURA_KPIS_BLOB_TIME_WAIT_IF_EXISTS	number	Time to wait in milliseconds if the KPIs blob exists to avoid duplicate headers. By default: 1000.	YES

3 - API definition

Aura file manager API definition