Tags:
Categories:

Environment variables

List of environment variables handled by aura-databricks-jobs and avro-to-dataset-job-cli

Aura Databricks Jobs variables

List of environment variables handled by aura-databricks-jobs.

  • Properties marked in bold are mandatory
  • Properties marked in italics are optional
Property Type Description Modifiable by OB?
AURA_LOGGING_FORMAT string Format to be used in monitoring logs: console, json, string or simple. By default: simple. NO.
AURA_LOGGING_LEVEL string Level to be used in monitoring logs, from more to less verbose: 'DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL', 'OFF', 'NOTSET', 'CRITICAL. By default: INFO. YES, for development set it to DEBUG. In pre/production, it should be INFO or ERROR. For the analysis of an issue in pre/production, it may be changed to DEBUG.
AURA_VERSION string Number of the Aura’s release being executed. NO

Avro to Dataset job cli variables

List of job’s variables handled by avro-to-dataset-job-cli

  • Properties marked in bold are mandatory
  • Properties marked in italics are optional
Property Type Description Modifiable by OB?
AURA_ENVIRONMENT_NAME string Name of the environment where aura-databricks-jobs is deployed. For example: ap-next, es-dev, de-pre NO
AURA_FP_SPARK_BASE_URL string Base URL for Kernel Spark SDK. NO
AURA_FP_SPARK_CLIENT_ID string Client ID for Kernel Spark SDK. NO
AURA_FP_SPARK_CLIENT_SECRET string Client secret for Kernel Spark SDK. NO
AURA_FP_SPARK_JARS_PACKAGES string The jar packages configured only for local run, because in Databricks cluster this configuration is set previously. NO
AURA_FP_SPARK_JARS_REPOSITORIES string The repositories configured only for local run, because in Databricks cluster this configuration is set previously. NO
AURA_FP_SPARK_SCOPES string Scopes for Kernel Spark SDK. NO
AURA_FP_SPARK_PURPOSES string Purposes for Kernel Spark SDK. NO
AURA_FP_SPARK_SUFFIX_DATASET_TEST string Suffix used in tests with Kernel Spark SDK. By default: ``. NO. It is used for testing in the development environment.
AURA_KPI_AVRO_ADAPTER_CONFIG_PATH string File path for getting Aura Avro adapter configuration. NO
AURA_KPI_AVRO_PROCESS_ERROR_FILENAME string File name that records an error in the last execution. By default: databricks.ERROR. NO
AURA_KPI_AVRO_PROCESSED_FOLDER_PATH string Destination path for the processed KPIs Avro files. NO
AURA_KPI_AVRO_SOURCE_PATH string Source path for the KPIs Avro data. NO
AURA_KPI_AVRO_SOURCE_SIZE_REPORT_PATH string The file path for getting size report. By default: avro/sizeReport.json. NO
AURA_KPI_AVRO_REPORTS_MODE string Behavior of avro-to-dataset-job-cli regarding the generation of reports. Possible values: all: a report is generated for each processed file; none: it does not generate any report; error: it generates a report if an error has occurred. By default: all. NO
AURA_KPI_AVRO_REPORTS_DESTINATION_PATH string Destination path for the KPIs Avro reports. YES
AURA_KPI_AVRO_REPORTS_SAS_EXPIRATION integer Time to expiration in minutes for the report SAS URL generated when an error occurs. Default: 43200 (30 days). NO
AURA_KPI_AVRO_SCHEMAS_NOT_TO_UPLOAD string Schemas not to be uploaded in the KPIs Avro data, included in a list formatted as follows: schema_1:dataset_id_1,schema_1:dataset_id_2,schema_2;dataset_id_3 Example: dimensional:D_Aura_Channel,entity:E_Aura_GROOT. NO
AURA_KPI_AVRO_SCHEMAS_PATH string Schema path where Avro schemas are stored. By default, schemas. NO
AURA_MICROSOFT_AZURE_RETRY_TOTAL integer Total number of allowed retries. Default value: 3. NO
AURA_MICROSOFT_AZURE_RETRY_BACKOFF_FACTOR float Backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). In ’exponential’ mode, retry policy will sleep for: {backoff factor} * (2 ** ({number of total retries} - 1)) seconds. If the backoff_factor is 0.1, then the retry will sleep for [0.0s, 0.2s, 0.4s, …] between retries. The default value is 0.3. NO
AURA_MICROSOFT_AZURE_RETRY_BACKOFF_MAX integer Maximum backoff time in seconds. Default value: 5. NO
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCOUNT string Microsoft Storage account of the environment. NO
AURA_MICROSOFT_AZURE_STORAGE_COMMON_ACCESS_KEY string Microsoft Storage password of the deployment. NO
AURA_MICROSOFT_AZURE_STORAGE_KPIS_CONTAINER_NAME string Name of the container where the KPIs are stored. NO
SPARK_CONTEXT_LOG_LEVEL string Log level for the Spark context. NO
Last modified December 20, 2024: feat: update databricks job, aura-kpis and others docs #AURA-26617 (#1904) (5d22a69a)