Manage Aura logs

Introduction

Logs are files that record specific single events, warnings and errors as they occur within a software environment. They can include contextual information, such as the time an event occurred and which user or endpoint was associated with it.

In Aura, logs are generated by specific components when an event happens and stored in order to monitor or debug the system.

Logs are stored in an ElasticSearch cluster.

Once stored, Aura integrates a logging system based on Kibana, which is the official tool to manage logs in Aura. Moreover, logs can be managed with Grafana and fluentd for specific features.

⚠️ You should not integrate third-party applications or scripts with ElasticSearch. These kinds of integrations are weak because the ElasticSearch API is not part of the public interface with the OB. This means that it could change without notice for several reasons such as updating the version of ElasticSearch or changing Aura internal architecture.

Manage logs in Kibana

The official Kibana User Guide is the reference guide to use Kibana.

Moreover, the current section includes certain useful points for managing Aura logs through this tool.

Policies in Kibana

Kibana includes index lifecycle policies.

By default, we add one policy for each index created (service and system index), to delete the logs older than seven days.

Snapshot in Kibana

Index snapshot is configured by default as long-term storage for the logs. These snapshots are taken daily and end in the cluster Azure Storage blob container (aura-backups/elk).

Manage logs in Grafana

Discover section

The “Discover” section in Grafana is very useful to look for logs and troubleshoot issues.

You can full-text search logs using Lucene query syntax.

Moreover, logs are tagged with many fields that can be useful to narrow down a search, such as:

• kubernetes.labels.app: name of the Kubernetes application that generated the log.
• kubernetes.pod_name: name of the Kubernetes pod that generated the log.
• corr: correlator that tracks E2E requests.
• lvl: log level (TRACE, DEBUG, INFO, WARN, ERROR or FATAL).

Queries that rely on a specific text are weak. Aura cannot guarantee that log messages do not change between versions. In fact, they do change. This is why metrics based on logs will not be reliable and it is not recommended to use Kibana to get metrics.

Manage logs in fluentd

Logs external forwarding feature

It is possible to send logs to an external system (a fluentd endpoint).

To enable this feature, add the following configuration to your config file:

external_forwarding:
  secret_shared_key: "mysecretkey"
  tls_config:
    tls_enabled: True
  remote_servers:
  - hostname: xxx
    port: yyy

• Set hostname and port fields with the remote endpoint. If you configure more than one remote server, fluentd load balances the traffic to them in a round-robin order.
• The hostname value can be an IP address, but it is not recommended if TLS is enabled. Turning off TLS is possible but discouraged for security reasons.
• secret_shared_key is used to verify client’s identity and must be configured properly in all the remote servers.

You can find additional information regarding receivers’ configuration (including TLS configuration and password authentication procedure) here.