Logging

Describes how logging works in HPE Ezmeral Unified Analytics Software and how to access log files for the platform and applications.

Logging is crucial for monitoring and troubleshooting applications and the cluster infrastructure. Logs capture data generated by applications, containers, and Kubernetes components running in a Kubernetes cluster.

In HPE Ezmeral Unified Analytics Software, you can easily access log files to monitor and troubleshoot issues.

Log Rotation

To prevent storage issues, logs are automatically rotated to ensure that the shared1 directory does not exceed the limit of 10 MB per file. When the file size surpasses 10 MB, the old copy is retained, and a new log file is created, effectively managing storage and keeping it under control.
NOTE
Only one old copy of 10 MB is retained at any time.

For example, the airflow-ui.log undergoes renaming as airflow-ui.log.1 when it exceeds 10 MB in size. Simultaneously, a new log file named airflow-ui.log is created. Similarly, if the size of the new airflow-ui.log file exceeds the 10 MB threshold, the current airflow-ui.log.1 log file is replaced with the new logs from the airflow-ui.log file. This log rotation process ensures efficient management of log files while maintaining the specified size limit.

1 The shared directory is persistent volume storage shared by all users.

Accessing Log Files

To access the log files:
  1. Sign in to HPE Ezmeral Unified Analytics Software.
  2. In the left navigation bar, go to Data Engineering > Data Sources.
  3. Select the Data Volumes tab.
  4. On the Data Volumes tab, select the logs/ folder. Under the logs/ folder is a folder named for the installation. The logs/<installation-name>/ folder contains the following subdirectories with logging data:
    apps/
    Contains platform and application component logs. This directory contains the following subdirectories:
    app-core/
    Contains logs for core application pods.
    app-user/
    Contains logs for user-initiated pods such as notebooks, inference jobs, Spark jobs, and others.
    platform/
    Contains infrastructure pod logs.
    audit/
    Contains the native Kubernetes audit logs. The native Kubernetes audit logs record requests made to the Kubernetes API server, such as which users or services requested access to cluster resources and why the requests were authorized or rejected.
    system/
    Contains system node service logs.
    TIP
    The Actions column provides options that you can use on directories and log files, for example:
    • Open, rename, or delete a folder or directory
    • Rename, download, or delete log files