Logging Options

In a HPE Ezmeral Data Fabric cluster, the logging option that you configure defines how the logs are stored and accessed:

  • Centralized logging. The logs are written to local volumes on the file system.
  • YARN log aggregation. The logs are written to the local file system, and then the container logs from each node are aggregated and stored on the file system.
  • Local logging. The log files for each job or application are written to the local file system. This method is the default behavior for MapReduce version 2 (MRv2) applications and non-MapReduce applications.
    • For MRv2 or other applications that run on YARN, the logs are written to the following directory on the local file system: /opt/mapr/hadoop/hadoop-2.x.x/logs/userlogs/

The logging options that you can choose from are determined by the type of jobs or applications that you run:

Type of Job or Application Available Logging Options
MRv2
  • Centralized logging
  • YARN log aggregation
  • Local logging (default)

YARN applications (non-MapReduce)

  • YARN log aggregation
  • Local logging (default)

If you enable centralized logging for MRv2, the MapReduce applications uses centralized logging, while the other YARN applications in the cluster use local logging.

NOTE
Select a logging option that stores the logs on the file system for the following reasons:
  • Prevent job or application failures due to a lack of space on the local file system for logs.
  • Prevent the loss or inaccessibility of logs due to node failure. Logs stored in a local volume are two-way replicated.