Enabling YARN Local-Node Log Aggregation

The steps in this procedure configure log aggregation for NodeManager processes, enabling you to store the logs on nodes (node-local volumes) where the YARN containers are launched.

To enable YARN local-node log aggregation, add or edit the following properties in the yarn-site.xml file:

  1. Set the yarn.node-local-log-aggregation.enable value to true.
    NOTE
    The default setting for YARN Log Aggregation (yarn.log-aggregation-enable) should be removed or set to false in the yarn-site.xml file.
  2. Optional: Set the yarn.node-local-log-aggregation.metadata-path value to a location in the system. By default the location is maprfs:///NM_REMOTE_APP_LOG_DIR/<user>/logsMeta. NM_REMOTE_APP_LOG_DIR should match the yarn.nodemanager.remote-app-log-dir property.
    NOTE
    The location should not be an absolute path (the location begins from /). In the file system (maprfs), the default setting for NM_REMOTE_APP_LOG_DIR is /tmp/logs.
  3. Optional: Set the node-local-log-aggregation.metadata-filename value to the name of the metafile that should contain the information about containers for each node. By default, the file name is containers.seq. If you use default paths, the file is stored at /tmp/logs/${user}/logsMeta/<appId>/<nodeName>/containers.seq.
  4. Restart NodeManager and HistoryServer services.

Aggregated logs are owned by the user who runs the job.

Different users cannot see each other's logs. For example, if the HPE Data Fabric user admin runs a job, the logs are stored in maprfs:///var/mapr/local/<nodeNames>/mapred/nodeManager/logs/admin/<appId>. If a user analyst runs a job, the logs are stored in maprfs:///var/mapr/local/<nodeNames>/mapred/nodeManager/logs/analyst/<appId>.