Fair Scheduler Allocation File

Describes an allocation file and the entities within an allocation file.

An allocation file is an XML manifest that describes queues and their properties, as well as certain policy defaults. The allocation file is automatically created during HPE Data Fabric installation in the following directory:
{$MAPR_HOME}/hadoop/hadoop-3.x/etc/hadoop/fair-scheduler.xml

The allocation file is reloaded every 10 seconds to refresh the scheduler with any modified settings that are specified in the file.

The allocation file contains the following types of elements:

Queue Elements

Queue elements represent queues and can contain the following elements:
  • minResources
  • maxResources
  • maxRunningApps
    TIP
    The queueMaxAppsDefault value is used for any parent queue that does not set a value for the maxRunningApps element.
  • weight
  • schedulingPolicy
  • aclSubmitApps
  • aclAdministerApps
  • minSharePreemptionTimeout
  • maxContainerAllocation
    TIP
    The maxContainerAllocation property sets a limit on the resources a queue can allocate for a single container. The value cannot exceed maxResources. If you do not set maxContainerAllocation, the value is inherited from a parent queue. The default values are set through the yarn.scheduler.maximum-allocation-mb, yarn.scheduler.maximum-allocation-vcores properties, which you can modify in /opt/mapr/hadoop/hadoop-3.x.x/etc/hadoop/yarn-site.xml. The maxContainerAllocation element is not valid for the root queue.
For more information on these elements, see Hadoop: Fair Scheduler.

User Elements

User elements represent settings that govern the behavior of individual users. They can contain a single property, maxRunningApps, which limits the number of running applications for a particular user. It contains the following elements:

  • userMaxAppsDefault
  • queueMaxAppsDefault
  • fairSharePreemptionTimeout
  • defaultQueueSchedulingPolicy
  • queuePlacementPolicy

For more information on these elements, see Hadoop: Fair Scheduler.

TIP
If you set a value for queueMaxAppsDefault and do not set a value for maxRunningApps for the root queue, the value of queueMaxAppsDefault sets the application limit for all queues under the root queue hierarchy.

Example Allocation File

See example allocation file in Hadoop: Fair Scheduler.

Queue Access Control Lists

Queue Access Control Lists (Access Control List (ACL)s) allow administrators to control who may take actions on particular queues. They are configured with the aclSubmitApps and aclAdministerApps properties, which can be set per queue. Currently, the only supported administrative action is killing an application. Anyone who has permission to administer a queue may also submit applications to it. These properties take values in a format such as user1,user2 group1,group2 or group1,group2. An action on a queue is permitted if its user or group is in the ACL of that queue or in the ACL of any of that queue's ancestors. Therefore, if queue2 is inside queue1, and user1 is in queue1's ACL, and user2 is in queue2's ACL, then both users may submit to queue2.

For more information, see Hadoop: Fair Scheduler.

The yarn.admin.acl and yarn.acl.enable Properties

By default, on a secure cluster, users cannot kill jobs that do not belong to them.

On a secure cluster, you do not need to set the yarn.acl or the yarn.admin.acl properties. By default, they are set as follows. On unsecured clusters, these properties are not set by default. 

<property>
  <name>yarn.acl.enable</name> > 
  <value>true</value> > 
</property>
<property>
  <name>yarn.admin.acl</name> > 
  <value> </value> > 
</property>

The yarn.admin.acl property is set by default to " ", meaning that an administrator is not specified on a cluster.

To allow users to kill jobs that do not belong to them, or to get access to their logs, you need to set the yarn.admin.acl property with the user or group name.
Fair and Capacity scheduler root queue admins

For both the Fair scheduler and Capacity scheduler, the default value of the administrators for the root queues is " " .