Fair Scheduler Allocation File
Describes an allocation file and the entities within an allocation file.
{$MAPR_HOME}/hadoop/hadoop-3.x/etc/hadoop/fair-scheduler.xml
The allocation file is reloaded every 10 seconds to refresh the scheduler with any modified settings that are specified in the file.
The allocation file contains the following types of elements:
Queue Elements
minResources
maxResources
maxRunningApps
TIPThequeueMaxAppsDefault
value is used for any parent queue that does not set a value for themaxRunningApps
element.weight
schedulingPolicy
aclSubmitApps
aclAdministerApps
minSharePreemptionTimeout
maxContainerAllocation
TIPThemaxContainerAllocation
property sets a limit on the resources a queue can allocate for a single container. The value cannot exceedmaxResources
. If you do not setmaxContainerAllocation
, the value is inherited from a parent queue. The default values are set through theyarn.scheduler.maximum-allocation-mb
,yarn.scheduler.maximum-allocation-vcores
properties, which you can modify in/opt/mapr/hadoop/hadoop-3.x.x/etc/hadoop/yarn-site.xml
. ThemaxContainerAllocation
element is not valid for the root queue.
User Elements
User elements represent settings that govern the behavior of individual users. They can
contain a single property, maxRunningApps
, which limits the number of
running applications for a particular user. It contains the following elements:
userMaxAppsDefault
queueMaxAppsDefault
fairSharePreemptionTimeout
defaultQueueSchedulingPolicy
queuePlacementPolicy
For more information on these elements, see Hadoop: Fair Scheduler.
queueMaxAppsDefault
and do not set a
value for maxRunningApps
for the root queue, the value of
queueMaxAppsDefault
sets the application limit for all queues under the
root queue hierarchy.Example Allocation File
See example allocation file in Hadoop: Fair Scheduler.
Queue Access Control Lists
Queue Access Control Lists (Access Control List (ACL)s)
allow administrators to control who may take actions on particular queues. They are
configured with the aclSubmitApps
and aclAdministerApps
properties, which can be set per queue. Currently, the only supported administrative action
is killing an application. Anyone who has permission to administer a queue may also submit
applications to it. These properties take values in a format such as user1,user2
group1,group2 or group1,group2. An action on a queue is permitted if its user or group is in
the ACL of that queue or in the
ACL of any of that queue's
ancestors. Therefore, if queue2 is inside queue1, and user1 is in queue1's ACL, and user2 is in queue2's ACL, then both users
may submit to queue2.
For more information, see Hadoop: Fair Scheduler.
- The
yarn.admin.acl and yarn.acl.enable
Properties By default, on a secure cluster, users cannot kill jobs that do not belong to them.
On a secure cluster, you do not need to set the
yarn.acl
or theyarn.admin.acl
properties. By default, they are set as follows. On unsecured clusters, these properties are not set by default.<property> <name>yarn.acl.enable</name> > <value>true</value> > </property> <property> <name>yarn.admin.acl</name> > <value> </value> > </property>
The
To allow users to kill jobs that do not belong to them, or to get access to their logs, you need to set theyarn.admin.acl
property is set by default to " ", meaning that an administrator is not specified on a cluster.yarn.admin.acl
property with the user or group name.- Fair and Capacity scheduler root queue admins
-
For both the Fair scheduler and Capacity scheduler, the default value of the administrators for the root queues is
" "
.