Provides an overview of YARN.
A YARN application can be a MapReduce version 2 (MRv2) application or a non-MapReduce
application. The Warden on each node calculates the resources that can be allocated to
process YARN applications. Each application has an ApplicationMaster that negotiates YARN
container resources. For MapReduce applications, YARN processes each map or reduce task in
a container. The ApplicationMaster requests resources from the ResourceManager based on
memory, CPU, and disk requirements for the YARN containers. For YARN containers that
process MRv2 tasks, there are additional considerations. See YARN Container Resources for MapReduce Version 2 Applications for details.
The ApplicationMaster requests YARN container resources based on the values of the
following parameters:
- yarn.scheduler.minimum-allocation-mb
- Default: 1024
- Description: Defines the minimum memory allocation available for a
container in MB.
To change the value, edit the yarn-site.xml file for the node that runs the ResourceManager.
Assign the new value to this property, then restart the
ResourceManager.
- yarn.scheduler.maximum-allocation-mb
- Default: 8192
- Description: Defines the maximum memory allocation available for a
container in MB.
To change the value, edit the yarn-site.xml file for the node that runs the ResourceManager.
Assign the new value to this property, then restart the
ResourceManager.
- yarn.nodemanager.resource.memory-mb
- Default: Variable. This value is calculated by Warden.
- Description: Defines the memory available to processing Yarn containers on
the node in MB.
Warden uses the following formula to calculate this value:
[total physical memory on node] - [memory required by the operating
system, file system, and HPE Ezmeral Data Fabric services
installed on the node].
To determine the value, go to the
ResourceManager UI and view the memory available for that node.
- yarn.nodemanager.resource.cpu-vcores
- Default: Variable. This value is calculated by Warden.
- Description: Defines the number of CPUs available to process YARN
containers on this node.
Warden uses the following formula to calculate this
value: [Number of CPU cores on node] - [Number of CPU cores assigned to
file system].
To determine the
value, go to the ResourceManager UI or the YARN pane on the Control System and
view the number of CPUs available for that node.
To change the value,
edit the yarn-site.xml file for the node,
assign the new value to this property, then restart the NodeManager.
- yarn.nodemanager.resource.io-spindles
- Default: Variable. This value is calculated by Warden.
- Description: Defines the number of disks available to process YARN
containers. Warden uses the following formula to calculate this value:
[Number of of disks on the node].To determine the value, go
to the ResourceManager UI or the YARN pane on the Control System and view the
disk information for this node.
