Allocating Cluster Resource from the Command-Line
Provides a general overview on allocating cluster resources for a Data Fabric Hadoop cluster.
In a Data Fabric Hadoop cluster, the Warden sets the default resource allocation for the operating system, file system, Data Fabric Hadoop services, and YARN applications. Warden allocates resources to Data Fabric Hadoop services and applications based on the roles installed on a node. For example, Warden allocates resources for YARN applications on nodes with NodeManager role installed.
In general, you should not need to override the values set in the default configuration files and by Warden. However, you can provide updated values by adding or updating parameters in the Hadoop site configuration files or Warden files. To override parameter values for a single job, the option can be overridden in the command line when submitting a YARN application to the cluster.
To determine the current value
of a hadoop parameter, run hadoop conf | grep
<ParameterName>
. In
the following example, the hadoop conf
command was
used to get the value of mapreduce.map.memory.md
:
hadoop conf | grep mapreduce.map.memory.mb
<property><name>mapreduce.map.memory.mb</name><value>1024</value><source>mapred-site.xml</source></property>
Alternatively, run hadoop conf
without the grep
command
to get a full list of the current parameter values. To determine the current value
of a Warden parameter, open the Warden files located in the following directories:
/opt/mapr/conf/conf.d
and
/opt/mapr/conf
.
Refer to Allocating Memory for Nodes to allocate memory and resources in a Data Fabric cluster.