mapred-site.xml
Lists the parameters for MapReduce configuration.
MapReduce is a type of application that can run on the Hadoop 2.x framework. MapReduce
configuration options are stored in the
/opt/mapr/hadoop/hadoop-2.x.x/etc/hadoop/mapred-site.xml
file and
are editable by the root
user. This file contains configuration
information that overrides the default values for MapReduce parameters. Overrides of the
default values for core configuration properties are stored in the HPE Ezmeral Data Fabric Parameters file.
To override a default value for a property, specify the new
value within the <configuration>
tags,
using the following format:
<property>
<name> </name>
<value> </value>
<description> </description>
</property>
Configurations for MapReduce Applications
The configuration comprises the following parameters:
- mapreduce.framework.name
- Value: yarn
- mapreduce.input.fileinputformat.split.maxblocknum
- Value: 0
- mapreduce.map.memory.mb
- Value: 1024
- mapreduce.map.java.opts
- Value: -Xmx900m --add-opens java.base/java.lang=ALL-UNNAMED -XX:+UseParallelGC
- mapreduce.reduce.memory.mb
- Value: 3072
- mapreduce.reduce.java.opts
- Value: -Xmx2560m --add-opens java.base/java.lang=ALL-UNNAMED -XX:+UseParallelGC
- mapreduce.task.io.sort.mb
- Value: 512
- mapreduce.task.io.sort.factor
- Value: 100
- mapreduce.reduce.shuffle.parallelcopies
- Value: 50
Configurations for MapReduce JobHistory Server
- mapr.localspill.expiration.date
- Value: days
- mapreduce.jobhistory.address
- Value: MapReduce JobHistory Server host:port
- mapreduce.jobhistory.webapp.address
- Value: MapReduce JobHistory Server Web UI host:port
- mapreduce.jobhistory.intermediate-done-dir
- Value:
/mr-history/tmp
- mapreduce.jobhistory.intermediate-done-scan-timeout
- Value: milliseconds
- mapreduce.jobhistory.done-dir
- Value:
/mr-history/done
- mapreduce.jobhistory.webapp.https.address
- Value: Secure MapReduce JobHistory Server Web UI host:port (HTTPS)
Sample Hadoop 2.x mapred-site.xml File
The following mapred-site.xml
file defines values for two job history
parameters.
<configuration>
<property>
<name>mapreduce.jobhistory.address</name>
<value>__HS_IP__:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>__HS_IP__:19888</value>
</property>
</configuration>
Configuration for Apache Shuffle
- mapreduce.job.shuffle.provider.services
- Value: mapreduce_shuffle
- mapreduce.job.reduce.shuffle.consumer.plugin.class
- Value: org.apache.hadoop.mapreduce.task.reduce.Shuffle
- mapreduce.job.map.output.collector.class
- Value: org.apache.hadoop.mapred.MapTask$MapOutputBuffer
- mapred.ifile.outputstream
- Value: org.apache.hadoop.mapred.IFileOutputStream
- mapred.ifile.inputstream
- Value: org.apache.hadoop.mapred.IFileInputStream
- mapred.local.mapoutput
- Value: true
- mapreduce.task.local.output.class
- Value: org.apache.hadoop.mapred.YarnOutputFiles