Configuring Drill Memory
A system administrator can modify the amount of system memory that Warden allocates to
the Drill service on each node in the warden.drill-bits.conf file. Drill users,
with file permissions, can modify the amount of heap and direct memory allocated to the Drill
service on each node in the drill-env.sh file.
drill-env.sh cannot exceed the memory
allocation in warden.drill-bits.conf.drill-env.sh, restart
Drill:$ maprcli node services -name drill-bits -action restart -nodes <space-separated-list-of-drill-hostnames>warden.drill-bits.conf, run the configuration script, configure.sh, to update the
node configuration and then restart
Drill:/opt/mapr/server/configure.sh -R
$ maprcli node services -name drill-bits -action restart -nodes <space-separated-list-of-drill-hostnames>
The following sections describe the warden.drill-bits.conf and
drill-env.sh files in detail.
Drill Memory Allocation in a Warden-Managed Cluster
If you install and run Drill under the Warden service, Warden manages the amount of system memory that Drill can use. By default, Warden allocates 20% of the system memory on a node to the Drill service. For example, if a node has 50GB of memory, Warden allocates 10GB (20% of 50GB) to the Drill service.
A system administrator can define the amount of memory that Warden allocates to
Drill by changing the value of the DRILLBIT_MAX_PROC_MEM variable in
/opt/mapr/drill/drill-<version>/conf/warden.drill-bits.conf.
When starting, Drill verifies that the amount of memory configured in
drill-env.sh does not exceed the limit set by the
service.env=DRILLBIT_MAX_PROC_MEM variable in
warden.drill-bits.conf. If the settings in drill-env.sh
exceed the setting in Warden, the system prints a message stating the issue; Warden does not
start the Drill service on the node.
warden.drill-bits.conf file contains the following
settings:service.heapsize
parameters. Do not modify
them.#Default Drill Mem Distrib: 20% of System memory
service.env=DRILLBIT_MAX_PROC_MEM=20%
//Specifies the maximum amount of memory that Warden will allocate to the Drill service on the node. You can set this value as a percentage of system memory or as an absolute value in GB. Memory configured in drill-env.sh cannot exceed this memory setting.
service.heapsize.min=5120
//Minimum heap size. Do not change this value. The value is auto-populated and represents the minimum memory that the Drillbit process will take.
service.heapsize.max=13312
//Maximum heap size. Do not change this value. The value is auto-populated and represents the maximum memory that the Drillbit process will take.
#Warden will allocate 20% of memory for Drill
service.heapsize.percent=20
//Do not change this value. Total heap size available based on the value set for the DRILLBIT_MAX_PROC_MEM variable. If the variable is defined in absolute values, it is represented as a percent of the system memory.
Drill Memory Allocation in drill-env.sh
You can configure the amount of heap and direct memory allocated to Drill on each
node in the/opt/mapr/conf/conf.d/drill-env.sh file. If you do not manually
configure the heap and direct memory, Drill calculates these values based on the amount of
system memory that Warden allocates to Drill and auto-populates the settings for the
variables.
The cumulative amount of memory allocated to Drill in drill-env.sh cannot
exceed the amount of memory that Warden allocates to Drill, which is set by the
DRILLBIT_MAX_PROC_MEM variable in
warden.drill-bits.conf.
drill-env.sh, such as 13G, are examples and do not
indicate the default memory limits for Drill. By default, Warden allocates 20% of the system
memory on a node to Drill.drill-env.sh file contains the following memory variables that you can
uncomment and
modify:#export DRILLBIT_MAX_PROC_MEM=${DRILLBIT_MAX_PROC_MEM:-"13G"}
//Specifies the maximum amount of system memory that the Drill service can use on a node. Must be equal to or less than the value set for DRILLBIT_MAX_PROC_MEM in warden.drill-bits.conf. You can set this value as a percentage of system memory or as an absolute value in GB. If you define this variable, without defining the heap and direct memory variables, Drill automatically calculates the heap and direct memory values.
#export DRILL_HEAP=${DRILL_HEAP:-"4G"}
//Maximum theoretical heap limit for the JVM per node.
#export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"8G"}
//Java direct memory limit per node.
#export DRILLBIT_CODE_CACHE_SIZE=${DRILLBIT_CODE_CACHE_SIZE:-"1G"}
//The memory limit for the compiled code generated by the JVM JIT compiler. Do not modify. The value for this parameter is auto-computed based on the heap size and cannot exceed 1GB.
export DRILL_JAVA_OPTS="$DRILL_JAVA_OPTS -Dbounds=false"