Configuring Drill Memory

A system administrator can modify the amount of system memory that Warden allocates to the Drill service on each node in the warden.drill-bits.conf file. Drill users, with file permissions, can modify the amount of heap and direct memory allocated to the Drill service on each node in the drill-env.sh file.

NOTE
The cumulative memory allocation in drill-env.sh cannot exceed the memory allocation in warden.drill-bits.conf.
After modifying drill-env.sh, restart Drill:
$ maprcli node services -name drill-bits -action restart -nodes <space-separated-list-of-drill-hostnames>
After modifying warden.drill-bits.conf, run the configuration script, configure.sh, to update the node configuration and then restart Drill:
/opt/mapr/server/configure.sh -R  
$ maprcli node services -name drill-bits -action restart -nodes <space-separated-list-of-drill-hostnames>

The following sections describe the warden.drill-bits.conf and drill-env.sh files in detail.

Drill Memory Allocation in a Warden-Managed Cluster

If you install and run Drill under the Warden service, Warden manages the amount of system memory that Drill can use. By default, Warden allocates 20% of the system memory on a node to the Drill service. For example, if a node has 50GB of memory, Warden allocates 10GB (20% of 50GB) to the Drill service.

A system administrator can define the amount of memory that Warden allocates to Drill by changing the value of the DRILLBIT_MAX_PROC_MEM variable in /opt/mapr/drill/drill-<version>/conf/warden.drill-bits.conf.

When starting, Drill verifies that the amount of memory configured in drill-env.sh does not exceed the limit set by the service.env=DRILLBIT_MAX_PROC_MEM variable in warden.drill-bits.conf. If the settings in drill-env.sh exceed the setting in Warden, the system prints a message stating the issue; Warden does not start the Drill service on the node.

The warden.drill-bits.conf file contains the following settings:
NOTE
Drill automatically configures the service.heapsize parameters. Do not modify them.
#Default Drill Mem Distrib: 20% of System memory
service.env=DRILLBIT_MAX_PROC_MEM=20%
//Specifies the maximum amount of memory that Warden will allocate to the Drill service on the node. You can set this value as a percentage of system memory or as an absolute value in GB. Memory configured in drill-env.sh cannot exceed this memory setting.

service.heapsize.min=5120
//Minimum heap size. Do not change this value. The value is auto-populated and represents the minimum memory that the Drillbit process will take.
 
service.heapsize.max=13312
//Maximum heap size. Do not change this value. The value is auto-populated and represents the maximum memory that the Drillbit process will take.
 
#Warden will allocate 20% of memory for Drill
service.heapsize.percent=20
//Do not change this value. Total heap size available based on the value set for the DRILLBIT_MAX_PROC_MEM variable. If the variable is defined in absolute values, it is represented as a percent of the system memory.

Drill Memory Allocation in drill-env.sh

You can configure the amount of heap and direct memory allocated to Drill on each node in the/opt/mapr/conf/conf.d/drill-env.sh file. If you do not manually configure the heap and direct memory, Drill calculates these values based on the amount of system memory that Warden allocates to Drill and auto-populates the settings for the variables.

The cumulative amount of memory allocated to Drill in drill-env.sh cannot exceed the amount of memory that Warden allocates to Drill, which is set by the DRILLBIT_MAX_PROC_MEM variable in warden.drill-bits.conf.

NOTE
The values in drill-env.sh, such as 13G, are examples and do not indicate the default memory limits for Drill. By default, Warden allocates 20% of the system memory on a node to Drill.
The drill-env.sh file contains the following memory variables that you can uncomment and modify:
#export DRILLBIT_MAX_PROC_MEM=${DRILLBIT_MAX_PROC_MEM:-"13G"}
//Specifies the maximum amount of system memory that the Drill service can use on a node. Must be equal to or less than the value set for DRILLBIT_MAX_PROC_MEM in warden.drill-bits.conf. You can set this value as a percentage of system memory or as an absolute value in GB. If you define this variable, without defining the heap and direct memory variables, Drill automatically calculates the heap and direct memory values. 

#export DRILL_HEAP=${DRILL_HEAP:-"4G"}
//Maximum theoretical heap limit for the JVM per node.

#export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"8G"}  
//Java direct memory limit per node.

#export DRILLBIT_CODE_CACHE_SIZE=${DRILLBIT_CODE_CACHE_SIZE:-"1G"} 
//The memory limit for the compiled code generated by the JVM JIT compiler. Do not modify. The value for this parameter is auto-computed based on the heap size and cannot exceed 1GB.
NOTE
If performance is an issue, add -Dbounds=false, as shown:
export DRILL_JAVA_OPTS="$DRILL_JAVA_OPTS -Dbounds=false"