Configuring Spark to Work with Hive Metastore

This topic describes how to configure Spark to use Hive Metastore on HPE Ezmeral Runtime Enterprise.

The main concept of running a Spark application against Hive Metastore is to place the correct hive-site.xml file in the Spark conf directory. To do this in Kubernetes:
  • The tenant namespace should contain a ConfigMap with hivesite content (for example, my-hivesite-cm). Contents of the hive-site.xml should be stored by any key in the configmap. In default ConfigMaps, the key is hive-site.xml. You can create the ConfigMap manually, or use any available ConfigMap, such as the Hive Metastore default ConfigMaps.
  • Assuming that your ConfigMap name is mapr-hivesite-cm and the key is hive-site.xml, you can mount it to the Spark application CR.
    For example:
    # Declare a volume in spec
volumes:
  - name: hive-site-volume 
    configMap:    
      name: mapr-hivesite-cm.......
      driver:… driver spec ... 

# Mount volume to driver pod  
volumeMounts:    
  - name: hive-site-volume      
    mountPath: /opt/mapr/spark/spark-<version>/conf/hive-site.xml      
    subPath: hive-site.xml