Integrating Spark Thrift Server with Hive Metastore

This topic describes how to integrate Spark Thrift Server with Hive Metastore in HPE Ezmeral Runtime Enterprise.

You can integrate Spark Thrift Server with Hive Metastore in two ways:
Using YAML

Set hiveSiteSource parameter in values.yaml file of Spark Thrift Server Helm chart.

Using HPE Ezmeral Runtime Enterprise GUI

Set Hive Metastore box with the name of a ConfigMap with hive-site.xml configuration of a Hive Metastore during Spark Thrift Server installation. See Installing and Configuring Spark Thrift Server.

The value for hiveSiteSource parameter in values.yaml file or Hive Metastore box in GUI is ConfigMap. You must enter the ConfigMap with hive-site.xml configuration of the Hive Metastore during the Spark Thrift Server installation.

There are three separate ConfigMap values for three situations:

  1. If you are installing and configuring the Spark Thrift Server in the same tenant namespace as the Hive Metastore, configure the Spark Thrift Server by using the default hivesite-cm ConfigMap.

    When you install the Hive Metastore in a tenant namespace, Hive Metastore auto generates a ConfigMap with the name hivesite-cm that contains the hive-site.xml configuration of the Hive Metastore.

  2. If you are using Hive Metastore installed in external Data Fabric, Hive Metastore auto generates a ConfigMap with the name hivesite-external-cm that contains the hive-site.xml configuration of the Hive Metastore.

    Configure the Spark Thrift Server by using hivesite-external-cm ConfigMap.

    NOTE External Data Fabric is HPE Ezmeral Data Fabric on Kubernetes configured in external Kubernetes cluster or HPE Ezmeral Data Fabric on Bare Metal.

  3. If you are using the Hive Metastore installed in another namespace or some external Hive Metastore, you must manually create a ConfigMap for that Hive Metastore in the Spark Thrift Server tenant namespace.

    Example of a ConfigMap with hive-site.xml configuration of the Hive Metastore:
    kind: ConfigMap
apiVersion: v1
metadata:
  name: hivesite-cm
data:
  hive-site.xml: <your-hive-site-configurations>
    To create a ConfigMap in the tenant namespace, run:
    kubectl apply -f -n <tenant-namespace>

    Configure the Spark Thrift Server with manually created ConfigMap.