Configuring Apache Livy for Hive Metastore

This page describes some common use cases for configuring Livy with Hive Metastore.

Starting from HPE Ezmeral Runtime Enterprise 5.4.0, you can configure Apache Livy for Hive Metastore in two different ways.
  1. Using the HPE Ezmeral Runtime Enterprise GUI during the Livy installation, see Installing and Configuring Apache Livy.
  2. Setting the ConfigMap name in the hiveSiteSource configuration option in values.yaml file in the Helm chart.
NOTE There is a known issue related to integration of Livy with Hive Metastore. See Spark on Kubernetes Issues (5.4.0).

Examples of SQL (Hive) Support in Livy

  • Create Livy session:
    curl -k -v \
    -X POST \
    -H "Content-Type:application/json" \
    -d '{}' \
    -u "username:password" \
    https://xx-xxx-xxx.xx.lab:10075/sessions
  • Create table:
    curl -k \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{"kind": "sql", "code": "CREATE TABLE test1 (id int)"}' \ 
    -u "username:password" \
    https://xx-xxx-xxx.xx.lab:10075/sessions/0/statements
  • Insert some data:
    curl -k \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{"kind": "sql", "code": "INSERT INTO test1 VALUES (1),(2),(3)"}' \
    -u "username:password" \
    https://xx-xxx-xxx.xx.lab:10075/sessions/0/statements
  • Select from table:
    curl -k \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{"kind": "spark", "code": "var a = sql(\"SELECT * FROM test1\"); a.show()"}' \ 
    -u "username:password" \
    https://xx-xxx-xxx.xx.lab:10075/sessions/0/statements