Managing Livy Dependencies

This topic describes how to pass the dependencies to Livy applications in HPE Ezmeral Runtime Enterprise using Notebook.

About this task

You can configure dependencies for Livy Applications using Notebook on HPE Ezmeral Runtime Enterprise. The following schemas are supported for passing dependencies to Livy applications:
  • local
  • dtap

Example

To configure the dependencies on Livy Applications using Notebook, perform the following steps:

  1. Log in to HPE Ezmeral Runtime Enterprise as Kubernetes Tenant Administrator or Kubernetes Tenant Member.
  2. Click DataTaps >TenantStorage.
  3. Create jars directory.
  4. Upload dependencies for Spark applications on jars directory.
  5. Click Notebooks > Notebook Endpoints > Access Points.
  6. Add the following configuration options for Livy applications:
    %%configure -f
{
    "conf": {
        "spark.hadoop.fs.dtap.impl": "com.bluedata.hadoop.bdfs.Bdfs"
        , "spark.hadoop.fs.AbstractFileSystem.dtap.impl": "com.bluedata.hadoop.bdfs.BdAbstractFS"
        , "spark.hadoop.fs.dtap.impl.disable.cache": "false"
        , "spark.kubernetes.driver.label.hpecp.hpe.com/dtap": "hadoop2-job"
        , "spark.kubernetes.executor.label.hpecp.hpe.com/dtap": "hadoop2-job"
        , "spark.driver.extraClassPath": "/opt/bdfs/bluedata-dtap.jar"
        , "spark.executor.extraClassPath": "/opt/bdfs/bluedata-dtap.jar"
    }
    , "jars": [
        "local:///opt/bdfs/bluedata-dtap.jar"
    ]
    , "pyFiles": [
        "dtap://TenantStorage/python/kazoo-2.8.0-py2.py3-none-any.whl"
    ]
}
  7. Restart the kernel.