Connecting to Livy from a KubeDirector Notebook Application with Spark Magic

This topic describes how to initiate a Spark session from a Livy endpoint and how to use the %setLivy magic to connect to a different Livy session.

Using the %%spark Magic to Start Spark Sessions

If you are using a PySpark kernel in a KubeDirector Notebook application, you can use %%spark magic to set the Livy endpoints. Executing the magic generates a request for the user password. If the Livy session needs authentication, enter the password.

For example:
Figure 1. Using %%spark Magic to Start a Spark Session


Using the %setLivy Magic to Connect to Livy Sessions

For other kernels in a KubeDirector Notebook application, you can use the %setLivy magic to connect to a different Livy session.
%setLivy --url <livy-endpoint>
For example:
%setLivy --url http://<endpoint-details>:<port-number>

To connect to a remote Livy session in a different cluster or in a different system, use %setLivy magic. Provide the --url argument followed by the Livy endpoint to which you want to connect. Executing the magic generates a request for the user password. If the Livy session needs authentication, enter the password.

You can then import a Spark session and proceed to work in Spark.

Using the %%configure Magic to Configure Livy Sessions

You can use %%configure magic command on HPE Ezmeral ML Ops to override the default Livy configuration and custom configure each Livy session.
%%configure -f
{"driverCores": 1,
 "executorCores": 1,
 "driverMemory": "1000M",
 "executorMemory": "1000M",
 "conf": {"spark.kubernetes.container.image": "gcr.io/mapr-252711/<livy-image-for-PySpark>",
  "spark.kubernetes.driver.limit.cores": "1",
  "spark.kubernetes.executor.limit.cores":"1"}}