Add JARs to Livy Sessions

When you submit Spark jobs through the Livy REST server, the process for managing JAR dependencies works differently than with spark-submit. In particular:

• Livy always overrides the spark.jars setting from spark-defaults.conf with its own value, populated from the livy.repl.jars property in livy.conf. This ensures that the JARs required to run the Livy session application are deployed.

• Therefore, setting spark.jars in spark-defaults.conf has no effect on Livy sessions.

  You can add JARs to Livy sessions using either of the following approaches.

Adding JARs to a specific Livy session

Use this method when individual sessions require specific JAR files.

1. Upload the JAR files to a location on the MapR file system.
2. When creating a Livy session, specify the JARs using the jars option in the POST /sessions request body. The jars option maps directly to the spark.jars Spark property. For example:

   {
     "kind": "spark",
     "jars": [
       "/user/myuser/jars/custom-lib.jar",
       "/user/myuser/jars/delta-spark_2.12-3.3.2.jar"
     ]
   }

   IMPORTANT

   • The default URI scheme for the jars option is maprfs:///. When a file path has no URI scheme, Livy uses the fs.defaultFS Hadoop property as the default. On Data Fabric clusters, this property is set to maprfs:///.

   • • The file:/// scheme is not allowed in the jars option. In Spark, file:/// refers to the local file system of the machine running spark-submit. This does not apply to Livy, where sessions are submitted over HTTP — the Livy server has no access to the client's local file system.

Adding default JARs to all Livy sessions

Use one of the following methods when you need specific JARs available to every Livy session.

Method 2: Set the livy.repl.jars property in livy.conf

Use this method when you want explicit control over the list of JARs deployed to every session.

1. Open /opt/mapr/livy/livy-<version>/conf/livy.conf in a text editor.
2. Add or update the livy.repl.jars property. You must list all standard Livy JARs explicitly in addition to your custom JARs. For example (shown for Livy from DEP 10.0.0):

   livy.repl.jars file:///path/to/custom.jar,file:///opt/mapr/livy/livy-0.8.0/repl_2.12-jars/commons-codec-1.9.jar,file:///opt/mapr/livy/livy-0.8.0/repl_2.12-jars/livy-core_2.12-0.8.0.200-dep-1000.jar,file:///opt/mapr/livy/livy-0.8.0/repl_2.12-jars/livy-repl_2.12-0.8.0.200-dep-1000.jar

   IMPORTANT

   If you omit any of the standard Livy JARs from this list, Livy sessions will fail to start. List the full contents of the repl_<scala_version>-jars directory along with your custom JARs.
3. Restart the Livy server for the changes to take effect.