Spark Support

This topic describes the Spark enhancements and limitations for HPE Ezmeral Runtime Enterprise.

No Support for Apache Spark 3.1.1 Operator

In HPE Ezmeral Runtime Enterprise 5.4, the unified version of Apache Spark 3.x.x (default and recommended) replaces the preview version of Apache Spark 3.1.1 introduced in HPE Ezmeral Runtime Enterprise 5.3.

Hewlett Packard Enterprise no longer supports Apache Spark 3.1.1 and recommends you to move all your Spark applications to Apache Spark 3.x.x.

Spark Limitations

  • When you configure the Amazon S3 to store event logs, Spark History Server does not show the list of running applications when you click Show incomplete applications. Amazon S3 doesn’t support the append functionality and web UI only displays the list of completed application runs.

  • Starting from HPE Ezmeral Runtime Enterprise 5.6.0, Spark 3.3.x and later versions support enhanced S3 features introduced in Hadoop 3.x.
  • Starting from HPE Ezmeral Runtime Enterprise 5.5.0, Spark Operator supports open-source Spark images. You can now build your Spark with Hadoop 3 profile or any other profile of your choice. See Spark Operator for details.
  • Starting from HPE Ezmeral Runtime Enterprise 5.4.0, Livy does not support ZooKeeper for session recovery. Configure pvc or disabled option for sessionRecovery in all data-fabric (default) tenants or none tenants.

For Spark issues, see Issues and Workarounds.