Spark Version Comparison Matrix
This matrix shows the different versions of Spark supported on HPE Ezmeral Runtime Enterprise.
HPE Ezmeral Runtime Enterprise 5.6.x
HPE Ezmeral Runtime Enterprise supports open-source Spark images, and HPE distribution for Apache Spark 2.4.7 and Apache Spark 3.3.1. For details, see Spark Operator.
The following table compares the two different versions of Spark on HPE Ezmeral Runtime Enterprise.
Capabilities | Spark 2.4.7 | Spark 3.3.1 |
---|---|---|
Enterprise readiness | Yes | Yes (Default and Recommended) |
Installation | Using KubeDirector applications in HPE Ezmeral Runtime Enterprise GUI or manual deployment using Helm charts. | Using KubeDirector applications in HPE Ezmeral Runtime Enterprise GUI or manual deployment using Helm charts. |
Delta Lake Support | No | Yes |
Data Fabric (Filesystem, Database, Streams) | Yes | Yes |
DTAP support for HDFS | Yes | Yes |
Spark History Server | Yes | Yes |
REST interface (Livy service) | Livy | Livy |
Spark Thrift Server | Yes | Yes |
Spark Client Pods | Yes | Yes |
Enterprise Security | Yes (AD/LDAP, Data Fabric SASL, Kubernetes RBAC) | Yes (AD/LDAP, Data Fabric SASL, Kubernetes RBAC) |
GPU (Nvidia RAPIDS) | No | Yes (RAPIDS 22.10.0) |