Spark 3.2.0.100 - 2305 (EEP 8.1.1) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 3.2.0.100.
The notes below relate specifically to the Hewlett Packard Enterprise Distribution for Apache Hadoop. For more information, you may also want to consult the open-source Spark 3.2.0 Release Notes.
These release notes contain only Hewlett Packard Enterprise specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.
Spark Version | 3.2.0.100 |
Release Date | May 2023 |
HPE Version Interoperability | See Component Versions for Released EEPs and EEP Components and OS Support. |
Source on GitHub | https://github.com/mapr/spark |
GitHub Release Tag | 3.2.0.100-eep-811 |
Maven Artifacts | https://repository.mapr.com/maven/ |
Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
- Beginning with EEP 6.0.0, the KeyStore and
TrustStore password can be removed from
spark-defaults.conf
and set in/opt/mapr/conf/ssl-client.xml
. - Beginning with Core 6.2 and EEP 7.0, Spark supports SSL for WebUI.
Hive Support
- Starting from Spark 3.1.2, Spark supports Hive 2.3.
New in This Release
For a complete list of new features, see the open-source Spark 3.2.0 Release Notes.- Bug fixes.
Fixes
This HPE release includes the following new fixes since the latest Spark release. For details, refer to the commit log for this project in GitHub.
GitHub Commit | Date (YYYY-MM-DD) | Comment |
cd2129b | 07/03/2023 | [SPARK-41952][SQL] Fix Parquet zstd off-heap memory leak as a workaround for PARQUET-2160 |
1ac75cd |
08/03/2023
|
MapR [SPARK-1154] CVE-2022-33891: Apache Spark Shell Command Injection Vulnerability for core 6.2.0 |
898ead4 | 26/04/2023 | MapR [SPARK-1170] Update base compatibility version for spark 3.2.0 in MEP-8.1.1 |
c489c34 | 02/05/2023 | MapR [SPARK-1175] CVE fixes for Jackson, Guava and Protobuf |
64ee8b9 | 05/05/2023 | MapR [SPARK-1173] Address CVE-2023-22946 |
12aa8af | 15/05/2023 | MapR [SPARK-1181] View creation fail with "Caused by: java.lang.NumberFormatException: For input string: "${pom"" |
Known Issues and Limitations
-
When you enable the SSL in a mixed (FIPS and non-FIPS) configuration, Spark application run fails. To run Spark applications, set
spark.ssl.ui.enabled
option tofalse
inspark-defaults.conf
configuration file. -
If you are using Spark SQL with Derby database without Hive or Hive Metastore installation, you will see the Java Runtime Exception. See Apache Spark Feature Support for workaround. Spark 3.1.2 does not support
log4j1.2
logging on HPE Ezmeral Data Fabric.
Resolved Issues
- None.