Spark 2.4.7.200 - 2201 (EEP 7.1.2) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.4.7.100.

The notes below relate specifically to the HPE Ezmeral Data Fabric Distribution for Apache Hadoop. For more information, you may also wish to consult the open-source Spark 2.4.7 Release Notes.

These release notes contain only HPE-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 2.4.7.200
Release Date March 2022
MapR Version Interoperability See Component Versions for Released EEPs and EEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.4.7.200-mapr-712
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names.
IMPORTANT
  • Beginning with EEP 6.0.0, the keyStore and trustStore password can be removed from spark-defaults.conf and set in /opt/mapr/conf/ssl-client.xml.
  • Beginning with EEP 6.0.0, after an upgrade, the previous version's configuration files are saved in the /opt/mapr/spark directory.
  • MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your data-fabric cluster, HPE scripts automatically configure Spark security features.
  • Beginning with Core 6.2 and EEP 7.0, Spark supports SSL for WebUI.

Hive Support

This version of Spark supports integration with Hive, but has the following exceptions:

New in This Release

Fixes

This HPE release includes the following new fixes since the latest data-fabric Spark release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Number Date (YYYY/MM/DD) HPE Fix Number and Description
54fc6c7 29/12/2021 MapR [SPARK-975] Spark CVE fixes for Jan 2022 release
cd2d2c0 06/01/2022 MapR [SPARK-986] log4j-1.2.17.jar vulnerability:CVE-2019-17571
4ab6ab0 14/01/2022 MapR [SPARK-994] Update jackson-mapper-asl v1.9.13 to 1.9.13-atlassian-5
0e59d27 25/01/2022 MapR [SPARK-1001] Update log4j v1 to the 1.3.1-mapr

Known Issues

  • If you are using Spark SQL with Derby database without Hive or Hive Metastore installation, you will see the Java Runtime Exception. See Apache Spark Feature Support for workaround. Spark does not support log4j1.2 logging on HPE Ezmeral Data Fabric.

  • SPARK-865: If you run a configure.sh command to configure HBase after Spark configuration, then you must manually copy the hbase-site.xml configuration file from the HBase configuration directory to the Spark configuration directory.

Resolved Issues

  • None.