Spark - 2104 (EEP 7.1.0) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark

The notes below relate specifically to the HPE Ezmeral Data Fabric Distribution for Apache Hadoop. For more information, you may also wish to consult the open-source Spark 2.4.7 Release Notes.

These release notes contain only HPE-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version
Release Date April 2021
MapR Version Interoperability See Component Versions for Released EEPs and EEP Components and OS Support.
Source on GitHub
GitHub Release Tag
Maven Artifacts
Package Names Navigate to and select your EEP and OS to view the list of package names.
  • Beginning with EEP 6.0.0, the keyStore and trustStore password can be removed from spark-defaults.conf and set in /opt/mapr/conf/ssl-client.xml.
  • Beginning with EEP 6.0.0, after an upgrade, the previous version's configuration files are saved in the /opt/mapr/spark directory.
  • MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your data-fabric cluster, HPE scripts automatically configure Spark security features.
  • Beginning with Core 6.2 and EEP 7.0, Spark supports SSL for WebUI.

Hive Support

This version of Spark supports integration with Hive, but has the following exceptions:

New in This Release


This HPE release includes the following new fixes since the latest data-fabric Spark release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Number Date (YYYY/MM/DD) HPE Fix Number and Description
ea8a479 2021/01/27 [SPARK-32723][WEBUI] Upgrade to jQuery 3.5.1
98ced49 2021/02/02 MapR [SPARK-701] CVE-2019-17571 security vulnerability for for log4j-1.2.17
a2b10a6 2021/02/08 MapR [SPARK-845] Update hadoop dependency for Spark-2.4.7
0dc3f73 2021/02/16 MapR [SPARK-843] Improve logging for SSL certs generation
ed0be85 2021/02/16 MapR [K8S-2129] Spark ignores impersonation in K8s
759896a 2021/02/17 MapR [SPARK-847] Spark can't read data from symlink
ea1a325 2021/03/01 MapR [K8S-2215] backport mapr-specific k8s commits from 2.4.5
584881f 2021/03/19 MapR [SPARK-846] Add service verifier to Spark package
127da25 2021/03/24 MapR [SPARK-856] Update Spark dependencies to latest artifacts for MEP-7.0.1 release
9d1e83d 2021/03/29 MapR [SPARK-861] Error logs in spark-historyserver
2ffdd06 2021/03/29 MapR [SPARK-859] Excessive logs for spark job
8b7abe5 2021/03/31 MapR [SPARK-863] Interaction with HBase via hbase spark connector fails
6b9d305 2021/04/05 MapR [SPARK-841] Backport SPARK-32723
ea9fc35 2021/04/08 MapR [SPARK-851] Spark SQL transient FS error handling when writing output
6e2aff7 2021/04/13 MapR [SPARK-869] fix logging location
976b271 2021/04/13 MapR [SPARK-867] Spark Hive Example fails from simple user
2bcc0c0 2021/04/13 MapR [SPARK-870] Can't download event logs from SHS twice

Known Issues

  • SPARK-865: If you run a command to configure HBase after Spark configuration, then you must manually copy the hbase-site.xml configuration file from the HBase configuration directory to the Spark configuration directory.

Resolved Issues

  • None.