Spark 3.1.2.0 - 2110 (EEP 8.0.0) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 3.1.2.0.

The notes below relate specifically to the HPE Ezmeral Data Fabric Distribution for Apache Hadoop. For more information, you may also want to consult the open-source Spark 3.1.2 Release Notes.

These release notes contain only HPE-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 3.1.2.0
Release Date October 2021
MapR Version Interoperability See Component Versions for Released EEPs and EEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag

3.1.2.0-eep-800
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names.
IMPORTANT
  • Beginning with EEP 6.0.0, the KeyStore and TrustStore password can be removed from spark-defaults.conf and set in /opt/mapr/conf/ssl-client.xml.
  • Beginning with EEP 6.0.0, after an upgrade, the previous version's configuration files are saved in the /opt/mapr/spark directory.
  • MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your data-fabric cluster, HPE scripts automatically configure Spark security features.
  • Beginning with Core 6.2 and EEP 7.0, Spark supports SSL for WebUI.

Hive Support

  • Starting from Spark 3.1.2, Spark supports Hive 2.3.

Delta Lake Support

Starting from EEP 8.0.0, Delta Lake support is available for Apache Spark 3.1.2 on HPE Ezmeral Data Fabric. See Apache Spark Feature Support.

New in This Release

Fixes

This HPE release includes the following new fixes since the latest Spark release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Number Date (YYYY/MM/DD) HPE Fix Number and Description
22b57ce 2021/07/29 MapR [SPARK-811] Updating to Spark 3.x
bfd5a1a 2021/07/29 MapR [SPARK-839] Unstable problem with security keys for job on yarn-cluster mode
9045b8f 2021/07/29 [SPARK-32723][WEBUI] Upgrade to jQuery 3.5.1
c4c0808 2021/07/29 MapR [SPARK-843] Improve logging for SSL certs generation
3cf3522 2021/07/29 MapR [SPARK-847] Spark can't read data from symlink
2f6cbe1 2021/07/29 MapR [SPARK-846] Add service verifier to Spark package
f11cc5a 2021/07/29 MapR [SPARK-861] Error logs in spark-historyserver
711eb00 2021/07/29 MapR [SPARK-863] Interaction with HBase via hbase spark connector fails
bd65fe6 2021/07/29 MapR [SPARK-841] Backport SPARK-32723
213226d 2021/07/29 MapR [SPARK-851] Spark SQL transient FS error handling when writing output
6f33abe 2021/07/29 MapR [SPARK-869] fix logging location
41ac719 2021/07/29 MapR [SPARK-867] Spark Hive Example fails from simple user
b234c78 2021/07/29 MapR [SPARK-870] Can't download event logs from SHS twice
df7dbfe 2021/07/29 MapR [SPARK-871] Spark job fails from mapr-client
165644d 2021/07/29 MapR [SPARK-846] Add service verifier to Spark package - moving to proper directory
92e3441 2021/07/29 MapR [SPARK-877] Update Jenkins file to build Spark-3.x from MEP-8.0.0 private package branch
42b5f36 2021/07/29 MapR [SPARK-881] fix duplicate heade
9c75ccb 2021/07/29 MapR [SPARK-879] Add changes to examples module to build for Spark-3.x
0282515 2021/07/29 MapR [SPARK-883] Spark-3.1.1 job submition
a87034c 2021/07/29 MapR [SPARK-885] Spark-submit fails on 8.2 and 8.3 centos
9d7de9e 2021/07/29 MapR [SPARK-888] Collect of selected result from orc table fails
b80ea8e 2021/07/29 MapR [SPARK-893] Clean deprecated kafka08 and kafka09 from pyspark code
c26d17c 2021/07/29 MapR [SPARK-891] Thrift server start fails due to unsupported Hive Metastore version
28a492a 2021/07/29 MapR [SPARK-897] Spar-3.1.1 doesn't suppot MapRSASL for Thriftserver
3b7064d 2021/07/29 MapR [SPARK-901] Spark-3.1.1 doesn't start by warden
37bc0ac 2021/07/29 MapR [SPARK-903] spark.loadFromMapRDB(tableName, schema) using v2 api fail
568f406 2021/07/29 MapR [SPARK-817] Update kafka client to v2.6.X
8b29cfa 2021/07/29 MapR [SPARK-886] MapRDB table loading fails via spark session + java api
e86c244 2021/07/29 MapR [SPARK-907] Update hadoop dependency for Spark-3.1.1
9ed986b 2021/07/29 MapR [SPARK-905] Can't connect to spark thriftserver via beeline with MAPRSASL
1422b69 2021/07/29 MapR [SPARK-904] Implement inferSchema method for MaprDBDataSource
c4a1990 2021/07/29 [EZSPA-212] Move creating of spark-env.sh script to Spark
078555a 2021/07/29 [SPARK-22769] Do not log rpc post message error when sparkEnv is already stopped
a8cb292 2021/07/29 [EZSPA-213] Add Spark-3.x to dockerfiles project
5e1ddd3 2021/08/02 MapR [SPARK-917] Porting Spark-3.1.2 to MapR
2c5d494 2021/08/09 MapR [SPARK-922] Move latest Spark commits to 3.1.2 branch
2f6b259 2021/08/10 MapR [SPARK-882] Making netcat to work on Ubuntu too
42328a6 2021/08/12 MapR [SPARK-878] remove redundant filter setting
13f8e12 2021/08/17 MapR [SPARK-919] Errors in spark DEBUG logs
605d290 2021/08/18 MapR [SPARK-927] Update Hive in Spark-3.1.2
1b188e3 2021/08/19 MapR [SPARK-923] Update Avro to 1.10.1 in Spark
6f8abad 2021/08/19 MapR [SPARK-915] CVE-2020-13956,WS-2017-3734 vulnerabilities in http-client
75d47c5 2021/08/24 MapR [SPARK-906] Spark streaming (structured and unstructured) fails with kafka 2.6.1.0
09f6deb 2021/08/27 MapR [SPARK-911] STS HA doesn't work
8252a91 2021/09/01 MapR [SPARK-928] Can't connect to spark thriftserver on kerberos cluster
dba8119 2021/09/03 MapR [SPARK-929] Spark 3.1.2 requires password when you try to remove packages
c942759 2021/09/06 MapR [SPARK-882] [Installer] Add verification scripts for spark-thrifserver and spark-historyserver
31f4b86 2021/09/06 MapR [SPARK-936] Investigate Spark warning on start of application
050309c 2021/09/06 MapR [SPARK-938] Run-example doesn't work
334bc98 2021/09/07 MapR [SPARK-919] Errors in spark DEBUG logs
354abf87 2021/09/09 [EZSPA-270] adopt mapr spark feature to work in non-mapr env
718fa44 2021/09/10 MapR [SPARK-939] Replace "mapr" to "eep" in Spark package
ac7871c 2021/09/10 MapR [SPARK-934] Spark and Livy jobs fail on core 7.0.0 with encrypted ssl password
a6b59d0 2021/09/20 MapR [SPARK-943] Spark 3 and S3 integration fails
4a04f88 2021/09/20 MapR [SPARK-945] Components can't read keyPassword
a5cb0b8 2021/09/21 MapR [SPARK-894] Hadoop artifacts should be taken from the cluster
e8c8667 2021/09/22 MapR [SPARK-946] Excessive warning messages in spark-shell
715edb7 2021/09/22 MapR [SPARK-947] Error when using Spark SQL with derby db

Known Issues

  • If you are using Spark SQL with Derby database without Hive or Hive Metastore installation, you will see the Java Runtime Exception. See Apache Spark Feature Support for workaround. Spark 3.1.2 does not support log4j1.2 logging on HPE Ezmeral Data Fabric.

  • HPE Ezmeral Data Fabric does not support GPU aware scheduling feature on Spark 3.1.2. See Apache Spark Feature Support.

Resolved Issues

  • None.