Spark 2.4.4-2009 (EEP 6.3.1) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.4.4.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. For more information, you may also wish to consult the open-source Spark 2.4.4 Release Notes.

These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 2.4.4
Release Date September 2020
MapR Version Interoperability See Component Versions for Released EEPs and EEP Components and OS Support.
Source on GitHub
GitHub Release Tag 2.4.4-mapr-2009
Maven Artifacts
Package Names Navigate to and select your EEP and OS to view the list of package names.
  • Beginning with EEP 6.0.0, the keyStore and trustStore password can be removed from spark-defaults.conf and set in /opt/mapr/conf/ssl-client.xml.
  • Beginning with EEP 6.0.0, after an upgrade, the previous version's configuration files are saved in the /opt/mapr/spark directory.
  • MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
  • Beginning with EEP 6.3.0, the Spark MapRDB JSON connector supports secondary indexes.
  • Beginning with EEP 6.3.0, Spark supports configurable HTTP security headers.
  • Beginning with MapR 6.2 and EEP 7.0, Spark supports SSL for WebUI.

Hive Support

This version of Spark supports integration with Hive, but has the following exceptions:

New in This Release


This MapR release includes the following new fixes since the latest MapR Spark release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
e5a8ff7 2020/01/09 MapR [SPARK-642] Improve work with security headers
13584a9 2020/01/13 MapR [SPARK-642] Improve work with security headers
f6f14db 2020/01/15 MapR [SPARK-654] Update protobuf version to 3.X.X for Spark
db6c15f 2020/01/21 MapR [SPARK-585] Sertificates generation for cluster mode
4e5d6fd 2020/02/10 [SPARK-27992][SPARK-28881][PYTHON][2.4] Allow Python to join with connection thread to propagate errors
79366d5 2020/02/11 [SPARK-29410][BUILD] Update commons-beanutils to 1.9.4
b05a4ae 2020/02/18 MapR [SPARK-662] Spark job is failed with updated protobuf version
2c7467b 2020/02/18 MapR [SPARK-664] commons-beanutils as dependency for mapr-spark
d95867e 2020/03/03 MapR [SPARK-677] No TLS protocol logging for Spark thrift server & spark history server
5a9b5f8 2020/04/07 [SPARK-30238][SQL][2.4] hive partition pruning can only support string and integral types
add846c 2020/04/08 MapR [SPARK-691] maprdb-spark jar MetaTableImpl class reference in DBOlderClientImpl
8259253 2020/04/08 MapR [SPARK-705] Spark streaming fail with NoSuchMethodError: org.apache.kafka.clients.producer.ProducerConfig.getBoolean(Ljava/lang/String;)Ljava/lang/Boolean;
8c4e21c 2020/04/19 MapR [SPARK-717] Spark Master start fail after installation due to
61b7376 2020/04/27 MapR [SPARK-724] Application startup fail with java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
eec85e1 2020/04/28 MapR [SPARK-715] Driver https port bind fail when application starts under non-mapr user
e28302b 2020/04/30 [SPARK-29445][CORE] Bump netty-all from 4.1.39.Final to 4.1.42.Final
3f8b017 2020/05/19 MapR [SPARK-740] UI shows "Kafka 0.9 direct stream" even if 0.10 is used
f014001 2020/05/20 MapR [SPARK-737] Calling poll(1000) from paranoidPoll causes batch scheduling delay
ccda955 2020/05/22 MapR [SPARK-738] CLONE - Check if jackson-databind-2.9.4.jar will work with Spark 2.4.X in MEP 6 and MEP7
5e30e36 2020/05/22 [SPARK-26966][ML] Update to JPMML 1.4.8
fbc0634 2020/05/22 MapR [SPARK-714] Update bcprov-jdk15on to fix CVE issue
3348810 2020/05/27 [K8S-1458] Spark 2.4.4 encounters error and does not start executors
2b903c6 2020/06/12 MapR [SPARK-746] Secondary index not picked
8154afe 2020/06/16 MapR [SPARK-751] Spark Jetty servers send webserver details to client
e8afcde 2020/06/25 [CORE-459] MapRDB spark connector shows incorrect results when Secondary Index enabled on fields
e74a28e 2020/07/01 [SPARK-29055][CORE] Update driver/executors' storage memory when block is removed from BlockManager
79912a9 2020/07/17 MapR [SPARK-767] Backport SPARK-29444 into Spark 2.4.4
432729b 2020/08/19 MapR [SPARK-781] SSL for spark UI services
ee0107a 2020/08/28 MapR [SPARK-759] Allow PAM and MapR-Sasl to be enabled at the same time

Known Issues

  • None.

Resolved Issues

  • None.