Spark 1.6.1-1703 Release Notes

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 1.6.1 Release Notes.

Spark Version 1.6.1
Release Date April 2017
MapR Version Interoperability See EEP Components and OS Support.
Source on GitHub
GitHub Release Tag 1.6.1-mapr-1703
Maven Artifacts
Package Names See Package Names for Ecosystem Packs (EEPs)
NOTE For some important Spark limitations, See "Known Issues and Limitations" later in this release note.

New in This Release

This version of Spark supports integration with Hive. However, note the following exceptions:


This MapR release includes the following new fixes since the latest MapR Spark 1.6.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) MapR Fix Number and Description
9275605 2017/03/16 Fixed gap handling.
dbf3b16 2017/03/10 Removed overridden methods count, countApprox, isEmpty from kafkaRDD.
3014abc 2017/03/07 [MAPR-26040] Kafka streaming module was refactored (#96).
62def95 2017/03/06 [BUILD][1.6] Fix compilation.
e092b35 2017/02/23 Merge pull request #90 from mapr/spark-1.6.3-porting.
04cbcb0 2017/02/23 [SPARK-17696][SPARK-12330][CORE] Partial backport of to branch-1.6.
9be6167 2017/02/13 [SPARK-14357][CORE] Properly handle the root cause being a commit denied exception.
566a5a9 2017/02/13 [SPARK-16182][CORE] Utils.scala -- terminateProcess() should call Process.destroyForcibly() if and only if Process.destroy() fails.
31843fc 2017/02/13 [SPARK-17618] Fix invalid comparisons between UnsafeRow and other row formats.
2781e0b 2017/02/13 [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort.

Known Issues and Limitations

  • To integrate Spark 1.6.1 with HPE Ezmeral Data Fabric Streams, you must install the latest Kafka package.
  • Full support of HPE Ezmeral Data Fabric Streams is available only on clusters with MapR 5.2 and later.
  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
  • MAPR-19761: On a secure cluster, MapR software does not support the Spark SQL Thrift JDBC server. When the cluster is secure, the Spark Thrift server will not start.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability

Resolved Issues