Hadoop 2.7.6.100 - 2110 (EEP 8.0.0) Release Notes

The notes below relate to the HPE Ezmeral Data Fabric distribution of Apache Hadoop.

These release notes contain only HPE-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Version 2.7.6.100
Release Date October 2021
Version Interoperability See EEP Components and OS Support.
GitHub Source Not applicable
GitHub Release Tag 2.7.6.100-eep-800
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names.
Documentation

New in this Release

Hadoop 2.7.6.100 - 2110 introduces the following enhancements or HPE platform-specific behavior changes:
  • All fixes from Apache Hadoop 2.7.6 have been backported.
  • Support for file exclusion list in DistCp.
  • Added a yarn top tool.
  • Support for BCFKS keystores for Hadoop Credential Provider.

Fixes

This HPE release includes the following fixes on the base Apache release:
GitHub Commit Number Date (YYYY-MM-DD) HPE Fix Number and Description
4d102084877 2021-09-13 MAPRHADOOP-200: Load ssl configuration after core-site.
48960d6b6e5 2021-09-01 MAPRYARN-323-325: Added BC provider to Hadoop services
b808c225390 2021-09-01 Backport HADOOP-17699. Remove hardcoded SunX509 usage from SSLFactory. (#3016)
ca635e0db98 2021-09-01 MAPRHADOOP-192: Exclude bc-fips jars from Hadoop build
0dc49e773b8 2021-09-01 Backport HADOOP-13011 - Clearly Document the Password Details for Keystore-based Credential Providers
5400bc53028 2021-09-01 Backport HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334)
c4ef4f0f162 2021-09-01 Backport HADOOP-12942. hadoop credential commands non-obviously use password of "none" (Mike Yoder via lmccay)
b93bcdf2592 2021-08-18 MAPRHADOOP-194: Update commons-compress to 1.21
4f1c4ec9828 2021-08-12 MAPRHADOOP-190: update htrace to 4.2.0-mapr-incubating
dc9fce5dd8f 2021-08-04 MAPRMR-21: ControlledJob#toString failed with NPE when job status is not successfully updated
f267f098be2 2021-08-04 Backport MAPREDUCE-6762. ControlledJob#toString failed with NPE when job status is not successfully updated (Weiwei Yang via Varun Saxena)
e0732a00771 2021-07-28 MAPRHADOOP-186: Update Jetty to 9.4.43.v20210629
a30951764be 2021-07-27 MAPRMR-20: add retry support for all Job actions
a42503a9153 2021-07-26 Update gradle-wrapper.properties
1d04de14c09 2021-07-26 Backport MAPRYARN-321: Upgraded Avro version to 1.10.1
2b2518f33a8 2021-07-23 Backport HADOOP-17341. Upgrade commons-codec to 1.15 (#2428)
b6988af1234 2021-07-20 Backport HADOOP-10075. addendum to fix compilation on Windows
7d0183ee3bd 2021-06-21 MAPRYARN-320: Kill child process from container-executor
ad8e8d08c0b 2021-06-18 Backport YARN-10490. yarn top command not quitting completely with ctrl+c. Contributed by Agshin Kazimli
1c737ee2f90 2021-06-18 MAPRYARN-316: Added disks to "yarn top" command
a35adb43f87 2021-06-18 Backport YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev
9f5418a1de5 2021-06-07 Backport HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein.
f3b647eea98 2021-06-04 Fixed LocatedFileStatusFetcher for work with symlinks
9b16bf0084b 2021-06-03 MAPRHADOOP-177: Fixed mkdir command for non existing parent dir
e96a8d7e003 2021-06-02 MAPRHADOOP-174: Fixed issue with copying multiple files to directory
740035b02f1 2021-05-31 Backport HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen.
27e38cbbb28 2021-05-31 Backport HADOOP-12862. LDAP Group Mapping over SSL can not specify trust store. Contributed by Wei-Chiu Chuang and Konstantin Shvachko.
290a908be21 2021-05-31 Backport HADOOP-15345. Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient adding/getting/removing nodes. Contributed by He Xiaoqiao.
532934140ad 2021-05-31 Backport HDFS-13195. DataNode conf page cannot display the current value after reconfig. Contributed by maobaolong.
f9918ab5550 2021-05-31 Backport HDFS-12884. BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo. Contributed by chencan.
1c3f997eb9b 2021-05-31 Backport HADOOP-13105. Support timeouts in LDAP queries in LdapGroupsMapping. Contributed by Mingliang Liu.
209c93fbe66 2021-05-31 Backport HADOOP-12001. Fixed LdapGroupsMapping to include configurable Posix UID and GID attributes during the search. Contributed by Patrick White.
fdb6bf9b675 2021-05-31 Backport HADOOP-15279. increase maven heap size recommendations
444cef90d29 2021-05-31 Backport HADOOP-15283. Upgrade from findbugs 3.0.1 to spotbugs 3.1.2 in branch-2 to fix docker image build.
a156fd13c95 2021-05-31 Backport HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot be resolved. Contributed by Charles Lamb and John Zhuge.
79158f841e8 2021-05-31 Backport HADOOP-15206. BZip2 drops and duplicates records when input split size is small. Contributed by Aki Tanaka
93547c2cb3b 2021-05-31 Backport HADOOP-12568. Update core-default.xml to describe posixGroups support. Contributed by Wei-Chiu Chuang.
28283d0740c 2021-05-31 Backport HADOOP-12793. Write a new group mapping service guide (Wei-Chiu Chuang via iwasakims)
a6aef9fa3fa 2021-05-31 Backport HADOOP-9477. Add posixGroups support for LDAP groups mapping service. (Dapeng Sun via Yongjun Zhang)
2b0f6773337 2021-05-31 Backport MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contr ibuted by Peter Bacsko
ac06e483c05 2021-05-31 Backport HDFS-10453. ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.. Contributed by He Xiaoqiao.
384340b76ed 2021-05-31 Backport HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee)
953f75abed0 2021-05-31 Backport HDFS-13120. Snapshot diff could be corrupted after concat. Contributed by Xiaoyu Yao.
8bc1c2ecbaa 2021-05-31 Backport HADOOP-15212. Add independent secret manager method for logging expired tokens. Contributed by Daryn Sharp.
f73da507697 2021-05-31 Backport MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko
ef3f06d320b 2021-05-31 Backport HDFS-12371. BlockVerificationFailures and BlocksVerified show up as 0 in Datanode JMX. Contributed by Hanisha Koneru.
7570e0e491d 2021-05-31 Backport HADOOP-13508. FsPermission string constructor does not recognize sticky bit. Contributed by Atul Sikaria.
27e4c5fe92e 2021-05-31 Backport HDFS-11003. Expose XmitsInProgress through DataNodeMXBean. Contributed By Brahma Reddy Battula
a537ad3ea2d 2021-05-31 Backport HADOOP-13263. Reload cached groups in background after expiry. (Contributed bt Stephen O'Donnell)
358d2fd6795 2021-05-31 Backport HADOOP-14246. Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets (Conttributed by Robert Kanter via Daniel Templeton)
fb29d47034b 2021-05-31 Backport HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc queue saturation. Contributed by Konstantin V Shvachko.
442c573b4e6 2021-05-31 Backport MAPREDUCE-7028. Concurrent task progress updates causing NPE in Application Master. Contributed by Gergo Repas
ead4809bbc1 2021-05-31 Backport HDFS-12881. Output streams closed with IOUtils suppressing write errors. Contributed by Ajay Kumar
f7842437eff 2021-05-31 Backport MAPREDUCE-5124. AM lacks flow control for task events. Contributed by Peter Bacsko
9316e6866df 2021-05-31 Backport YARN-4167. NPE on RMActiveServices#serviceStop when store is null. (Bibin A Chundatt via rohithsharmaks) Backport YARN-6633 by Inigo Goiri.
573618f058e 2021-05-31 Backport YARN-3425. NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed. (Bibin A Chundatt via wangda) Backport YARN-6632 by Inigo Goiri.
38565378f4f 2021-05-31 Backport YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang
cb027e006c4 2021-05-28 Backport MAPRHADOOP-174: Added wildcard usage for path with symlinks
ffc31a97594 2021-05-26 Backport YARN-4348. ZKRMStateStore.syncInternal shouldn't wait for sync completion for avoiding blocking ZK's event thread. (ozawa)
226dfdeca76 2021-05-26 Backport YARN-3798. ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED. (ozawa and Varun Saxena)
4f4bf8a1446 2021-05-20 MAPRHADOOP-173: Moving cluster from secure to unsecure breaks yarn-site.xml
6a83c9d9918 2021-05-19 COMSECURE-384: Add Bouncy Castle JARs to Hadoop class path - unitTested
fa833c7d5bb 2021-05-17 MAPRHADOOP-171: Distcp to S3A does not work
ff9e411bb5f 2021-05-13 MAPRHADOOP-172: configure ssl-client/server.xml as part of hadoop-util configuration
da5ef438a37 2021-05-11 Backport HADOOP-15970. Upgrade plexus-utils from 2.0.5 to 3.1.0.
4e64e475366 2021-05-06 MAPRYARN-299: change service name identification approach
6d453dc22df 2021-04-28 MAPRHADOOP-169: Updated JQuery to 3.5.1 for SLS
2ac48dc37c5 2021-04-28 Backport HADOOP-17302. Upgrade to jQuery 3.5.1 in hadoop-sls. (#2379)
32e551d6ae2 2021-04-28 Backport HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory
8394c1d02c1 2021-04-28 Backport HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory.
1c36296b9e8 2021-04-28 Backport HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong.
eed76986ecd 2021-04-26 Backport HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase
912a88cdc84 2021-04-20 MAPRHADOOP-167: jackson-mapper-asl updated to 1.9.13-atlassian-5

Known Issues and Limitations

  • None.

Resolved Issues

  • None.