Hive 2.3.9 - 2110 (EEP 8.0.0) Release Notes

The following notes relate specifically to the HPE Data Fabric Distribution for Apache Hive. You may also be interested in the Apache Hive-2.3.9 Release Notes and the Apache Hive homepage.
Hive Version 2.3.9
Release Date October 2021
HPE Version Interoperability See Hive and HCatalog Support Matrix and Ecosystem Support Matrix and Ecosystem Pack Components and OS Support.
Source on GitHub https://github.com/mapr/hive
GitHub Release Tag 2.3.9-eep-2110
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to http://package.ezmeral.hpe.com/releases/MEP/, and select your EEP(MEP) and OS to view the list of package names.
ODBC/JDBC Drivers
Hive 2.3.9 works with the following HPE Hive drivers:

For additional driver information, see Connecting to HiveServer2.

Feature support

The following list describes support of various components and functionality with Hive 2.3.9 - 2110:

  • Supports Hive-2.3.9 on Tez-0.9.2 For more information, see Tez 0.9.2 - 2110 (EEP 8.0.0) Release Notes.

  • Does not support Hive on Spark. You cannot use Spark as a query engine for Hive. However, you can run Hive and Spark on the same cluster. You can also use Spark SQL and Drill to query Hive tables.

  • Does not support HDFS encryption in Hive tables.

  • Does not support LLAP with Hive-2.3.9 because Apache Slider is not an HPE supported ecosystem component.

  • Starting from Hive 2.1, Hive must run the schematool command as an initialization step.

Changes in default security configuration

The following list describes changes in default security for Hive 2.3.9 - 2110:
  • None.

New in This Release

Hive 2.3.9 - 2110 introduces the following enhancements or HPE platform-specific behavior changes:

  • Improved Describe table operator in terms of fetching statistics of partitions. Starting from EEP 8.0.0, you can fetch the partition information using the describe command with formatted or extended statements.
    Configure the hive.describe.partitionedtable.ignore.stats property to change the behaviour of fetching statistics of partitions. It is set to the default value of false.
    <property>
    <name>hive.describe.partitionedtable.ignore.stats</name>
    <value>false</value>
    <description>Enables partitioned table stats collection for 'DESCRIBE FORMATTED' or 'DESCRIBE EXTENDED' commands</description>
</property>
    <property>
    <name>hive.describe.partitionedtable.ignore.stats</name>
    <value>true</value>
    <description>Disables partitioned table stats collection for 'DESCRIBE FORMATTED' or 'DESCRIBE EXTENDED' commands</description>
 </property
  • Hive supports symbolic links on file system. See Hive Features in HPE Data Fabric.

Fixes

This HPE release includes the following fixes on the base release:
GitHub Commit Number Data (YYYY-MM-DD) HPE Fix Number and Description
2549e5350a 2021-09-17 MAPR-HIVE-994: Non mapr user unable to read SSL configuration from XML files on Core 7.0
e28a1edd63 2021-09-14 MAPR-HIVE-1007: Permission denied to hbase temp files while running hcat jobs from other user
c2864e3d06 2021-09-08 MAPR-HIVE-999 : Make mapr-db jar with provided scope
705bab5a12 2021-09-06 MAPR-HIVE-998 : Update htrace version to 4.2.0-eep-incubating
46862cf932 2021-09-03 MAPR-HIVE-990 : mapr-security-web jar should be taken from the cluster
c4aed1b675 2021-09-03 MAPR-HIVE-995 : Upadate pig vesion to 0.17.0.0-eep-SNAPSHOT
7e012fa650 2021-09-03 MAPR-HIVE-993 : Update hbase version to 1.4.13.0-eep-SNAPSHOT
1a886bbe78 2021-09-03 MAPR-HIVE-992 : Update tez version to 0.9.2.0-eep-SNAPSHOT
1439299a5c 2021-09-03 MAPR-HIVE-991 : Update hadoop version to 2.7.6.0-eep-800-SNAPSHOT
09724c6192 2021-09-03 MAPR-HIVE-987 : Update the maven artifact version strings to eep
6dbd5a3ac8 2021-08-25 MAPR-HIVE-981: [symlink functionality] Implement LOAD DATA INPATH functionality from symlinks with relative path
29c2c6f3ab 2021-08-18 MAPR-HIVE-979: [symlink functionality] cannot insert in the external table based on symlinked directory
1073cecb48 2021-08-17 MAPR-HIVE-977 : Downgrade jackson to v2.11.1 or to 2.11.3 to be consistent with core version
13ec2228e5 2021-08-17 MAPR-HIVE-976 : Update tez version from 0.9.2-mapr-SNAPSHOT to 0.9.2.0-mapr-SNAPSHOT for development artifacts
bfc754f1ca 2021-08-11 MAPR-HIVE-973 : FAILURE! - in org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
7add0d42af 2021-08-09 MAPR-HIVE-972 : Replace Apache htrace-4.2.0-incubating with 4.2.0-mapr-incubating dependency
ad1eeb8b20 2021-08-09 MAPR-HIVE-971 : Exclude htrace-3.1
bcddd3cc00 2021-08-09 MAPR-HIVE-969: Add possibility to run MR jobs against source files that are symlinks to original data
74302e7f0d 2021-08-09 MAPR-HIVE-970 : Update hadoop version to 2.7.6.0-mapr-720-SNAPSHOT
f6953ff0a8 2021-08-05 MAPR-HIVE-968: Add possibility to run TEZ jobs against source files that are symlinks to original data
8186a55978 2021-07-29 MAPR-HIVE-880: Add possibility to distinguish file/dir links during Hive DML/DDL operations
10779d1b56 2021-07-29 MAPR-HIVE-960 : CVE-2012-5783 vulnerability in commons-httpclient
0be9010773 2021-07-29 MAPR-HIVE-959 : Update derbyclient and derbynet to most feasible version
d0825bf854 2021-07-29 MAPR-HIVE-963 : CVE-2020-13956,WS-2017-3734 vulnerabilities in httpclient
7982b4be51 2021-07-29 MAPR-HIVE-962 : WS-2019-0379: commons-codec vulnerability
d70a411074 2021-07-29 MAPR-HIVE-953 : FAILURE! - in org.apache.hive.hcatalog.templeton.TestCustomHeadersE2e
88056bb162 2021-07-29 MAPR-HIVE-952 : FAILURE! - in org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader
054a0b73b2 2021-07-29 MAPR-HIVE-896 : CVE-2020-17521 vulnerability in Groovy
5b44866042 2021-07-29 MAPR-HIVE-927 : NPE thrown from XmlUtil by Hive Client
2e446568b2 2021-07-29 MAPR-HIVE-858 : WARNING: Illegal reflective access org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer
b3520d3c85 2021-07-29 MAPR-HIVE-950 : HiveVersionInfo.getShortVersion returns wrong version
9b4c3ad944 2021-06-08 MAPR-HIVE-949 : Make SchemaEvolution class behavior the same as in Orc-1.5.12
9b75385d22 2021-06-06 MAPR-HIVE-894 : CVE-2020-13955 vulnerability in Calcite
2a7526fdea 2021-05-25 MAPR-HIVE-947 : org.apache.hadoop.hive.ql.exec.tez.TezTask at creating session
a64ce8195c 2021-05-23 MAPR-HIVE-945 : FAILURE! - in org.apache.hadoop.hive.maprdb.json.MapRDbJsonFetchByIdOptimizerPositiveTest
4215a9ab82 2021-05-23 MAPR-HIVE-944 : FAILURE! - in org.apache.hadoop.hive.hbase.TestHBaseSerDe
30655745a5 2021-05-23 MAPR-HIVE-943 : Fix org.apache.hadoop.hive.cli.TestCliDriverMethods
bfda9cc9a8 2021-05-23 MAPR-HIVE-942 : FAILURE! - in org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler
This release from HPE also includes the following back-ported issues. For complete details, refer to the commit log for this project in GitHub.
GitHub Commit Number Data (YYYY-MM-DD) HPE Fix Number and Description
3a7b6db040 2021-09-15 HIVE-24965: Describe table partition stats fetch should be configurable
1d1bb4c2fd 2021-09-15 HIVE-22453: Describe table unnecessarily fetches partitions
96b37ab69c 2021-09-14 HIVE-23756 : Added more constraints to the package.jdo file
40cdec9dd2 2021-09-12 HIVE-24177 : hive mapjoin throws udf class not found
606c3c240b 2021-09-08 HIVE-17659 : get_token thrift call fails for DBTokenStore in remote HMS mode (Vihang Karajgaonkar, reviewed by Aihua Xu)
a2416a115c 2021-09-06 HIVE-25054: Upgrade `jodd-core` dependency to get rid of CVE-2018-21234 (Abhay Chennagiri, reviewed by Jesus Camacho Rodriguez)
e593c8cb4e 2021-08-18 HIVE-17824 : msck repair table should drop the missing partitions from metastore (Janaki Lahorani, reviewed by Peter Vary, Alexander Kolbasov and Vihang Karajgaonkar)
3591ea65fa 2021-08-18 HIVE-16143: Improve msck repair batching (Vihang Karajgaonkar, reviewed by Sahil Takiar & Aihua Xu)
fab9a7603a 2021-07-29 HIVE-19228: Remove commons-httpclient 3.x usage (Janaki Lahorani reviewed by Aihua Xu)
0ffeae33b1 2021-06-13 HIVE-21200: Vectorization: date column throwing java.lang.UnsupportedOperationException for parquet (#2276)
c6300400bd 2021-06-13 HIVE-24608: Switch back to get_table in HMS client for Hive 2.3.x (#2080)
0518323174 2021-06-13 HIVE-18147 : Tests can fail with java.net.BindException: Address already in use (Janaki Lahorani, reviewed by Andrew Sherman and Vihang Karajgaonkar)
d6766f34fb 2021-06-13 HIVE-21563 : Improve Table#getEmptyTable performance by disable registerAllFunctionsOnce
a3477edb7f 2021-06-13 HIVE-24797: Disable validate default values when parsing Avro schemas (#1994)
1fc7585a2e 2021-06-08 ORC-437: Make acid schema checks case insensitive
9120da5c4f 2021-05-31 HIVE-21075 : Metastore: Drop partition performance downgrade with Postgres DB
39d42ddf12 2021-05-31 HIVE-9447: Metastore: inefficient Oracle query for removing unused column descriptors when add/drop table/partition (Selina Zhang reviewed by Ashutosh Chauhan, Adam Szita)
1ccb218119 2021-05-22 HIVE-21085: Materialized views registry starts non-external tez session (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
dea7190511 2021-05-22 HIVE-19691: Start SessionState in materialized views registry (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
09b4ca437f 2021-05-22 HIVE-17853: RetryingMetaStoreClient loses UGI impersonation-context when reconnecting after timeout (Chris Drome, reviewed by Mithun Radhakrishnan)
b8902a7bb8 2021-05-22 HIVE-23534: NPE in RetryingMetaStoreClient#invoke when catching MetaException with no message (Stamatis Zampetakis, reviewed by Jesus Camacho Rodriguez)
11db00d681 2021-05-22 HIVE-18494: Regression: from HIVE-18069, the metastore directsql is getting disabled (Jesus Camacho Rodriguez, reviewed by Gopal V)
1920988b66 2021-05-22 HIVE-18069: MetaStoreDirectSql to get tables has misplaced comma (Jesus Camacho Rodriguez, reviewed by Aihua Xu) (addendum)
e5ed2cb9ed 2021-05-22 HIVE-18069: MetaStoreDirectSql to get tables has misplaced comma (Jesus Camacho Rodriguez, reviewed by Aihua Xu)
9b506546a4 2021-05-22 HIVE-15436: Enhancing metastore APIs to retrieve only materialized views (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
051002d23a 2021-05-22 HIVE-6990 : Direct SQL fails when the explicit schema setting is different from the default on (Bing Li, Sergey Shelukhin via Ashutosh Chauhan)

Known Issues and Limitations

  • HIVE-19502 Unable to insert values into table stored by JdbcStorageHandler

  • HIVE-19286 NPE in MERGE operator on MR mode

  • HIVE-760 [Hive-2.3] Could not start hive-metastore on Centos 8 MetaException(message:Version information not found in metastore)

    Starting in MEP 7.0.0, use the MySQL driver with MariaDB.
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
  </property>
 <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
 </property>
    Some SELECT queries can be converted to a single FETCH task minimizing latency. Currently, the query should be single sourced and should not have a subquery or any aggregations or distincts (which incurs RS), lateral views and joins:
    none : disable hive.fetch.task.conversion
minimal : SELECT star, filter on partition columns, LIMIT only
more :  SELECT, filter, LIMIT only (support TABLESAMPLE and virtual columns)

Resolved Issues