Hive 2.3.8 - 2201 (EEP 7.1.2) Release Notes

The notes below relate specifically to the HPE Ezmeral Data Fabric Distribution for Apache Hive. You may also be interested in the Apache Hive-2.3.8 Release Notes and the Apache Hive homepage.
Hive Version 2.3.8
Release Date March 2022
HPE Version Interoperability See Hive and HCatalog Support Matrix andEcosystem Support Matrix and MEP Components and OS Support.
Source on GitHub https://github.com/mapr/hive
GitHub Release Tag 2.3.8-mapr-2201
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate tohttp://package.ezmeral.hpe.com/releases/MEP/, and select your EEP and OS to view the list of package names.
ODBC/JDBC Drivers
Hive 2.3.8 works with the following HPE Hive drivers:

For additional driver information, see Connecting to HiveServer2.

Feature support

The following list describes support of various components and functionality with Hive 2.3.8 - 2201:

  • Supports Hive-2.3.8 on Tez-0.9.2 For more information, see Tez 0.9.2 - 2201 (EEP 7.1.2) Release Notes.

  • Does not support Hive on Spark. You cannot use Spark as an query engine for Hive. However, you can run Hive and Spark on the same cluster. You can also use Spark SQL and Drill to query Hive tables.

  • Does not support HDFS encryption in Hive tables.

  • Does not support LLAP with Hive-2.3.8 because Apache Slider is not a HPE supported ecosystem component.

  • Starting from Hive 2.1, Hive needs to run the schematool command as an initialization step.

Changes in default security configuration

The following list describes changes in default security for Hive 2.3.8 - 2201:
  • None.

New in This Release

Hive 2.3.8 - 2201 introduces the following enhancements or HPE platform-specific behavior changes:

  • Added configuration to view audit logs for connected, disconnected, and total connected users in HiveServer2.
  • Added Service verifier.

Fixes

This HPE release includes the following fixes on the base release:
Commit Date (YYYY-MM-DD) Comment
3a48a4b629 2022-01-12 MAPR-HIVE-1071 : Fix SslDefaultTest
53ac871854 2022-01-12 EEP-HIVE-1132 : Update Hadoop version to 2.7.6.0-mapr-712-SNAPSHOT in MEP-712
f0d52b51ec 2022-01-06 EEP-HIVE-1065: CVE-2021-37136, CVE-2021-37137, WS-2020-0408, CVE-2021-21290: netty-*-4.1.55.Final.jar
7071a6a977 2022-01-05 EEP-HIVE-1062: CVE-2016-5007, CVE-2016-9878 ,CVE-2018-1271, CVE-2018-1272, CVE-2020-5421: spring-*-3.2.16.RELEASE.jar
6c800f885f 2022-01-05 EEP-HIVE-1068: CVE-2020-9480: spark-network-common_2.11-2.3.0.jar, CVE-2018-17190: spark-core_2.11-2.3.0.jar
70364dcbcb 2022-01-04 EEP-HIVE-1117: Update log4j v2 to the latest available (to 2.17+)
2a3f1318a5 2022-01-04 EEP-HIVE-1116: Hive returns an incorrect number of columns
563da5a5fa 2022-01-04 Revert "MAPR-HIVE-930: Cannot run join with Order by and Limit clause specified at the same time"
d51bbd22c5 2021-12-30 EEP-HIVE-1119: com.fasterxml.jackson.annotation.JsonFormat.empty()Lcom/fasterxml/jackson/annotation/JsonFormat
737df2650d 2021-12-24 EEP-HIVE-1064: CVE-2021-30639 ; CVE-2021-33037: tomcat-coyote-10.0.4.jar
2d0a5f247f 2021-12-24 EEP-HIVE-1059: CVE-2019-10172, CVE-2019-10202: jackson-mapper-asl-1.9.13.jar, jackson-mapper-asl-1.9.2.jar
a90c3290ca 2021-12-24 EEP-HIVE-1056: CVE-2021-35515, CVE-2021-35516, CVE-2021-35517, CVE-2021-36090: commons-compress-1.20.jar
5fd89c2f7f 2021-12-24 EEP-HIVE-1055: CVE fixes of bcprov-jdk15on-1.52.jar
d1fec1c964 2021-12-24 EEP-HIVE-1054: WS-2021-0419: gson-2.2.4.jar
3b092554d6 2021-12-16 MAPR-HIVE-1002 : Hive-2.3 does not remove old compressed logs
9ca658cde2 2021-12-14 EEP-HIVE-1097: CVE-2021-44228 - Log4j vulnerability
1f6b1fc3cf 2021-12-03 EEP-HIVE-1087: CAST gives NULL values during insert when vectorization enabled.
81dacfe359 2021-11-25 MAPR-HIVE-1086 : Upgrade Jetty to 9.4.44.v20210927
f858bbd8a3 2021-11-22 MAPR-HIVE-1007: Permission denied to hbase temp files while running hcat jobs from other user
b9aed0dfb9 2021-11-22 MAPR-HIVE-977 : Downgrade jackson to v2.11.1 or to 2.11.3 to be consistent with core version
57e796b851 2021-11-22 MAPR-HIVE-960 : CVE-2012-5783 vulnerability in commons-httpclient
aecc898b7a 2021-11-22 MAPR-HIVE-965 : Throws exception at INSERT statement with Avro table on MapReduce engine
d71410e462 2021-11-22 MAPR-HIVE-959 : Update derbyclient and derbynet to most feasible version
e1d0cdc7ca 2021-11-22 MAPR-HIVE-963 : CVE-2020-13956,WS-2017-3734 vulnerabilities in httpclient
993b83adb9 2021-11-22 MAPR-HIVE-962 : WS-2019-0379: commons-codec vulnerability
642c2b0ad2 2021-11-22 MAPR-HIVE-896 : CVE-2020-17521 vulnerability in Groovy
8dd7dc8ff3 2021-11-22 MAPR-HIVE-927 : NPE thrown from XmlUtil by Hive Client
9f152f2318 2021-11-22 MAPR-HIVE-950 : HiveVersionInfo.getShortVersion returns wrong version
f986d83c36 2021-11-11 MAPR-HIVE-1031 : logError: command not found if any error happens during configuring Hive
9f85d37d4b 2021-11-11 MAPR-HIVE-1024 : Replace deprecated AuthMethod.DIGEST with AuthMethod.TOKEN in HadoopThriftAuthBridge25Sasl
5d7040f540 2021-11-11 MAPR-HIVE-975: Customer request to investigate temporary hive session files cleanup improvements
a6e9bde12c 2021-09-30 MAPR-HIVE-994 : Non mapr user unable to read SSL configuration from XML files on Core 7.0
f0f1d78c10 2021-09-16 DFDEVOPS-1979:Move WS scan to the shared library.
9b4c3ad944 2021-06-08 MAPR-HIVE-949 : Make SchemaEvolution class behavior the same as in Orc-1.5.12
9b75385d22 2021-06-06 MAPR-HIVE-894 : CVE-2020-13955 vulnerability in Calcite
2a7526fdea 2021-05-25 MAPR-HIVE-947 : org.apache.hadoop.hive.ql.exec.tez.TezTask at creating session
a64ce8195c 2021-05-23 MAPR-HIVE-945 : FAILURE! - in org.apache.hadoop.hive.maprdb.json.MapRDbJsonFetchByIdOptimizerPositiveTest
4215a9ab82 2021-05-23 MAPR-HIVE-944 : FAILURE! - in org.apache.hadoop.hive.hbase.TestHBaseSerDe
30655745a5 2021-05-23 MAPR-HIVE-943 : Fix org.apache.hadoop.hive.cli.TestCliDriverMethods
bfda9cc9a8 2021-05-23 MAPR-HIVE-942 : FAILURE! - in org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler
This release from HPE also includes the following back-ported issues. For complete details, refer to the commit log for this project in GitHub.
Commit Date (YYYY-MM-DD) Comment
0e6ddbd638 2021-12-16 HIVE-16820 : TezTask may not shut down correctly before submit (Sergey Shelukhin, reviewed by Siddharth Seth)
ba928dded2 2021-11-22 HIVE-24965: Describe table partition stats fetch should be configurable
a1d6b137d1 2021-11-22 HIVE-22453: Describe table unnecessarily fetches partitions
03bb3ad2ef 2021-11-22 HIVE-23756 : Added more constraints to the package.jdo file
b5670273db 2021-11-22 HIVE-24177 : hive mapjoin throws udf class not found
37f70e1f09 2021-11-22 HIVE-25054: Upgrade `jodd-core` dependency to get rid of CVE-2018-21234 (Abhay Chennagiri, reviewed by Jesus Camacho Rodriguez)
f870fdc54b 2021-11-22 HIVE-17659 : get_token thrift call fails for DBTokenStore in remote HMS mode (Vihang Karajgaonkar, reviewed by Aihua Xu)
9dbd2ceb60 2021-11-22 HIVE-17824 : msck repair table should drop the missing partitions from metastore (Janaki Lahorani, reviewed by Peter Vary, Alexander Kolbasov and Vihang Karajgaonkar)
cbc7f88a61 2021-11-22 HIVE-16143: Improve msck repair batching (Vihang Karajgaonkar, reviewed by Sahil Takiar & Aihua Xu)
f2067dedea 2021-12-08 HIVE-17774: compaction may start with 0 splits and fail
1ccb218119 2021-05-22 HIVE-21085: Materialized views registry starts non-external tez session (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
dea7190511 2021-05-22 HIVE-19691: Start SessionState in materialized views registry (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
09b4ca437f 2021-05-22 HIVE-17853: RetryingMetaStoreClient loses UGI impersonation-context when reconnecting after timeout (Chris Drome, reviewed by Mithun Radhakrishnan)
b8902a7bb8 2021-05-22 HIVE-23534: NPE in RetryingMetaStoreClient#invoke when catching MetaException with no message (Stamatis Zampetakis, reviewed by Jesus Camacho Rodriguez)
11db00d681 2021-05-22 HIVE-18494: Regression: from HIVE-18069, the metastore directsql is getting disabled (Jesus Camacho Rodriguez, reviewed by Gopal V
1920988b66 2021-05-22 HIVE-18069: MetaStoreDirectSql to get tables has misplaced comma (Jesus Camacho Rodriguez, reviewed by Aihua Xu) (addendum)
e5ed2cb9ed 2021-05-22 HIVE-18069: MetaStoreDirectSql to get tables has misplaced comma (Jesus Camacho Rodriguez, reviewed by Aihua Xu)
9b506546a4 2021-05-22 HIVE-15436: Enhancing metastore APIs to retrieve only materialized views (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
051002d23a 2021-05-22 HIVE-6990 : Direct SQL fails when the explicit schema setting is different from the default on (Bing Li, Sergey Shelukhin via Ashutosh Chauhan)
9120da5c4f 2021-05-31 HIVE-21075 : Metastore: Drop partition performance downgrade with Postgres DB
39d42ddf12 2021-05-31 HIVE-9447: Metastore: inefficient Oracle query for removing unused column descriptors when add/drop table/partition (Selina Zhang reviewed by Ashutosh Chauhan, Adam Szita)
0912498e97 2021-11-11 HIVE-20072 : Write access being requested when performing select on a table
11e8219ace 2021-11-22 HIVE-19228: Remove commons-httpclient 3.x usage (Janaki Lahorani reviewed by Aihua Xu)
1fc7585a2e 2021-06-08 ORC-437: Make acid schema checks case insensitive

Known Issues and Limitations

  • HIVE-19502 Unable to insert values into table stored by JdbcStorageHandler

  • HIVE-19286 NPE in MERGE operator on MR mode

  • HIVE-789 [Hive-Hbase integration] Unable to run queries against hive-hbase tables. ClassNotFoundException: HiveHBaseTableInputFormat [MEP-7.0.0]

If you run an HBase + Hive + Tez integration in MEP 7.1.0, you may encounter the following exception:
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.mapr.BaseTableMappingRules
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    ... 39 more

This exception can occur due to the new Tez classloader implemented in the Tez project. To resolve this issue, put the following additional JAR files in the /apps/tez/tez-0.9 folder.

Issue the following commands before you run HBase + Hive + Tez integration in MEP 7.1.0:
hadoop fs -mkdir /apps/tez/tez-0.9/hbase
hadoop fs -put /opt/mapr/hbase/hbase-1.4.12/lib/* /apps/tez/tez-0.9/hbase/
Add the following property to /opt/mapr/tez/tez-0.9/conf/tez-site.xml:
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez/tez-0.9,${fs.defaultFS}/apps/tez/tez-0.9/lib,${fs.defaultFS}/apps/tez/tez-0.9/hbase/</value>
 </property>

It is assumed that the Hive version is 2.3, Hbase version is 1.4.12, Tez version is 0.9, Hadoop version is 2.7.4, Zookeeper version is 3.5.6.0, and ecosystem release is 2009.

HIVE-760 [Hive-2.3] Could not start hive-metastore on Centos 8 MetaException(message:Version information not found in metastore)

Starting in MEP 7.0.0, use the MySQL driver with MariaDB.
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
  </property>
 <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
 </property>
Some SELECT queries can be converted to a single FETCH task minimizing latency. Currently, the query should be single sourced and should not have a subquery or any aggregations or distincts (which incurs RS), lateral views and joins:
none : disable hive.fetch.task.conversion
minimal : SELECT star, filter on partition columns, LIMIT only
more :  SELECT, filter, LIMIT only (support TABLESAMPLE and virtual columns)

Resolved Issues

  • None.