Apache Hive overview
Also available as:
PDF

Changes after upgrading to Apache Hive 3

To locate and use your Apache Hive 3 tables after an upgrade, you need to understand the changes that occur during the upgrade process. Changes to the location of tables, permissions to HDFS directories, table types, formats, and ACID-compliance occur.

File locations and ownership

The /apps/hive directory, which is the former location of the Hive 2.x warehouse, does not exist in HDP 3.x. After upgrading to HDP 3.x, tables reside in the following locations:

  • /warehouse/tablespace/managed/hive
  • /warehouse/tablespace/external/hive

Managed tables in HDP 3.0 have file ownership set to hive and directory permissions set to 700.

State of tables before and after upgrading

Hive 2.x and 3.x have transactional and non-transactional tables. Transactional tables have atomic, consistent, isolation, and durable (ACID) properties. In Hive 2.x, the initial version of ACID transaction processing is ACID v1. In Hive 3.x, the mature version of ACID is ACID v2, which is the default table type in HDP 3.0.

The following factors determine the features of a table after upgrading from Hive 2.x to 3.x:

  • ACID (transactional) or no ACID (non-transactional)
  • Security: storage-based authorization (SBA) or other, such as Ranger
  • Storage format of a table

Hive supports the following Hadoop native and non-native storage formats:

  • Native: Tables with built-in support in Hive, such as those in the following file formats:
    • Text
    • Sequence File
    • RC File
    • AVRO File
    • ORC File
    • Parquet File
  • Non-native: Tables that use a storage handler, such as the DruidStorageHandler or HBaseStorageHandler
The following table compares Hive table types, ACID operations, and storage format before an upgrade from HDP 2.x and after an upgrade to HDP 3.x:
Table 1. Hive Upgrade Path for Tables
HDP 2.x HDP 3.x
Table Type ACID v1 Format Security Table Type ACID v2 Format
External No Native or non-native N/A External No Native or non-native
Managed Yes ORC N/A Managed, updatable Yes ORC
Managed No ORC No SBA Managed, updatable Yes ORC
Managed No Native (but non-ORC) No SBA Managed, insert only Yes Native
Managed No Native (including ORC) Some type of security, such as doAs=true plus SBA External No Native
Managed No Non-native N/A External No Native
  • External tables, which are usually not ACID tables, remain external tables.
  • ACID v1 tables become ACID v2 tables.
  • Managed ORC tables without SBA are converted to ACID v2 tables.
  • Managed native tables without SBA are converted to insert-only tables.
  • Managed native tables with SBA are converted to external tables.
  • Managed non-native tables are converted to external tables.

During the upgrade process, you can override the conversion of ACID v1 tables to ACID v2. For example, you can choose to first convert everything except ACID v1 tables to external tables, and then later convert them to ACID tables one by one.

After upgrading, to convert a non-transactional table to an ACID v2 transactional table, you use the ALTER TABLE command and set table properties to 'transaction'='true'.For example:
ALTER TABLE T3 SET TBLPROPERTIES ('transactional'='true');