8. Upgrade Hive and HCatalog

  1. Stop the Hive Metastore, if you have not done so already.

  2. Upgrade Hive and HCatalog. On the Hive and HCatalog host machines, execute the following commands:

    • For RHEL/CentOS:

      yum upgrade hive
      yum erase hcatalog
      yum install hive-hcatalog 
    • For SLES:

      Run the following commands:

      zypper up hive
      zypper erase hcatalog
      zypper install hive-hcatalog

    • For Ubuntu:

      apt-get update hive hcatalog

  3. Upgrade the Hive Metastore database schema by running the upgrade scripts included in HDP for your metastore database. HDP includes upgrade scripts for the following databases:

    • MySQL

    • Postgres

    • Oracle

    • MS SQL Server

    • Derby

    Upgrading MySQL Hive Metastore

    Run the following scripts as the root user to upgrade a MySQL Hive Metastore from 1.3.x to 2.1.x:

    > cd /usr/lib/hive/scripts/metastore/upgrade/mysql
    > mysql hive
    mysql> source upgrade-0.11.0-to-0.12.0.mysql.sql
    mysql> source upgrade-0.12.0-to-0.13.0.mysql.sql

    Upgrading Postgres Hive Metastore

    Run the following scripts as the root user to upgrade a Postgres Hive Metastore from 1.3.x to 2.1.x:

    cd /usr/lib/hive/scripts/metastore/upgrade/postgres
    psql -d hive -a -f upgrade-0.11.0-to-0.12.0.postgres.sql
    psql -d hive -a -f upgrade-0.12.0-to-0.13.0.postgres.sql

    Upgrading Oracle Hive Metastore

    Run the following scripts as the root user to upgrade an Oracle Hive Metastore from 1.3.x to 2.1.x:

    cd /usr/lib/hive/scripts/metastore/upgrade/oracle
    sqlplus
    SQL> @upgrade-0.11.0-to-0.12.0.oracle.sql
    SQL> @upgrade-0.12.0-to-0.13.0.oracle.sql

  4. Edit hive-env.sh to point to the new hive-hcatalog.jar:

    if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
    export HIVE_AUX_JARS_PATH=/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar:${HIVE_AUX_JARS_PATH}
    else
      export HIVE_AUX_JARS_PATH=/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar
    fi                  
  5. Download and extract the HDP companion files. Copy the hive-site.xml file in the configuration_files/hive directory of the extracted companion files to the etc/hive/conf directory on your Hive host machine. This new version of the hive-site.xml file contains new properties for HDP-2.1 features.

  6. Edit the hive-site.xml file and modify the properties based on your environment. Search for TODO in the file for the properties to replace.

    1. Edit the connection properities for your Hive metastore database in hive-site.xml:

      <property>
          <name>javax.jdo.option.ConnectionURL</name>
          <value>jdbc:mysql://TODO-HIVE-METASTORE-DB-SERVER:TODO-HIVE-METASTORE-DB-PORT/TODO-HIVE-METASTORE-DB-NAME?createDatabaseIfNotExist=true</value>
          <description>Enter your Hive Metastore Connection URL, for example if MySQL: jdbc:mysql://localhost:3306/mysql?createDatabaseIfNotExist=true</description>    
        </property>
          
        <property>
          <name>javax.jdo.option.ConnectionUserName</name>
          <value>TODO-HIVE-METASTORE-DB-USER-NAME</value>
          <description>Enter your Hive Metastore database user name.</description>
        </property>
        
        <property>       
         <name>javax.jdo.option.ConnectionPassword</name>       
         <value>TODO-HIVE-METASTORE-DB-PASSWORD</value>  
         <description>Enter your Hive Metastore database password.</description>
        </property>
        
        <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>TODO-HIVE-METASTORE-DB-CONNECTION-DRIVER-NAME</value>
        <description>Enter your Hive Metastore Connection Driver Name, for example if MySQL: com.mysql.jdbc.Driver</description>
      </property>

      Optional: If you want Hive Authorization, set the following Hive authorization parameters in the hive-site.xml file:

      <property>
        <name>hive.security.authorization.enabled</name>
        <value>true</value>
      </property>
        
      <property>
        <name>hive.security.authorization.manager</name>
        <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
      </property>
      
      <property>
        <name>hive.security.metastore.authorization.manager</name>
        <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
      </property>
      
      <property>
        <name>hive.security.authenticator.manager</name>
        <value>org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator</value>
      </property>

      For a remote Hive metastore database, use the following hive-site.xml property value to set the IP address (or fully-qualified domain name) and port of the metastore host. To enable HiveServer2, leave this property value empty.

      <property>       
       <name>hive.metastore.uris</name>       
       <value>thrift://$metastore.server.full.hostname:9083</value>  
       <description>URI for client to contact metastore server. To enable HiveServer2, leave the property value empty. </description>
      </property>
       

    [Note]Note

    You can also use the HDP utility script to fine-tune your configuration settings based on node hardware specifications.

  7. If you are using a MySQL database for your Hive Metastore, run the following scripts to upgrade the Metastore:

    • /usr/lib/hive/scripts/metastore/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql

    • /usr/lib/hive/scripts/metastore/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql

  8. Edit hive-site.xml to set the value of datanucleus.autoCreateSchema to true:

    <property>
      <name>datanucleus.autoCreateSchema</name>
      <value>true</value>
      <description>Creates necessary schema on a startup if one doesn't exist</description>
     </property>  
  9. Edit hive-site.xml to update the value of hive.metadata.export.location to the new, joint hive-hcatalog jar (previously hcatalog-core.jar):

    <property>
       <name>hive.metadata.export.location</name>
       <value>export HIVE_AUX_JARS_PATH=/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar</value>
    </property>
  10. Start the Hive Metastore service.

     Login as $HIVE_USER
    nohup hive --service metastore>$HIVE_LOG_DIR/hive.out 2>$HIVE_LOG_DIR/hive.log & 
  11. Start HiveServer2. On the Hive Metastore host machine, execute the following command:

    sudo su -l $HIVE_USER -c "nohup /usr/lib/hive/bin/hiveserver2 -hiveconf hive.metastore.uris=' ' -hiveconf hive.log.file=hs2.log -hiveconf hive.log.dir=/grid/0/var/log/hive > /grid/0/var/log/hive/hive.out 2> /grid/0/var/log/hive/hive.log &"                  

    where

    • $HIVE_USER is the Hive Service user. For example, hive.

    • $HIVE_LOG_DIR is the directory where Hive server logs are stored (example: /var/log/hive).


loading table of contents...