Managing High Availability
Also available as:
PDF

Enable NameNode high availability

To ensure that another NameNode in your cluster is always available if the active NameNode host fails, you should enable and configure NameNode high availability on your cluster using Ambari Web.

The Enable HA wizard describes the set of automated and manual steps you must take to set up NameNode high availability. HDFS and ZooKeeper must stop and start when enabling NameNode HA. Maintenance Mode will prevent those start and stop operations from occurring. If the HDFS or ZooKeeper services are in Maintenance Mode the NameNode HA wizard will not complete successfully.
  • Verify that you have at least three hosts in your cluster and are running at least three Apache ZooKeeper servers.
  • Verify that the Hadoop Distributed File System (HDFS) and ZooKeeper services are not in Maintenance Mode.
  1. In Ambari Web, browse to Services > HDFS > Summary.
  2. Click Service Actions, then click Enable NameNode HA.
    The Enable HA wizard launches.
  3. On Get Started, type in a Nameservice ID and click Next.
    You use this Nameservice ID instead of the NameNode FQDN after HA is set up.


    If you are using Hive, you must manually change the Hive Metastore FS root to point to the Nameservice URI instead of the NameNode URI, as follows:
    1. Find the current FS root on the Hive host.

      hive --config /etc/hive/conf/conf.server --service metatool -listFSRoot

      utput should look like: Listing FS Roots... hdfs://[NAMENODE_HOST]/apps/hive/warehouse.

    2. Change the FS root.

      $ hive --config /etc/hive/conf/conf.server --service metatool -updateLocation [NEW_LOCATION][OLD_LOCATION]

      For example, if your Nameservice ID is mycluster, you input:

      $ hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://mycluster/apps/hive/warehouse hdfs://c6401.ambari.apache.org/apps/hive/warehouse

      The output looks similar to:

      Successfully updated the following locations... Updated X records in SDS table
      Important
      Important

      The Hive configuration path for a default HDP 2.3.x or later stack is /etc/hive/conf/conf.server

      The Hive configuration path for a default HDP 2.2.x or earlier stack is /etc/hive/conf

    3. Adjust the ZooKeeper Failover Controller retries setting for your environment.
      1. Browse to Services > HDFS > Configs > Advanced core-site.
    4. Set ha.failover-controller.active-standby-elector.zk.op.retries=120.
  4. On Select Hosts, select a host for the additional NameNode and the JournalNodes, and then click Next.


  5. On Review, confirm your host selections and click Next:


  6. Follow the directions on the Manual Steps Required: Create Checkpoint on NameNode page.
    You must log in to your current NameNode host and run the commands to put your NameNode into safe mode and create a checkpoint.


    When Ambari detects success and the message on the bottom of the window changes to Checkpoint created, click Next.
  7. On Configure Components, monitor the configuration progress bars.


    When all componets have been configured successgfully, click Next
  8. Follow the instructions on Manual Steps Required: Initialize JournalNodes.


    You must log in to your current NameNode host to run the command to initialize the JournalNodes. When JournalNodes initiate successfully, click Next.
  9. On Start Components, monitor the progress bars as the ZooKeeper servers and NameNode start.


    When ZooKeeper servers and NameNode have starteed, click Next.
    In a cluster with Ranger enabled, and with Hive configured to use MySQL, Ranger will fail to start if MySQL is stopped. To work around this issue, start the Hive MySQL database and then retry starting components.
  10. On Manual Steps Required: Initialize NameNode HA Metadata, complete all steps, using the instructions on the page.


    For this step, you must log in to both the current NameNode and the additional NameNode. Make sure you are logged in to the correct host for each command. Click OK to confirm, after you complete each command. When you have completed the four steps required to intialize metadata, click Next.
  11. On Finalize HA Setup, monitor the progress bars, the wizard completes HA setup.


    When the wizard completes HA setup, click Done to finish the wizard.
    After Ambari Web reloads, you may see some alert notifications. Wait a few minutes until all the services restart.
  12. Restart any components using Ambari Web, if necessary.
Review and confirm all recommended configuration changes.