4.2. Configure Hadoop

Use the following instructions to configure Hadoop:

These configuration variables are under the [hadoop] section in the hue.ini configuration file.

  1. Configure HDFS Cluster.

    Hue supports only one HDFS cluster currently.

    Ensure that you define the HDFS cluster under the [hadoop][[hdfs_clusters]][[[default]]] sub-section. Use the following variables to configure the HDFS cluster:

    fs_defaultfs

    This is equivalent to fs.defaultFS (fs.default.name) in Hadoop configuration. For example, hdfs://fqdn.namenode.host:8020

    webhdfs_url

    You can also set this to be the WebHDFS URL. The default value is the HTTP port on the NameNode. For example, http://fqdn.namenode.host:50070/webhdfs/v1

    hadoop_hdfs_home

    This is the home of your Hadoop HDFS installation. It is the root of the Hadoop untarred directory or usually /usr/lib/hadoop.

    hadoop_bin

    Use this as the HDFS Hadoop launcher script, which is usually /usr/bin/hadoop.

    hadoop_conf_dir

    This is the configuration directory of the HDFS, typically /etc/hadoop/conf.

  2. Configure YARN (MR2) Cluster.

    Hue supports only one YARN cluster currently.

    Ensure that you define the YARN cluster under the [hadoop][[yarn_clusters]][[[default]]] sub-section. Use the following variables to configure the YARN cluster:

    resourcemanager_host

    The host running the ResourceManager.

    resourcemanager_port

    The port for the ResourceManager IPC service.

    submit_to

    Set this property to true. Hue will be submitting jobs to this YARN cluster. But note that JobBrowser will not be able to show MR2 jobs.

    hadoop_mapred_home

    This is the home of your Hadoop MapReduce installation. It is the root of HDP Hadoop-MapReduce directory (/usr/lib/hadoop-mapreduce). If submit_to is true for this cluster, this configuration value is set as the $HADOOP_MAPRED_HOME for BeeswaxServer and child shell processes.

    hadoop_bin

    Use this as the YARN/MR2 Hadoop launcher script (/usr/bin/hadoop).

    hadoop_conf_dir

    This is the configuration directory of the YARN/MR2 service, typically set to /etc/hadoop/conf.

    resourcemanager_api_url

    The URL of the ResourceManager API. For example, http://fqdn.resourcemanager.host:8088.

    proxy_api_url

    The URL of the ProxyServer API. For example, http://fqdn.resourcemanager.host:8088.

    history_server_api_url

    The URL of the HistoryServer API. For example, http://fqdn.historyserver.host:19888.

    node_manager_api_url

    The URL of the NodeManager API. For example, http://fqdn.resourcemanager.host:8042.


loading table of contents...