4.2. Configure Hadoop

Use the following instructions to configure Hadoop:

These configuration variables are under the [hadoop] section in the hue.ini configuration file.

  1. Configure HDFS Cluster.

    Hue supports only one HDFS cluster currently.

    Ensure that you define the HDFS cluster under the [[[default]]] sub-section. Use the following variables to configure the HDFS cluster:

    fs_defaultfs

    This is equivalent to fs.defaultFS (fs.default.name) in Hadoop configuration.

    webhdfs_url

    You can also set this to be the HttpFS URL. The default value is the HTTP port on the NameNode.

    hadoop_hdfs_home

    This is the home of your Hadoop HDFS installation. It is the root of the Hadoop untarred directory or usually /usr/lib/hadoop.

    hadoop_bin

    Use this as the HDFS Hadoop launcher script, which is usually /usr/bin/hadoop.

    hadoop_conf_dir

    This is the configuration directory of the HDFS, typically /etc/hadoop/conf.

  2. Configure YARN (MR2) Cluster.

    Hue supports only one YARN cluster currently.

    Ensure that you define the YARN cluster under the [[[default]]] sub-section. Use the following variables to configure the YARN cluster:

    resourcemanager_host

    The host running the ResourceManager.

    resourcemanager_port

    The port for the ResourceManager IPC service.

    submit_to

    Set this property to true. Hue will be submitting jobs to this Yarn cluster. But note that JobBrowser will not be able to show MR2 jobs.

    hadoop_mapred_home

    This is the home of your Hadoop MapReduce installation. It is the root of HDP Hadoop-MapReduce directory (/usr/lib/hadoop-mapreduce). If submit_to is true for this cluster, this configuration value is set as the $HADOOP_MAPRED_HOME for BeeswaxServer and child shell processes.

    hadoop_bin

    Use this as the Yarn/MR2 Hadoop launcher script (/usr/bin/hadoop).

    hadoop_conf_dir

    This is the configuration directory of the Yarn/MR2 service, typically set to /etc/hadoop/conf.


loading table of contents...