5.2. Create HDFS Directories

You must create the following HDFS directories after you upgrade:

  • YARN NodeManager remote applications log

  • HDFS Job History

To create the YARN NodeManager remote applications log

  1. Open /etc/hadoop/conf/yarn-site.xml.

  2. Write down the value of the yarn.nodemanager.remote-app-log-dir so that you can use it in place of the ${yarn.nodemanager.remote-app-log-dir} variable in later examples. For example: ${yarn.nodemanager.remote-app-log-dir} = /app-logs

  3. Create the ${yarn.nodemanager.remote-app-log-dir} in HDFS.

    hdfs dfs -mkdir ${yarn.nodemanager.remote-app-log-dir}
    hdfs dfs -chown -R yarn:hadoop ${yarn.nodemanager.remote-app-log-dir}
    hdfs dfs -chmod -R 777 ${yarn.nodemanager.remote-app-log-dir}
  4. Create a JobHistory directory in HDFS.

    1. Open mapred-site.xml.

    2. Write down the value of the mapreduce.jobhistory.done-dir so that you can use it in place of the ${mapreduce.jobhistory.done-dir} variable in later examples.

    3. Write down the value of the mapreduce.jobhistory.intermediate-done-dir so that you can use it in place of the ${mapreduce.jobhistory.intermediate-done-dir} variable in later examples.

    4. Create the JobHistory directories in HDFS.

      hadoop dfs -mkdir ${mapreduce.jobhistory.done-dir}
      hadoop dfs -mkdir ${mapreduce.jobhistory.intermediate-done-dir}
      hadoop dfs -chown -R mapred:hadoop ${mapreduce.jobhistory.done-dir}
      hadoop dfs -chown -R mapred:hadoop ${mapreduce.jobhistory.intermediate-done-dir}
      hadoop dfs -chmod -R 777 ${mapreduce.jobhistory.done-dir}
      hadoop dfs -chmod -R 777 ${mapreduce.jobhistory.intermediate-done-dir}
      [Note]Note

      You have to create the parent directories in HDFS by yourself. Grant the parent directories the same permissions.


loading table of contents...