Non-Ambari Cluster Installation Guide
Also available as:
PDF
loading table of contents...

Configuring the Spark History Server to Use HDFS

To configure the Spark History Server for HDFS, complete the following steps:

  1. As the hdfs service user, create an HDFS directory called spark-history with user:spark, user group:hadoop, and permissions = 777:

    hdfs dfs -mkdir /spark-history
    hdfs dfs -chown -R spark:hadoop /spark-history
    hdfs dfs -chmod -R 777 /spark-history
  2. Add the following properties to the spark-defaults.conf file.

    spark.yarn.historyServer.address c6401.ambari.apache.org:18080
    spark.history.ui.port 18080
    spark.eventLog.dir hdfs:///spark-history
    spark.eventLog.enabled true
    spark.history.fs.logDirectory hdfs:///spark-history
  3. Start the Spark History Server:

    su - spark -c "usr/hdp/current/spark-historyserver/sbin/start-history-server.sh"