Apache Spark Component Guide
Also available as:
PDF
loading table of contents...

Setting Path Variables for Python

To change the Python executable used by a Livy session, follow the instructions for your version of Python.

pyspark

Livy reads the path from the PYSPARK_PYTHON environment variable (this is the same as PySpark).

  • If Livy is running in local mode, simply set the environment variable (this is the same as PySpark).

  • If the Livy session is running in yarn-cluster mode, set spark.yarn.appMasterEnv.PYSPARK_PYTHON in the SparkConf file, so that the environment variable is passed to the driver.

pyspark3

Livy reads the path from environment variable PYSPARK3_PYTHON.

  • If Livy is running in local mode, simply set the environment variable.

  • If the Livy session is running in yarn-cluster mode, set spark.yarn.appMasterEnv.PYSPARK3_PYTHON in SparkConf file, so that the environment variable is passed to the driver.