Using Apache Phoenix to store and access data
Also available as:

Considerations for setting up spark

Set up Spark based on your requirement. Following are some of the considerations that you will have to take into account.

  • You should configure the 'spark.executor.extraClassPath' and ‘spark.driver.extraClassPath’ in spark-defaults.conf file to include the ‘phoenix-<version>-client.jar’ to ensure that all required Phoenix and HBase platform dependencies are available on the classpath for the Spark executors and drivers.

  • For HDP prior to 2.5, you must use "phoenix-<version>-client-spark.jar". For HDP verison above 2.5, you can use "phoenix-<version>-spark.jar" and "phoenix-<version>-client.jar".
  • To enable your IDE, you can add the following provided dependency to your build:

    <dependency> <groupId>org.apache.phoenix</groupId> <artifactId>phoenix-spark</artifactId> <version>${phoenix.version}</version><scope>provided</scope></dependency>