Considerations for setting up spark

Set up Spark based on your requirement. Following are some of the considerations that you will have to take into account.

You should configure the 'spark.executor.extraClassPath' and ‘spark.driver.extraClassPath’ in spark-defaults.conf file to include the ‘phoenix-<version>-client.jar’ to ensure that all required Phoenix and HBase platform dependencies are available on the classpath for the Spark executors and drivers.
For HDP prior to 2.5, you must use "phoenix-<version>-client-spark.jar". For HDP verison above 2.5, you can use "phoenix-<version>-spark.jar" and "phoenix-<version>-client.jar".
To enable your IDE, you can add the following provided dependency to your build:
<dependency> <groupId>org.apache.phoenix</groupId> <artifactId>phoenix-spark</artifactId> <version>${phoenix.version}</version><scope>provided</scope></dependency>