Spark Guide
Also available as:
PDF
loading table of contents...

Dynamic Resource Allocation Properties

See the following tables for more information about dynamic resource allocation properties.

Table 3.1. Dynamic Resource Allocation Properties

Property NameValueMeaning
spark.dynamicAllocation. enabledtrue

Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload. Note that this is currently only available in YARN mode. For more information, see the Apache Dynamic Resource Allocation documentation.

DRA requires spark.shuffle.service.enabled to be set. The following configurations are also relevant: spark.dynamicAllocation. minExecutors, spark.dynamicAllocation. maxExecutors, and spark.dynamicAllocation. initialExecutors

spark.shuffle.service. enabledtrue

Enables the external shuffle service, which preserves shuffle files written by executors so that the executors can be safely removed.

This property must be set to true if spark.dynamicAllocation. enabled is true.

The external shuffle service must be set up before enabling the property. For more information, see "Starting the Shuffle Service" at the end of this section.

spark.dynamicAllocation. initialExecutorsdefault is spark.dynamicAllocation. minExecutors

Initial number of executors to run if dynamic resource allocation is enabled. This value must be greater than or equal to the minExecutors value, and less than or equal to the maxExecutors value.

spark.dynamicAllocation. maxExecutorsDefault is infinity

Upper bound for the number of executors if dynamic resource allocation is enabled.

spark.dynamicAllocation. minExecutorsDefault is 0

Lower bound for the number of executors if dynamic resource allocation is enabled.


Optional Settings: The following table lists several advanced settings for dynamic resource allocation.

Table 3.2. Dynamic Resource Allocation: Optional Settings

Property NameValueMeaning
spark.dynamicAllocation. executorIdleTimeoutDefault is 60 seconds (60s)

If dynamic resource allocation is enabled and an executor has been idle for more than this duration, the executor will be removed.

spark.dynamicAllocation. cachedExecutorIdleTimeoutDefault is infinity

If dynamic resource allocation is enabled and an executor with cached data blocks has been idle for more than this duration, the executor will be removed.

spark.dynamicAllocation. schedulerBacklogTimeout1 second (1s)

If dynamic resource allocation is enabled and there have been pending tasks backlogged for more than this duration, new executors will be requested.

spark.dynamicAllocation. sustainedSchedulerBacklogTimeoutDefault is schedulerBacklogTimeout

Same as spark.dynamicAllocation. schedulerBacklogTimeout,but used only for subsequent executor requests.


[Note]Note

Dynamic resource allocation is enabled by default for the Spark Thrift Server.