Dynamic Allocation Spark 3

Spark Configuration

Spark allows you to configure the system according to your needs. One of the locations to configure a system is Spark Properties. Spark Properties control most application parameters and can be set using the SparkConf object. One of the sub-domains of these properties is Dynamic Allocation.

Dynamic Allocation

Whenever you run an application on Spark cluster, the cluster allocates resources to your application for the entirety of its lifecycle. When the application is terminated, these resources are freed and allocated to another application. This method of resource-allocation will be sufficient for applications that are memory-heavy and continually processing. For example, batch processing applications need the resources for almost the entire time during its lifecycle. On the contrary, if we deploy applications that use minimal resources at some instances and maximal resource for other instances, then the tactic mentioned above will be inefficient. This is where Dynamic Allocation jumps in. Dynamic Allocation lets Spark applications request more resources for the executor if the tasks queued exceed the load current resources can bear. It also releases resources when the application is sitting idle so that they can be used by another application.

Spark Dynamic Allocation is enabled by default. To disable Dynamic Allocation, set spark.dynamicAllocation.enabled to false.

You can also specify the upper and lower bound of the resources that should be allocated to your application. Use the spark.dynamicAllocation.maxExecutors to set the maximal number of executors for your application, and spark.dynamicAllocation.minExecutors to set the minimal number of executors for your application. By default, maxExecutors is set to infinity and minExecutors is set to zero.

Here is a listhttps://spark.apache.org/docs/3.5.2/configuration.html of all the methods you can use to configure Spark Dynamic Allocation to your needs.