Delay requesting new executor in dynamic allocation

Question

When I enable dynamic allocation and there pending tasks backlogged after 1 second, the application requests for new executors. I want to increase this time from 1 second to more duration. How can I do this?

Answer 1

You can set the duration like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.dynamicAllocation.schedulerBacklogTimeout= <time duration>