How to set max executors for dynamic allocation

Question

Hi. I want to set a max number of executors that can be used for dynamic allocation in my Spark application. How can I set it? What command should I use?

Answer 1

You can set it by assigning the max number of executors to the property as follows:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.dynamicAllocation.maxExecutors= <max number of executors>