How to automatically kill executors on blacklisting

Question

I have certain executors that keep going into blacklist again and again and after a particular time, they come back to execution. What I want is that whenever an executor is blacklisted, I want that executor to be killed automatically. How can I do this?

Answer 1

You can set the property to directly kill the executor when it is blacklisted like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.blacklist.killBlacklistedExecutors=true

Now whenever an executor is blacklisted, it will be killed.