How to set time for task speculation

Question

I have enabled task speculation for my application. It is working as expecting apart from one tiny thing. I want the check for task speculation to be done faster than the time it is taking right now. How can I change the time?

Answer 1

By default, the check for task speculation when speculation is enabled happens every 100ms. You can change the time using the spark.speculation.interval property. Something like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.speculation.interval=<set new time here>