How to increase worker timeout in Spark application

Question

Need help with increasing the worker timeout. After a minute the master considers the workers lost. I want to increase this time to 2 minutes. How can I do this?

Answer 1

By default, the timeout is set to 60 seconds. To change it to 2 minutes, you have to set it to 120 seconds and you can do it using the following commands:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.worker.timeout=120