How to make Spark wait for more time for acknowledgement

Question

My connection is taking more time to receive the acknowledgment. And because the timeout is set less, the connection is ending before the acknowledgment is received. How can I increase the time to wait for acknowledgment?

Answer 1

Use the following command to increase the wait time:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.core.connection.ack.wait.timeout=<time in seconds>