How to enable SASL authentication after Spark authentication

Question

Hi. I am using some older shuffle services that do not support Spark authentication. So, I want to fall back to SASL authentication so that these shuffle services can be authenticated. How can I do this?

Answer 1

You can do this by setting the SASL fallback property to true. You can do it like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.network.crypto.saslFallback=true