How to get SQL configuration in Spark using Python

Question

Hi. Can someone help me get the Spark SQL configuration of my Spark Application? I am using for this. And I want to know how I can get the configuration details using python. Please help

Answer 1

You can get the configuration details through the Spark session. What you have to do is, first create a spark session and then use the set command to get the configuration details. For example:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('abc').getOrCreate()
spark.sql("SET -v").show(n=200, truncate=False)