Passing condition dynamically to Spark application

Question

How to pass the condition "dF.filter(Id==3000) " dynamically to a spark Application? 

Comments

I need this question answer how to pass the condition do.filter(I'd==300)  dynamically to a spark application

---

Did you try the below-mentioned answer? https://www.edureka.co/community/38774/passing-condition-dynamically-to-spark-application?show=38778#a38778

Answer 1

You can try this:

d.filter(col("value").isin(desiredThings: _*))

and if you want to foldLeft you have to provide the base condition:

d.filter(desiredThings.foldLeft(lit(false))(

  (acc, x) => (acc || col("value") === (x)))

)

Alternatively, to use with a filter or where, you can generate a SQL expression using:

val filterExpr = desiredThings.map( v => s"value = $v").mkString(" or ")

And then use it like

d.filter(filterExpr).show

// or

d.where(filterExpr).show