How to use Spark jars for Yarn distribution

Question

Hello. I have an archive that has all the jars need for Yarn cache. I want to know how to use this archive for the application. Where should I store it and how to make the application use it?

Answer 1

First, store upload this archive to hdfs and note the path to the archive on hdfs. Then open spark shell and run the below command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.archive=<hdfs path to archive>