本文主要承接上篇spark-submit解读后,来说说在spark-class.sh中调用org.apache.spark.launcher.Main,最后spark launcher进行了怎样的操作。
经过spark-submit脚本的处理后,在spark-class脚本中传入当前函数的参数如下:
org.apache.spark.deploy.SparkSubmit
--verbose
--jars /data/clife/clife-bigdata-dispense/libs/activation-1.1.jar;/data/clife/clife-bigdata-dispense/libs/activemq-core-5.7.0.jar
--master yarn-cluster
--num-executors 2
--executor-cores 1
--executor-memory 2G
--conf spark.executor.extraClassPath=/data/clife/clife-bigdata-dispense/libs/activation-1.1.jar;/data/clife/clife-bigdata-dispense/libs/activemq-core-5.7.0.jar
--conf spark.driver.extraClassPath=/data/clife/clife-bigdata-dispense/libs/activation-1.1.jar;/data/clife/clife-bigdata-dispense/libs/activemq-core-5.7.0.jar
--class com.clife.data.action.KafkaDataDispense /data/clife/clife-bigdata-dispense/clife-bigdata-dispense-1.0.0.jar
接下来我们来看看Main.main()方法:
image.png
image.png
紧跟着调用
image.png
image.png
image.png
image.png
image.png
image.png