固定消费补丁包位置
1.检查下所有spark节点，如下路径的jar包
/opt/huawei/Bigdata/FusionInsight/FusionInsight-Spark-1.5.1/spark/lib/spark-core_2.10-1.5.1.jar
如何校验：
cd /opt/huawei/Bigdata/FusionInsight/FusionInsight-Spark-1.5.1/spark/lib/
sha256sum spark-core_2.10-1.5.1.jar
2.检查hdfs上上着两个个jar包：
hdfs dfs -get /user/oozie/share/lib/spark/spark-core_2.10-1.5.1.jar .
hdfs dfs -get /user/oozie/share/lib/spark/spark-streaming-kafka_2.10-1.5.1.jar .
如何校验：
sha256sum spark-core_2.10-1.5.1.jar
sha256sum spark-streaming-kafka_2.10-1.5.1.jar
3.正常的校验码校验值：
9ab12ca8bccf10e55ee34622d76d5d54697d00177deb7f8364f0ce175afb1b3d *spark-core_2.10-1.5.1.jar
ce1361b12a3fa02267856b23431e65a9793595f9c12563dad5e075059b2baaa1 *spark-streaming-kafka_2.10-1.5.1.jar
4.检查配置是否更改
找到rest所在节点，
//1.进入配置目录
cd /opt/huawei/Bigdata/apache-tomcat-7.0.72/webapps/rest/WEB-INF/classes/oozie-job-conf/sparkstreaming/TfcPassDriver
//2.检查kafka分区
vi job.properties进行修改num-executors后面数字保证和3中kafka分区一致
//3.检查配置开关
cd /opt/huawei/Bigdata/apache-tomcat-7.0.72/webapps/rest/WEB-INF/classes/oozie-job-conf/sparkstreaming/
检查spark-defaults.conf如下配置项：
spark.scheduler.minRegisteredResourcesRatio = 1
spark.scheduler.maxRegisteredResourcesWaitingTime = 300s
spark.kafka.newfeature = 1
如何判断补丁生效
检查executor日志stdout，执行命令
grep " Computing topic" stdout | less
如果发现，存在计算了多个不同分区,比如partition后面的数据不一致，就说明读取了多个分区数据：
2019-07-02 10:22:36,030 | INFO | [Executor task launch worker-9209] | Computing topic TOPIC_TFC_PASS, partition 13 offsets 259337415 -> 259337456 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:36,031 | INFO | [Executor task launch worker-9175] | Computing topic TOPIC_TFC_PASS, partition 1 offsets 763405663 -> 763405707 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:38,032 | INFO | [Executor task launch worker-9175] | Computing topic TOPIC_TFC_PASS, partition 1 offsets 763405707 -> 763405747 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:38,033 | INFO | [Executor task launch worker-9209] | Computing topic TOPIC_TFC_PASS, partition 13 offsets 259337456 -> 259337490 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:40,030 | INFO | [Executor task launch worker-9175] | Computing topic TOPIC_TFC_PASS, partition 1 offsets 763405747 -> 763405786 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:40,031 | INFO | [Executor task launch worker-9209] | Computing topic TOPIC_TFC_PASS, partition 13 offsets 259337490 -> 259337526 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:42,029 | INFO | [Executor task launch worker-9209] | Computing topic TOPIC_TFC_PASS, partition 13 offsets 259337526 -> 259337580 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:42,029 | INFO | [Executor task launch worker-9175] | Computing topic TOPIC_TFC_PASS, partition 1 offsets 763405786 -> 763405819 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
2019-07-02 10:22:44,033 | INFO | [Executor task launch worker-9209] | Computing topic TOPIC_TFC_PASS, partition 1 offsets 763405819 -> 763405861 | org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.<init>(KafkaRDD.scala:171)
结论
补丁成功，由于资源不够executor没有启动完成。
