kafka和sparkstreaming整合打包运行错误处理

问题一:

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: kafka/serializer/StringDecoder
at DirectKafkaWordCountDemo$.main(DirectKafkaWordCountDemo.scala:19)
at DirectKafkaWordCountDemo.main(DirectKafkaWordCountDemo.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)

此问题是缺少kafka_2.11-0.8.2.1.jar包所导致,将这个jar包上传spark/jars下就可以了

问题二:

-org.apache.spark.streaming.kafka.KafkaUtils

如果是这个异常,是缺少spark-streaming-kafka_2.11-1.6.2.jar包所导致,将这个包上传到spark/jars下就可以了

问题三:

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: com/yammer/metrics/Metrics
at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:85)
at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:26)
at kafka.consumer.FetchRequestAndResponseMetrics.<init>(FetchRequestAndResponseStats.scala:35)
at kafka.consumer.FetchRequestAndResponseStats.<init>(FetchRequestAndResponseStats.scala:46)

此问题是缺少metrics-core-2.2.0.jar包所致,将这个包上传到spark/jars下就可以了。注:在spark/jars下,metrics-core-3.1.2.jar和metrics-core-2.2.0.jar都要存在,metrics-core-3.1.2.jar是spark core所需要的

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
【社区内容提示】社区部分内容疑似由AI辅助生成,浏览时请结合常识与多方信息审慎甄别。
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

推荐阅读更多精彩内容