问题描述
运行一天发现访问hbase失败,异常如下token cant be found In cache。
分析
1.检查配置项,怀疑是缓存导致,
spark.inputFormat.cache.enabled是否为false
spark.yarn.security.credentials.hbase.enabled是否为true
2.日志打印:
2019-07-20 16:07:24,773 | WARN | [main] | spark.yarn.security.credentials.hbase.enabled is deprecated. Please use spark.security.credentials.hbase.enabled instead. | org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66)
2019-07-20 16:07:24,775 | DEBUG | [main] | Using the following delegation token providers: hadoopfs, hive. | org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
从这里看,只有hadoopfs, hive了,没有hbase的token了
结论
初步结论,hbase开关在2.3将hbase的开关从spark.yarn.security.credentials.hbase.enabled变化成了spark.security.credentials.hbase.enabled
解决措施
spark.security.credentials.hbase.enabled 开关在Spark2.1和Spark2.3使用了不同的参数。
2.1:
spark.inputFormat.cache.enabled是否为false
spark.yarn.security.credentials.hbase.enabled是否为true
2.3:
spark.security.credentials.hbase.enabled