Dolphinscheduler3.1.8兼容Hive2.1.1-cdh6.3.2

前言

dolphinscheduler安装版本:3.1.8
hive版本:2.1.1-cdh6.3.2

发现问题&分析问题

  1. 添加数据源报错
[ERROR] 2021-01-08 10:55:40.553 org.apache.dolphinscheduler.api.service.DataSourceService:
[433] - Could not establish connection to jdbc:hive2://10.0.4.78:10000/vfs_dw: Required field 'client_protocol' is unset! 
Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=vfs_dw})
java.sql.SQLException: Could not establish connection to jdbc:hive2://10.0.4.78:10000/vfs_dw: 
Required field 'client_protocol' is unset! 
Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=vfs_dw})

2.运行任务报错
1.Dolphinscheduler任务启动报错,协议错误
2.Dolphinscheduler任务全部启动异常,所有实例都变成提交成功,但是没有日志,也不会提交

解决办法

解决1:将hive-jdbc-2.1.1-cdh6.3.2-standalone.jar 放到api和work的lib目录,删除多余hive jar包
发现Hive数据可以配置成功,但是ds所有工作流启动不了了,启动就变成提交成功,没有日志。

解决2:将cdh所有hive-*.jar都放到api和work的lib目录,删除多余hive jar包
发现work启动异常,日志出现hive-jdbc-2.1.1-cdh6.3.2-standalone相关报错

Exception in thread "Worker-Server" java.lang.IllegalArgumentException: 
LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. 
Either remove Logback or the competing implementation 
(class org.apache.logging.slf4j.Log4jLoggerFactory loaded from file:/opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar). 
If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.apache.logging.slf4j.Log4jLoggerFactory

解决3:将cdh hive包选择性替换到api和work的lib目录
成功解决

#api-server 
#备份libs
cp -ar /opt/software/dolphinscheduler-3.1.8/api-server/libs/ /opt/software/dolphinscheduler-3.1.8/api-server/libs-bac
#复制到api-server 
cp /opt/cloudera/parcels/CDH/jars/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar /opt/software/dolphinscheduler-3.1.8/api-server/libs/
#权限改为dolphinscheduler 
chown dolphinscheduler:dolphinscheduler /opt/software/dolphinscheduler-3.1.8/api-server/libs/hive-*.jar
#删除ds自带的所有hive错误版本相关驱动依赖
rm -rf /opt/software/dolphinscheduler-3.1.8/api-server/libs/hive-*2.3.9.jar
#check 
ll /opt/software/dolphinscheduler-3.1.8/api-server/libs/|grep hive

#备份libs
cp -ar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/ /opt/software/dolphinscheduler-3.1.8/worker-server/libs-bac

#worker-server
#删除ds自带的所有hive错误版本相关驱动依赖 
rm -rf /opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-*.jar

#复制到worker-server
cp -n /opt/cloudera/parcels/CDH/jars/hive-common*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-jdbc-2.1.1-cdh6.3.2.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-metastore*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-serde*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-service*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-service-rpc*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/
cp -n /opt/cloudera/parcels/CDH/jars/hive-storage-api*.jar /opt/software/dolphinscheduler-3.1.8/worker-server/libs/

#权限改为dolphinscheduler
chown dolphinscheduler:dolphinscheduler /opt/software/dolphinscheduler-3.1.8/worker-server/libs/hive-*.jar
#check 
ll /opt/software/dolphinscheduler-3.1.8/worker-server/libs/|grep hive
最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

推荐阅读更多精彩内容