hudi 0.8,使用insert_overwrite_table模式覆写hudi表。之前写入正常,但hive由1.x升级到2.7.2后,报下面的错误:
Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL ALTER TABLE `dw_t1`.`contact` REPLACE COLUMNS(`_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `id` bigint, `ts` bigint, `tag_sqlrqx1` bigint, `tag_y_testmulmetadat` double, `tag_sjsmc1` bigint, `tag_qgbq1` string, `tag_sjzb1` double, `tag_sjph1` ARRAY< string>, `tag_sqlbq1` bigint, `tag_sqlsz1` int ) cascade
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:369)
at org.apache.hudi.hive.HoodieHiveClient.updateTableDefinition(HoodieHiveClient.java:251)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:206)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:154)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:108)
... 50 more
Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions :
tag_sqlrqx1,tag_sjsmc1,tag_qgbq1,tag_sjph1
at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:385)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:367)
查阅资料后,发现此问题是hive的问题,和hudi无关
问题原因:
而hive.metastore.disallow.incompatible.col.type.changes这个参数,在Hive2.0后,默认为True。即在字段类型转换时,不再允许显式转换(强制转换),而是只能进行隐式转换,即由低级类型转换为高级类型
解决方法:
在hive服务端的hive-site.xml文件中添加下面配置,然后重启hive和hivemetastore服务
<property>
<name>hive.metastore. disallow.incompatible.col.type.changes</name>
<value>false</value>
</property>
补充:自测,在hive客户端,执行set hive.metastore.disallow.incompatible.col.type.changes=false后,再执行REPLACE COLUMNS的sql,还是会报错。