背景

使用idea调试spark + hive sql 程序时候，经常会碰到这个问题，比较难以解决。顾此处我总结了一下常见的解决方案。

异常

Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
    at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:639)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:567)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:185)
    at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:118)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:271)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:384)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)

代码

// 是否需要做kerberos认证，需要自己实现
System.load("D:\\hadoop2.6.0\\bin\\hadoop.dll");
SparkSession.Builder builder = SparkSession.builder().appName("test");
builder.master("local");
builder.enableHiveSupport();
SparkSession spark = builder.getOrCreate();
Configuration conf = spark.sparkContext().hadoopConfiguration();
conf.addResource("core-site.xml");
conf.addResource("hdfs-site.xml");
conf.addResource("yarn-site.xml");
conf.addResource("hive-site.xml");
Dataset<Row> df = spark.sql("select * from test.a ");
df.show();

注意：hive-site.xml等xml文件（生产或者测试环境中的配置）要放在项目根目录下，原因是 new Hiveconf()时候会默认读取根目录下的配置文件

解决方案

• 确保本地安装hadoop，我使用的版本是2.6.0，hadoop路径是D:\hadoop-2.6.0。配置环境变量HADOOP_HOME=D:\hadoop-2.6.0。
• 确保已经下载hadoop window版winutils集成包（请注意有版本区别），github下载地址https://github.com/steveloughran/winutils。或者大家直接在GitHub上搜索winutils 也会有很多的。
• 确保winutils解压后的bin包覆盖hadoop-2.6.0中的bin包，并将bin目录下的hadoop.dll文件拷贝到C:\Windows\System32下
• 如果UnsatisfiedLinkErro还存在，你需要在idea的debug/run配置 vm options 中添加 -Djava.library.path="D:\hadoop-2.6.0"，这样保证程序运行的时候加载的是2.6.0中的jar包而不是其他地方的。
• OK了，我的程序能够正常运行了。

关于大数据方面技术问题可以咨询，替你解决你的苦恼。微信 hainanzhongjian