idea 工具创建java项目,添加hadoop依赖
maven依赖
<properties>
<java.version>1.8</java.version>
<hadoop.version>2.8.5</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependencies>
windows系统解压hadoop-2.8.5.tar.gz
将winutils下选相应的版本的bin目录下的文件复制到hadoop解压出来的bin目录下
报错处理
hadoop权限问题
org.apache.hadoop.ipc.RemoteException: Permission denied: user=administration, access=WRITE, inode="/":root:supergroup:drwxr-xr-x
解决 :
设置名称,设置hadoop的用户名,我这里是root
System.setProperty("HADOOP_USER_NAME","root");
找不到hadoop.home.dir
16:42:22.156 [main] DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
解决 :
设置访问路径
//就是hadoop的HADOOP_HOME路径,可能是设置hadoop的path没有生效需要这个设置
System.setProperty("hadoop.home.dir", "D:\\user\\hadoop\\");
注意: 远程的hadoop的配置文件中不能使用localhost或者127.0.0.1,需要换成对应的ip
<property>
<name>fs.defaultFS</name>
<value>hdfs://127.0.0.1:9000</value>
</property>
