告别中文互联网搜索的毒瘤CSDN,把CSDN上自己原创的还算有价值的博客搬运过来=。=
在自己的mac上安装了hadoop,折腾了一下午总算成功了,总结一下遇到的几个问题。报错信息没有保存,文中的报错信息都是复制的网上的,可能会有出入,后面附上了搜到答案的链接。有问题最好还是上外网搜,没条件用谷歌就用必应的国际版,百度上的答案基本上都是复读机,一个不能用就都不能用。
基本安装流程
参考以下链接做的,如果一帆风顺就不用往后看了。
https://www.jianshu.com/p/a86ccd96b78a
遇到的问题
1、Java版本问题
报错信息不记得了,大致包含export: PATH;: not a valid identifier
,具体解决流程包括
- 安装Java环境,这里不介绍了
- 部署hadoop之前最好检查一下,命令行输入
open ~/.bash_profile
,查看JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home
这一行 - 输入
open /usr/local/Cellar/hadoop/3.2.1/libexec/etc/hadoop/hadoop-env.sh
,查看大致54行,export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home
,这里的JAVA_HOME内容需要和上一步的一样,尤其要注意jdk版本是否一致;
2、bash问题
报错信息大致为
[root@master logs]# start-dfs.sh
Starting namenodes on [master]
Last login: Wed Jul 4 16:30:19 CST 2018 on pts/0
bash v3.2+ is required. Sorry.
Starting datanodes
Last login: Wed Jul 4 16:31:04 CST 2018 on pts/0
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master]
Last login: Wed Jul 4 16:31:04 CST 2018 on pts/0
bash v3.2+ is required. Sorry.
可能是使用了zsh,输入chsh -s /bin/bash
切换bash,参考链接https://segmentfault.com/q/1010000015478820
3、用户名登录问题
报错信息包括
[root@master sbin]# ./start-dfs.sh
Starting namenodes on [master]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [slave1]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
这个问题困扰了我很久,百度上的方法都是修改start-dfs.sh等4个文件,加上用户名等信息,参考
https://blog.csdn.net/lglglgl/article/details/80553828,但是我实际上操作后还是有同样的问题,在外网上搜到了解决方法,实际操作后可行,步骤包括
- 打开hadoop-env.sh,打开方式前面有
- 在文件最后添加如下代码,参考http://www.installationbyravi.co.in/2018/04/error-attempting-to-operate-on-hdfs.html
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"
- 如果不再出现上述问题说明正确,如果出现类似于
Starting namenodes on [localhost]
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: ` done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 244: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 252: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 213: hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 214: hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 263: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 270: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 273: hadoop_generic_java_subcmd_handler: command not found
这样大段错误代码,把hadoop-env.sh中的root替换为自己的用户名,命令行里能看到,参考https://stackoverflow.com/questions/54696265/anyone-know-how-to-fix-hadoop-functions-sh-syntax-error-near-unexpected-token
4.ssh问题
报错信息包括
U:sbin U$ ./start-dfs.sh
Starting namenodes on [localhost]
localhost: U@localhost: Permission denied (publickey,password,keyboard-interactive).
Starting datanodes
localhost: U@localhost: Permission denied (publickey,password,keyboard-interactive).
Starting secondary namenodes [U.local]
U.local: U@pc.local: Permission denied (publickey,password,keyboard-interactive).
2018-02-25 14:52:15,505 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- 生成ssh key
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
- 注册这个key
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
- 登录到localhost,命令行输入
ssh localhost