环境准备：三个节点
bigdata01 192.168.182.100
bigdata02 192.168.182.101
bigdata03 192.168.182.102
修改每台的主机名
[root@bigdata01 ~]# hostname bigdata01
[root@bigdata01 ~]# vi /etc/hostname
bigdata01
修改每台机器的hosts文件
[root@bigdata01 ~]# vi /etc/hosts
192.168.182.100 bigdata01
192.168.182.101 bigdata02
192.168.182.102 bigdata03
临时关闭防火墙+永久关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
ssh免密码登录
首先在bigdata01机器上执行下面命令，将公钥信息拷贝到两个从节点
[root@bigdata01 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:I8J8RDun4bklmx9T45SRsKAu7FvP2HqtriYUqUqF1q4 root@bigdata01
The key's randomart image is:
+---[RSA 2048]----+
| o . |
| o o o . |
| o.. = o o |
| +o* o * o |
|..=.= B S = |
|.o.o o B = . |
|o.o . +.o . |
|.E.o.=...o |
| .o+=*.. |
+----[SHA256]-----+
[root@bigdata01 ~]# ll ~/.ssh/
total 12
-rw-------. 1 root root 1679 Apr 7 16:39 id_rsa
-rw-r--r--. 1 root root 396 Apr 7 16:39 id_rsa.pub
[root@bigdata01 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@bigdata01 ~]# scp ~/.ssh/authorized_keys bigdata02:~/
[root@bigdata01 ~]# scp ~/.ssh/authorized_keys bigdata03:~/
然后在bigdata02和bigdata03上执行
[root@bigdata02 ~]# cat ~/authorized_keys >> ~/.ssh/authorized_keys
[root@bigdata03 ~]# cat ~/authorized_keys >> ~/.ssh/authorized_keys
集群节点之间时间同步
安装
[root@bigdata01 ~]# yum install -y ntpdate
确认是否可以正常执行
[root@bigdata01 ~]# ntpdate -u ntp.sjtu.edu.cn
7 Apr 21:21:01 ntpdate[5447]: step time server 185.255.55.20 offset 6.252298 sec
配置定时任务
[root@bigdata01 ~]# vi /etc/crontab
* * * * * root /usr/sbin/ntpdate -u ntp.sjtu.edu.cn
bigdata02和bigdata03节点上重复上述操作
安装jdk1.8
略
安装Hadoop
首先把hadoop的安装包上传到/data/soft目录下，解压
[root@bigdata01 soft]# tar -zxvf hadoop-3.2.0.tar.gz
修改环境变量
[root@bigdata01 hadoop-3.2.0]# vi /etc/profile
.......
export JAVA_HOME=/data/soft/jdk1.8
export HADOOP_HOME=/data/soft/hadoop-3.2.0
export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH
[root@bigdata01 hadoop-3.2.0]# source /etc/profile
进入配置文件所在目录
[root@bigdata01 hadoop-3.2.0]# cd etc/hadoop/
修改 hadoop-env.sh 文件
[root@bigdata01 hadoop]# vi hadoop-env.sh
.......
export JAVA_HOME=/data/soft/jdk1.8
export HADOOP_LOG_DIR=/data/hadoop_repo/logs/hadoop
修改 core-site.xml 文件
[root@bigdata01 hadoop]# vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://bigdata01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/data/hadoop_repo</value>
</property>
</configuration>
修改hdfs-site.xml文件,把hdfs中文件副本的数量设置为2，最多为2，因为现在集群中有两个从节点，还有secondaryNamenode进程所在的节点信息
[root@bigdata01 hadoop]# vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>bigdata01:50090</value>
</property>
</configuration>
修改mapred-site.xml，设置mapreduce使用的资源调度框架
[root@bigdata01 hadoop]# vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
修改yarn-site.xml，设置yarn上支持运行的服务和环境变量白名单
[root@bigdata01 hadoop]# vi yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bigdata01</value>
</property>
</configuration>
修改workers文件，增加所有从节点的主机名，一个一行
[root@bigdata01 hadoop]# vi workers
bigdata02
bigdata03
修改start-dfs.sh，stop-dfs.sh这两个脚本文件，在文件前面增加如下内容
[root@bigdata01 hadoop]# cd /data/soft/hadoop-3.2.0/sbin
[root@bigdata01 sbin]# vi start-dfs.sh
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
[root@bigdata01 sbin]# vi stop-dfs.sh
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
修改start-yarn.sh，stop-yarn.sh这两个脚本文件，在文件前面增加如下内容
[root@bigdata01 sbin]# vi start-yarn.sh
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
[root@bigdata01 sbin]# vi stop-yarn.sh
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
把bigdata01节点上将修改好配置的安装包拷贝到其他两个从节点
[root@bigdata01 sbin]# cd /data/soft/
[root@bigdata01 soft]# scp -rq hadoop-3.2.0 bigdata02:/data/soft/
[root@bigdata01 soft]# scp -rq hadoop-3.2.0 bigdata03:/data/soft/
在bigdata01节点上格式化HDFS
[root@bigdata01 soft]# cd /data/soft/hadoop-3.2.0
[root@bigdata01 hadoop-3.2.0]# bin/hdfs namenode -format
启动集群
在bigdata01节点上执行下面命令
[root@bigdata01 hadoop-3.2.0]# sbin/start-all.sh
Starting namenodes on [bigdata01]
Last login: Tue Apr 7 21:03:21 CST 2020 from 192.168.182.1 on pts/2
Starting datanodes
Last login: Tue Apr 7 22:15:51 CST 2020 on pts/1
bigdata02: WARNING: /data/hadoop_repo/logs/hadoop does not exist. Creating.
bigdata03: WARNING: /data/hadoop_repo/logs/hadoop does not exist. Creating.
Starting secondary namenodes [bigdata01]
Last login: Tue Apr 7 22:15:53 CST 2020 on pts/1
Starting resourcemanager
Last login: Tue Apr 7 22:15:58 CST 2020 on pts/1
Starting nodemanagers
Last login: Tue Apr 7 22:16:04 CST 2020 on pts/1
验证集群
分别在3台机器上执行jps命令，进程信息如下所示：
在bigdata01节点执行
[root@bigdata01 hadoop-3.2.0]# jps
6128 NameNode
6621 ResourceManager
6382 SecondaryNameNode
在bigdata02节点执行
[root@bigdata02 ~]# jps
2385 NodeManager
2276 DataNode
在bigdata03节点执行
[root@bigdata03 ~]# jps
2326 NodeManager
2217 DataNode
