搭建一个简单的 Hadoop 环境用于编程。

环境

硬件

软件

• VMware® Workstation 16 Pro 16.1.1 build-17801498
• CentOS Linux release 7.6.1810 (Core)
• jdk-8u202-linux-x64
• hadoop-3.3.2

系统设置

ssh 命令

安装ssh

yum install ssh

检查是否能够免密登录本机

ssh localhost

开启本机的免密访问权限

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

关闭selinux

setenforce 0
vi /etc/selinux/config

重启

shutdown -r now

下载 Hadoop

下载地址

下载页面

安装JDK

参考文章CentOS7 安装jdk

安装 Hadoop

上传安装包并解压

上传安装文件

复制安装文件到指定目录并解压

mkdir /joinway                              # 创建工作目录
cp /mnt/hadoop-3.3.2.tar.gz /joinway/      # 复制安装文件到工作目录
cd /joinway                                # 进入安装目录
chmod 755 hadoop-3.3.2.tar.gz              # 修改安装文件权限
tar -xvf hadoop-3.3.2.tar.gz               # 解压安装文件
chown -R root hadoop-3.3.2
chgrp -R root hadoop-3.3.2                 # 修改解压文件归属用户、用户组
mv hadoop-3.3.2/ hadoop/                   # 修改文件夹名，可以不做的，强迫症犯了(〃'▽'〃)

配置 JAVA_HOME

cd /joinway/hadoop/
vim etc/hadoop/hadoop-env.sh

指定java_home

# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=/usr/java/jdk1.8.0_191

配置core-site

vim /joinway/hadoop/etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

配置hdfs-site

vim /joinway/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

配置用户

vim /joinway/hadoop/sbin/start-dfs.sh
vim /joinway/hadoop/sbin/stop-dfs.sh

添加如下信息

HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

格式化系统文件

/joinway/hadoop/bin/hdfs namenode -format

简单测试hadoop

启动、关闭操作

/joinway/hadoop/sbin/start-dfs.sh
/joinway/hadoop/sbin/stop-dfs.sh

管理后台地址：http://ip:9870/

执行一个MR任务

/joinway/hadoop/bin/hdfs dfs -mkdir /user
/joinway/hadoop/bin/hdfs dfs -mkdir /user/root
/joinway/hadoop/bin/hdfs dfs -mkdir input
/joinway/hadoop/bin/hdfs dfs -put etc/hadoop/*.xml input
/joinway/hadoop/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.2.jar grep input output 'dfs[a-z.]+'

查看运行结果

/joinway/hadoop/bin/hdfs dfs -cat output/*

安装YARN

配置mapred-site

vim /joinway/hadoop/etc/hadoop/mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
         <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

配置yarn-site

vim /joinway/hadoop/etc/hadoop/yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

配置root用户

vim /joinway/hadoop/sbin/start-yarn.sh
vim /joinway/hadoop/sbin/stop-yarn.sh

添加如下信息

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

简单测试yarn

启动、关闭操作

/joinway/hadoop/sbin/start-yarn.sh
/joinway/hadoop/sbin/stop-yarn.sh

管理后台地址：http://ip:8088/

/joinway/hadoop/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.2.jar pi 2 3

可以通过浏览器查看任务的运行情况

YARN后台查看任务运行