腾讯云中伪分布式配置:
首先给主机定义一个名称:注意这里需要配置本机的内网机器,其它机器的外网地址
10.104.222.163 hadoopmaster
127.0.0.1 VM_222_163_centos VM_222_163_centos
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4
# The following lines are desirable for IPv6 capable hosts
::1 VM_222_163_centos VM_222_163_centos
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
hadoop安装目录假定为${HADOOOP_HOME},当前hadoop版本为2.9.1:
hadoop版本
1 在${HADOOOP_HOME}/etc/hadoop目录下,修改下面几个文件:
core-site.xml
<configuration>
<!-- 指定HDFS namenode 的通信地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoopmaster:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储路径 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/hadoop-2.9.1/hadoop</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/usr/local/hadoop/hdfs/name</value>
<description>namenode上存储hdfs名字空间元数据 </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/hadoop/hdfs/data</value>
<description>datanode上数据块的物理存储位置</description>
</property>
<!-- 设置hdfs副本数量 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
通过拷贝生成mapred-site.xml
cp mapred-site.xml.template mapred-site.xml
内容如下:
<configuration>
<!-- 通知框架MR使用YARN -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<!-- reducer取数据的方式是mapreduce_shuffle -->
<property>
<name>yarn.acl.enable</name>
<value>0</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoopmaster</value>
</property>
</configuration>
启动hdfs
${HADOOOP_HOME}/sbin/start-dfs.sh
启动yarn
${HADOOOP_HOME}/sbin/start-yarn.sh
检查hadoop相关进程启动情况:
hadoop进程
如果想要关闭hadoop进程,可以执行:
${HADOOOP_HOME}/sbin/stop-dfs.sh
${HADOOOP_HOME}/sbin/stop-yarn.sh
web中查看hadoop状态:http://outerIP:50070
hadoop状态
web中查看集群中应用程序状态:http://outerIP:8088
集群状态