image

useradd hadoop

/etc/hadoop/hadoop-env.sh

JAVA_HOME=${JAVA_HOME}
JAVA_HOME=/usr/local/java

配置文件

公共参数

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- name node-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://mini1:9000</value>
</property>

<!-- 进程工作的数据目录-->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hdpdata</value>
</property>

</configuration>

hdfs配置（可直接使用默认值）

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- 副本的数量，默认值 3 -->
<property>
<name>dfs.replication</name>
<value>2</value>
</property>

</configuration>

mapreduce配置

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!-- 运行平台 默认local -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

</configuration>

yarn-site.xml


<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>mini1</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

hadoop添加环境变量

sudo vim /etc/profile

# HADOOP
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.8.5
export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

hdfs格式化

hadoop namenode -format

image

启动hadoop

namenode

hadoop-daemon start namenode
hadoop-daemon stop namenode
http://****:50070
(http://10.0.1.101:50070/dfshealth.html#tab-overview)
hadoop-daemon start datanode
hadoop-daemon stop datanode

批量启动

配置节点
vi etc/hadoop/slaves

mini2
mini3
mini4

执行命令
start-all.sh(不推荐)
start-hdfs.sh
start-yarn.sh

问题

[hadoop@mini1 hadoop]$ ssh-copy-id mini1

/usr/bin/ssh-copy-id: ERROR: No identities found
~/.ssh/id_rsa id_rsa.pub文件不存在
执行ssh-keygen 解决
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

【大数据运维之三】Hadoop集群搭建