hadoop入门（三）

认识Yarn

一.yarn

#cd /usr/local/hadoop/etc/hadoop
#vim yarn-site.xml    //四台都要
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
</property>
 
<property>  
    <name>yarn.nodemanager.aux-services</name>  
    <value>mapreduce_shuffle</value>  
</property>  
 
<property>
    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>


//namenode上配置mapred-site.xml(要复制)
#vi mapred-site.xml
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>


#start-yarn.sh
#jps
8225 NameNode
8418 SecondaryNameNode
8870 ResourceManager
9134 Jps
#vim mapred-site.xml
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
#vi input.txt
hello java
hello java
hello c
hello c++
hello python
hello java
#cd 
# vi input.txt
# hadoop fs -mkdir /wcinput
# hadoop fs -put input.txt /wcinput/
# hadoop fs -ls /wcinput
Found 1 items
-rw-r--r--   2 root supergroup         64 2017-10-07 00:22 /wcinput/input.txt
#find /usr/local/hadoop -name *example*.jar      //查找示例文件
#hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /wcinput/input.txt /wcoutput

通过网页来观察该job的运行情况
192.168.56.10:8088

image.png

查询次数

#hadoop fs -text /wcoutput/part-r-00000
c   1
c++ 1
hello   6
java    3
python  1

如何通过日志观察错误

#start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-master1.out
slave11: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-slave11.out
slave13: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-slave13.out
slave12: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-slave12.out
#tailf /usr/local/hadoop/logs/yarn-root-resourcemanager-master1.log  //其他同理