在了解了Hadoop中的存储组件HDFS之后，我们再来看一下Hadoop中另一个重要组件的计算MapReduce。HDFS搞定海量的存储，MapReduce搞定海量的计算。hadoop如其他优秀的开源组件一样，也提供了丰富的demo，下面我们就来看一下如何使用mapreduce自带demo进行词频统计。
1 系统、环境和约束条件
- 在CentOS7中安装hadoop并启动，作者的hadoop安装目录在/root/hadoop-2.5.2
https://www.jianshu.com/p/b7ae3b51e559
2 操作
- 1 使用putty登录centos
- 2 执行以下命令：
# 切换到家目录
cd
# 进入hadoop的bin目录
cd hadoop-2.5.2/bin
# vim word，在其中加入以下内容并保存退出，读者可以随意加入别的内容，这是我们待会要统计词频的文件
hello i am zhangli
hello i am xiaoli
hi i am ali
who are you
i am xiaoli
# 上传word文件
./hdfs dfs -put word /word
# 查看上传结果
./hdfs dfs -cat /word
# 开始统计，其中
# ./yarn是执行命令
# jar是表示执行的是jar包
# /root/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar 表示要执行的jar包
# wordcount 是要执行过程的名字
# /word 是我们上传的待分析的文件在HDFS中的路径
# /output 是我们分析之后结果的输出路径
./yarn jar /root/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /word /output
# 等待一阵子，会有以下输出
19/05/30 12:29:41 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.100.192:8032
19/05/30 12:29:46 INFO input.FileInputFormat: Total input paths to process : 1
19/05/30 12:29:47 INFO mapreduce.JobSubmitter: number of splits:1
19/05/30 12:29:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559056674360_0002
19/05/30 12:29:50 INFO impl.YarnClientImpl: Submitted application application_1559056674360_0002
19/05/30 12:29:51 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1559056674360_0002/
19/05/30 12:29:51 INFO mapreduce.Job: Running job: job_1559056674360_0002
19/05/30 12:30:19 INFO mapreduce.Job: Job job_1559056674360_0002 running in uber mode : false
19/05/30 12:30:19 INFO mapreduce.Job: map 0% reduce 0%
19/05/30 12:30:36 INFO mapreduce.Job: map 100% reduce 0%
19/05/30 12:30:46 INFO mapreduce.Job: map 100% reduce 100%
19/05/30 12:30:49 INFO mapreduce.Job: Job job_1559056674360_0002 completed successfully
19/05/30 12:30:50 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=111
FILE: Number of bytes written=194141
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=156
HDFS: Number of bytes written=65
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=15451
Total time spent by all reduces in occupied slots (ms)=7614
Total time spent by all map tasks (ms)=15451
Total time spent by all reduce tasks (ms)=7614
Total vcore-seconds taken by all map tasks=15451
Total vcore-seconds taken by all reduce tasks=7614
Total megabyte-seconds taken by all map tasks=15821824
Total megabyte-seconds taken by all reduce tasks=7796736
Map-Reduce Framework
Map input records=5
Map output records=17
Map output bytes=135
Map output materialized bytes=111
Input split bytes=89
Combine input records=17
Combine output records=10
Reduce input groups=10
Reduce shuffle bytes=111
Reduce input records=10
Reduce output records=10
Spilled Records=20
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=697
CPU time spent (ms)=8200
Physical memory (bytes) snapshot=445980672
Virtual memory (bytes) snapshot=4215586816
Total committed heap usage (bytes)=322437120
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=67
File Output Format Counters
Bytes Written=65
# 查看/output输出，在以下路径中会看到有两个文件，其中_SUCCESS代表成功，part-r-00000代表输出结果
./hdfs dfs -ls /output
以下为输出：
Found 2 items
-rw-r--r-- 2 root supergroup 0 2019-05-30 12:30 /output/_SUCCESS
-rw-r--r-- 2 root supergroup 65 2019-05-30 12:30 /output/part-r-00000
# 查看词频统计结果
./hdfs dfs -cat /output/part-r-00000
# 以下为输出
ali 1
am 4
are 1
hello 1
hi 1
i 4
who 1
xiaoli 2
you 1
zhangli 1
以上就是利用hadoop自带的词频统计demo进行统计并查看统计结果的过程。
