因为要协作使用，各组件之间的版本要适应，Cloudera对大数据组件进行了匹配，这里我使用CDH5.3.6套件、JDK使用1.8：
大数据组件及JDK
准备工作
- 下载压缩包：http://archive.cloudera.com/cdh5/cdh/5/ 下载CDH5.3.6组件
- 启动系统：虚拟机下CentOS 6.7
- 网络设置：
- 创建工作用户和工作目录：
[root@localhost ~]# useradd hadoop
[root@localhost ~]# passwd
[root@localhost ~]# su hadoop
[hadoop@localhost root]# cd
[hadoop@localhost ~]# mkdir cdh
- 上传压缩包到cdh目录
- 解压并移动压缩包到Downloads：
[hadoop@localhost ~]# cd cdh
[hadoop@localhost cdh]# ls *.tar.gz | xargs -n1 tar xzvf
[hadoop@localhost cdh]# find ./ -name "*.tar.gz" | xargs -i mv {} ../Downloads/
- 环境变量：
[hadoop@localhost cdh]# vi ~/.bashrc
export JAVA_HOME=/home/hadoop/cdh/jdk1.8.0_111
export PATH=$JAVA_HOME/bin:$PATH
Hadoop
- 测试
Local (Standalone) Mode
[hadoop@localhost cdh]$ cd hadoop-2.5.0-cdh5.3.6/
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ mkdir input
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ touch input/wc.input
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ vi input/wc.input
hello world !
hello hadoop ~
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar wordcount input/wc.input output
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ cat output/part-r-00000
! 1
hadoop 1
hello 2
world 1
~ 1
Pseudo-Distributed Mode
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ vi etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/cdh/data/tmp</value>
</property>
</configuration>
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ vim etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ bin/hdfs namenode -format
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ sbin/start-dfs.sh
# 提示输入密码，按提示走。如果配置免密登录就不需要输入了。
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ bin/hdfs dfs -mkdir /user
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ bin/hdfs dfs -ls /
Fully-Distributed Mode
// 配置主机名 IP
[hadoop@localhost .ssh]$ su
Password:
[root@localhost .ssh]# vi /etc/hosts
192.168.2.131 master
192.168.2.132 slave1
192.168.2.133 slave2
// 先配置ssh免密登录
# -P表示密码，-P '' 就表示空密码，也可以不用-P参数，这样就要三车回车，用-P就一次回车。
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ cd ~/.ssh
[hadoop@localhost .ssh]$ ssh-keygen -t rsa -P ''
[hadoop@localhost .ssh]$ cat id_rsa.pub >> ./authorized_keys
[hadoop@localhost .ssh]$ chmod 600 authorized_keys
// 修改配置文件
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ vim etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
[hadoop@localhost hadoop-2.5.0-cdh5.3.6]$ vim etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>127.0.0.1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
先启动bin/hadoop start namenode
Sqoop
Apache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.
就是将常用的MapReduce（数据的导入导出）进行封装,通过传递参数的形式，运行mapreduce程序。
安装MySQL
bin/mysqld --initialize --user=hadoop --basedir=/home/hadoop/cdh/mysql --datadir=/home/hadoop/cdh/mysql/data
Flume
# 修改配置文件
$ cd /home/hadoop/cdh/apache-flume-1.5.0-cdh5.3.6-bin
$ cd conf
$ cp flume-conf.properties.template a1.conf
$ vi a1.conf
# define agent
a1.sources = r1
a1.channels = c1
a1.sinks = k1
# define sources
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# define channels
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# define sink
a1.sinks.k1.type = logger
# a1.sinks.k1.maxBytesToLog = 1024
# bind the sources and sinks to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channels = c1
先去 https://pkgs.org/ 搜索并下载以下三个程序，上传到Downloads目录下
Telnet
# 使用root用户安装上述三个程序 然后重启服务
$ rpm -ivh ./*.rpm
$ /etc/rc.d/init.d/xinetd restart
$ exit
# 启动服务
$ bin/flume-ng agent \
> -c conf \
> -n a1 \
> -f conf/a1.conf \
> -Dflume.root.logger=DEBUG,console
Oozie
system requirement
Linux
Java 1.6
hadoop
ExtJs Library
Hue
