前言：本文主要记录CDH6.3.1集群安装记录。总共在虚拟机上实践了2次，第二次增加了MySQL的双主服务和keepalived 服务。后续会实体机实操，和记录正式服务器的。第一次用了3台master，10台node，第二次用3台master，4台node。等服务器购买后实操的话，会有14个节点，预计如下图。具体硬件信息，等续物理机安装时再记录。如有问题，还请指教，qq：315675542。
1. 软件准备：
System：CentOS7.8(CentOS Linux release 7.8.2003(core)) 64bit
Packages needed to download offline：
mysql-5.7.31-1.el7.x86_64.rpm-bundle.tar
jdk-8u181-linux-x64.tar.gz
mysql-connector-java-5.1.46.jar
keepalived-2.0.8.tar.gz
cm6.3.1-redhat7.tar.gz
allkeys.asc
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
manifest.json
DownUrls：
http://mirrors.163.com/mysql/Downloads/MySQL-5.7/mysql-5.7.31-1.el7.x86_64.rpm-bundle.tar
https://repo.huaweicloud.com/java/jdk/8u181-b13/jdk-8u181-linux-x64.tar.gz
http://mirrors.sohu.com/mysql/Connector-J/mysql-connector-java-5.1.46.tar.gz
https://www.keepalived.org/software/keepalived-2.0.8.tar.gz
https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/cm6.3.1-redhat7.tar.gz
https://archive.cloudera.com/cm6/6.3.1/allkeys.asc
https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
https://archive.cloudera.com/cdh6/6.3.2/parcels/manifest.json
2.安装过程
JDK配置（所有节点）
查看原有JDK: java –version //基本都是OpenJDK, CDH需要Oracle 的JDK,
查看所有JDK安装包：rpm -qa|grepjava
java-1.7.X-openjdk-1.6.0.0-1.66.1.13.0.el786_64
…………
卸载它们：
rpm -e --nodeps java-1.7.0-openjdk
rpm -e --nodeps java-1.7.0-openjdk-headless
rpm -e --nodeps java-1.8.0-openjdk
rpm -e --nodeps java-1.8.0-openjdk-headless
安装JDK：
tar -zxvf jdk-8u181-linux-x64.tar.gz -C/usr/lib/jvm
把安装目录rename成java-8，即为：/usr/lib/jvm/java-8
cd /usr/lib/jvm
mv jdk1.8.0_181/ java-8
然后在/etc/profile和/etc/bashrc两文件中添加：
export JAVA_HOME=/usr/lib/jvm/java-8
export JRE_HOME=/usr/lib/jvm/java-8/jre
export CLASSPATH=.:/usr/lib/jvm/java-8/lib:/usr/lib/jvm/java-8/jre/lib
export PATH=$PATH:/usr/lib/jvm/java-8/bin
source /etc/profile
source /etc/bashrc
在/usr/中创建：mkdir /usr/java （5.9.2版本Spark内核只认这个Java路径）
在/usr/java中创建软连接：ln-s /usr/lib/jvm/java-8 /usr/java/default
最后查看java –version验证
修改Hostname（所有节点）
hostnamectl set-hostname master-41
hostnamectl set-hostname master-42
…………
………
重启生效：reboot
在/etc/hosts中添加IP，hostname关系表
vi /etc/hosts
192.168.1.41 master-41
192.168.1.42 master-42
192.168.1.43 master-43
192.168.1.31 node-31
192.168.1.32 node-32
192.168.1.33 node-33
192.168.1.34 node-34
192.168.1.35 node-35
192.168.1.36 node-36
192.168.1.37 node-37
192.168.1.38 node-38
192.168.1.39 node-39
192.168.1.40 node-40
重启：
/etc/init.d/network restart
SSH 免密配置
Master中:
ssh-keygen –t rsa // 一路回车
cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys
Node-1, Node-2, Node-3….中分别操作:
ssh-keygen –t rsa // 一路回车
可以使用xshell的“发送键盘输入的所有会话”
cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys_$(hostname)
scp ~/.ssh/authorized_keys_$(hostname) root@master：~/.ssh
或者逐个进行：
~/.ssh/authorized_keys_node1 //在node1
scp ~/.ssh/authorized_keys_node1 root@master：~/.ssh
cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys_node2 //在node2
scp ~/.ssh/authorized_keys_node2 root@master：~/.ssh
cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys_node3 //在node3
scp ~/.ssh/authorized_keys_node3 root@master：~/.ssh
Master中：
cat ~/.ssh/authorized_keys_node1 >>~/.ssh/authorized_keys
cat ~/.ssh/authorized_keys_node2 >>~/.ssh/authorized_keys
cat ~/.ssh/authorized_keys_node3 >>~/.ssh/authorized_keys
最后再把master中的 ~/.ssh/authorized_keys 分发到各个节点
scp ~/.ssh/authorized_keys root@node1:~/.ssh
scp ~/.ssh/authorized_keys root@node2:~/.ssh
scp ~/.ssh/authorized_keys root@node3:~/.ssh
所以节点两两测试免密登录， 第一次可能还会提醒是否需要连接，另外可以删除~/.ssh目录下的kown_hosts文件，重新ssh 测试。
ssh node1
ssh node2
防火墙和其他安全设置（所有节点）
FireWall：
service firewalld stop (临时关闭)
systemctl disable firewalld (重启后生效)
SELINUX：
setenforce 0 (临时生效)
修改/etc/selinux/config 下的 SELINUX=disabled（重启后生效）
Swappiness：
echo 0 > /proc/sys/vm/swappiness (临时生效)
vi /etc/sysctl.conf 文末添加vm.swappiness= 10 (重启生效)
设置透明大页面压缩：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
NTP服务
systemctl start ntpd.service
systemctl enable ntpd.service
vim /etc/ntp.conf （主master：192.168.1.41）
添加：
server 127.127.1.0 iburst local clock
fudge 127.127.1.0 stratum 10
systemctl restart ntpd
vi /etc/ntp.conf (主master之外的所有其他节点)
systemctl restart ntpd （需要等15分钟）
所有机台同步系统时间到NTP Server:
ntpdate -u 192.168.1.41
再查看状态：ntpstat
所有机台同步硬件时钟到系统时钟：
hwclock --systohc
Mysql安装（对所有节点）和双主设置（对VIP）
Mysql安装：
1．Centos7卸载mariadb
rpm -qa|grep mariadb
rpm -e --nodeps mariadb-libs-5.5.65-1.el7.x86_64
2.安装
安装包：
mysql安装包
解压：
tar -xvf mysql-5.7.31-1.el7.x86_64.rpm-bundle.tar
依次安装：
rpm -ivh mysql-community-common-5.7.31-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.31-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.31-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.31-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.31-1.el7.x86_64.rpm
检查是否成功，是否启动mysql服务
service mysqld status //检查
service mysqld start //启动
检查密码并登录：
grep password /var/log/mysqld.log
mysql -uroot -p
修改密码：
set global validate_password_policy=0;
set global validate_password_length=1;
set password = password("123456");
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
flush privileges;
mysql双主服务和keepalived服务（选取2个node节点）
mysql双主服务 node-48和node-49示例：
1，配置文件
分别对这两个节点：
vi /etc/my.cnf
Node-48中添加
# mysql服务id
server-id=1
#二进制日志 存储数据发生改变的sql语句
log-bin=mysql-bin
#增量值得起点
auto-increment-offset=1
#每次增量间隔
auto-increment-increment=2
#设置需要同步的数据库名称，目前不需要设置，保持注释状态
#binlog_do_db=<dbname>
node-49中添加：
# mysql服务id
server-id=2
#二进制日志 存储数据发生改变的sql语句
log-bin=mysql-bin
#增量值得起点
auto-increment-offset=2
#每次增量间隔
auto-increment-increment=2
#设置需要同步的数据库名称，目前不需要设置，保持注释状态
#binlog_do_db=<dbname>
保存退后，都重启mysql服务
service mysqld restart
2，数据同步授权
如果遇到密码设置过去简单问题，先执行：
set global validate_password_policy=0;
set global validate_password_length=1;
Node-48中：
mysql> grant replication slave,replication client on *.* to root@'192.168.1.%' identified by "123456";
Query OK, 0 rows
affected, 1 warning (0.00sec)
mysql>flush privileges;
Query OK, 0 rows
affected (0.00sec)
最好将库锁住，仅仅允许读，以保证数据一致性；待主主同步环境部署后再解锁；
锁住后，就不能往表里写数据，但是重启mysql服务后就会自动解锁！
注意该参数设置后，如果自己同步对方数据，同步前一定要记得先解锁！
mysql>flushtables with read lock;
Query OK, 0 rows
affected (0.01sec)
Node-49中：
mysql> grant replication slave,replication client on *.* to root@'192.168.1.%' identified by "123456";
Query OK, 0 rows affected, 1 warning (0.01 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> flush tables with read lock;
Query OK, 0 rows affected (0.01 sec)
同步操作：
解锁上个步骤中对node-48中的表解锁，保持数据的一致性。
mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)
先去stop slave 以下报错是slave未开启
mysql> slave stop;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'slave stop' at line 1
mysql> change master to master_host='192.168.1.49',
master_user='root',master_password='123456',
master_log_file='mysql-bin.000006',master_log_pos=630;
Query OK, 0 rows affected, 2 warnings (0.20 sec)
mysql> start slave;
Query OK, 0 rows affected (0.01 sec)
mysql> show slave status \G;
解锁上个步骤中对node-49中的表解锁，保持数据的一致性。
mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)
先去stop slave 以下报错是slave未开启
mysql> slave stop;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'slave stop' at line 1
mysql> change master to master_host='192.168.1.48',
master_user='root',master_password='123456',
master_log_file='mysql-bin.000003',master_log_pos=1106;
Query OK, 0 rows affected, 2 warnings (0.20 sec)
mysql> start slave;
Query OK, 0 rows affected (0.01 sec)
mysql> show slave status \G;
设置root远程登录：
Node-49都操作设置root远程登录：
mysql> grant all on *.* to root@'192.168.1.%' identified by "123456";
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)
Keepalived 服务：
安装
cd /usr/local/src
wget http://www.keepalived.org/software/keepalived-2.0.8.tar.gz #已经下载
tar zxvf keepalived-2.0.8.tar.gz /usr/local/src/
cd keepalived-2.0.8/
yum install -y openssl-devel libnl-devel libnl3-devel libnfnetlink-devel
yum groupinstall "Development Tools"
./configure --prefix=/usr/local/keepalived
make && make install
cd /usr/local
tar zcvf keepalived.tar keepalived
cp keepalived/bin/* /bin/
cp keepalived/sbin/* /sbin/
cp keepalived/etc/keepalived/ /etc/ -rf
cp keepalived/etc/sysconfig/* /etc/sysconfig/
配置
Node-48中：vi /etc/keepalived/keepalived.conf
! ConfigurationFile for keepalived
global_defs {
router_id MASTER-HA
}
vrrp_instanceVI_1 {
state MASTER
interface ens192 #根据实际网络接口进行更改
virtual_router_id 51
priority 100 #优先权，backup设置为90
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.100
}
}
virtual_server192.168.1.100 3306 {
delay_loop 2
lb_algo rr
lb_kind DR
persistence_timeout 60
protocol TCP
real_server 192.168.1.48 3306 {
weight 3
notify_down /etc/keepalived/mysql.sh
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
保存，退出后：
service restart keepalived
systemctl enable keepalived
Node-49中：vi /etc/keepalived/keepalived.conf
! ConfigurationFile for keepalived
global_defs {
router_id MASTER-HA
}
vrrp_instanceVI_1 {
state BACKUP
interface ens192 #根据实际网络接口进行更改
virtual_router_id 51
priority 90
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.100
}
}
virtual_server192.168.1.100 3306 {
delay_loop 2
lb_algo rr
lb_kind DR
persistence_timeout 60
protocol TCP
real_server 192.168.1.48 3306 {
weight 3
notify_down /etc/keepalived/mysql.sh
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
保存，退出后：
service restart keepalived
systemctl enable keepalived
以下node-48，node-49都操作：
Mysql.sh 脚本：主要监控3306端口是否被监听，若不被监听关闭keepalived服务
#!/bin/bash
counter=$(netstat -na|grep "LISTEN"|grep "3306"|wc -l)
if [ "${counter}" -eq 0 ]; then
service keepalived stop
fi
启动服务：
service keepalived start
service keepalived status
设置开机启动。
Vim /opt/start_services.sh
#!/bin/bash
service mysqld start
while true
do
counter=$(netstat -na|grep "LISTEN"|grep "3306"|wc -l)
if [ "${counter}" -ne 0 ]; then
service keepalived start
exit 0
fi
done
/etc/rc.d/rc.local 文件末尾追加sh /opt/start_services.sh,并修改/opt/start_services.sh和/etc/rc.d/rc.local的执行权限
chmod +x /opt/start_services.sh
chmod +x /etc/rc.d/rc.local
验证：
命令：IP a #看是否漂移
Shutdown 某一个台，看仍否能够通过VIP进入mysql，并成功建表，等该机台重启数据是否同步。
创建DB、用户以及权限：
连接VIP，进入mysql
mysql -h192.168.1.100 -uroot -p
创建mysql脚本：createDB.sql (内容如下或参见createDB-sql.txt)
vi /root/createDB.sql
GRANT ALLPRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON *.* TO 'amon'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
CREATE DATABASErmgr DEFAULT CHARACTER SET utf8;
CREATE USER'rmgr'@'%'IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON rmgr .* TO 'rmgr'@'%' IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON *.* TO 'rmgr'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
CREATE DATABASEhive DEFAULT CHARACTER SET utf8;
CREATE USER'hive'@'%'IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON hive.* TO 'hive'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGESON *.* TO 'hive'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
CREATE DATABASEhue DEFAULT CHARACTER SET utf8;
CREATE USER'hue'@'%'IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON *.* TO 'hue'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
CREATE DATABASEoozie DEFAULT CHARACTER SET utf8;
CREATE USER'oozie'@'%'IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';
GRANT ALLPRIVILEGES ON *.* TO 'oozie'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
FLUSH PRIVILEGES;
进入mysql环境：
mysql -uroot-p
source /root/createDB.sql
所有节点部署JDBC
mkdir-p /usr/share/java
mvmysql-connector-java-5.1.46.jar mysql-connector-java.jar
cp mysql-connector-java-5.1.46.jar/usr/share/java/
cp mysql-connector-java.jar/usr/lib/java
所有节点安装httpd服务
yum install httpd
systemctl starthttpd
systemctl enable httpd.service
CM安装（在主master中）
mkdir -p /var/www/html/cloudera-repos/
tar -zxvf cm6.3.1-redhat7.tar.gz-C /var/www/html/cloudera-repos/
拷贝整理cloudera-repos文件夹中内容，必须和下图保持一致：
vim /etc/yum.repos.d/cloudera-manager.repo
写入：
[cloudera-manager]
name=ClouderaManager, Version cm6.3.1
baseurl=http://master-42/cloudera-repos/
gpgcheck=true
enabled=true
yum clean all
yum makecache
rpm --import http://master-42/cloudera-repos/RPM-GPG-KEY-cloudera
yum开始安装：
yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
安装好后，在/opt/cloudera/parcel-repo中放入：
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
manifest.json
重命名：
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1=> CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h虚拟IP scm scm 123456
启动Server服务：
systemctl start cloudera-scm-server
网页进入安装页面：
master-42:7180
后面一路简单配置，基本没什么问题。
注意: 安装服务界面后，数据库设置的时候，使用VIP。
常用查错命令：
tail -f/var/log/cloudera-scm-server/cloudera-scm-server.log
tail -f/var/log/cloudera-scm-agent/cloudera-scm-agent.log
service cloudera-scm-server start
后记：
cm安装页面中hdfs，hive，hue等各项服务如何配置，还需要具体看机器的内存和cpu配置而定。
