步骤
- 配置环境
- ntp 安装
- 阿里 yum repo
- ssh 免密
- 安装java
- 安装配置cm
- 安装配置mysql
机器
| IP | Hostname | remark |
|---|---|---|
| 172.30.3.210 | wei-data1 | 主节点 |
| 172.30.3.211 | wei-data2 | 从节点 |
| 172.30.3.212 | wei-data3 | 从节点 |
环境
-
Network
- 修改ip [each machine]
- 修改网络类型(static) [each machine]
- 修改DNS [each machine]
- 添加Host [each machine]
172.30.3.210 wei-data1
172.30.3.211 wei-data2
172.30.3.212 wei-data3
- Firewall[each machine]
systemctl stop firewalld.service
systemctl disable firewalld.service
vi /etc/selinux/config
SELINUX=disabled
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
- ssh免密登陆
#从节点
ssh-keygen -t rsa
#主节点
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub root@wei-data1
scp ~/.ssh/authorized_keys root@wei-data2:~/.ssh/
scp ~/.ssh/authorized_keys root@wei-data3:~/.ssh/
-
阿里 yum repo [each machine]
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
-
lib install [each machine]
yum install net-tools vim -y
-
安装NTP [each machine]
- 安装
yum install -y ntp
- 配置
vim /etc/ntp.conf
#主节点
server time.windows.com prefer #time.windows.com是时间服务器，在内网环境可让用户提供统一的时间服务器地址
//注释掉
#server 0.centos.pool.ntp.org iburst #在这行上面增加内容
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
#从节点
server wei-data1
//注释掉
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
# 所有节点
systemctl enable ntpd
systemctl start ntpd
-
java安装 [each machine]（省略）
rpm -ivh -i java...
-
CM安装[each machine]
- yum repo 地址
https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
安装yum repo
wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/
安装cm[后续会自动安装agent,所以这里不装 ]
yum install cloudera-manager-daemons cloudera-manager-server
-
mysql安装 [master node]
- 安装
wget http://repo.mysql.com/mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql-community-server
- 启动
# 开机启动
systemctl enable mysqld.service
# 启动
systemctl start mysqld.service
修改mysql配置
停服务
systemctl stop mysqld.service
-
改配置
vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
启动mysql
systemctl start mysqld
-
初始化mysql (需要复杂密码)
/usr/bin/mysql_secure_installation
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!
-
Installing the MySQL JDBC Driver
- Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html
- Extract the JDBC driver JAR file from the downloaded file. For example:
tar zxvf mysql-connector-java-5.1.46.tar.gz - Copy the JDBC driver, renamed, to /usr/share/java/. If the target directory does not yet exist, create it. For example:
mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
-
创建数据库
mysql -u root -p
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Bigdata@123';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Bigdata@123';
-
Running the script when MySQL or MariaDB is co-located with the Cloudera Manager Server
/usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm
-
安装CDH
- Start Cloudera Manager Server:
sudo systemctl start cloudera-scm-server - Wait several minutes for the Cloudera Manager Server to start. To observe the startup process, run the following on the Cloudera Manager Server host:
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
- Start Cloudera Manager Server:
#When you see this log entry, the Cloudera Manager Admin Console is ready:
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
-
In a web browser, go to http://<server_host>:7180, where <server_host> is the FQDN or IP address of the host where the Cloudera Manager Server is running.
Log into Cloudera Manager Admin Console. The default credentials are:
Username: admin
Password: admin
-
其他
检查主机正确性时出现 “Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 30。” 的警告，进行如下设定
# vi /etc/sysctl.conf
vm.swappiness = 0
# sysctl –p
检查主机正确性时出现 “已启用“透明大页面”，它可能会导致重大的性能问题。” 的警告，进行如下设定
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
安装引导(略) 网上很多
-
kafka集成
image.png
- 注意事项
jdk要1.8
修改内存限制到256m
- spark2 安装
image.jpeg
-在官网下载
# cd /opt/cloudera/csd/
wget http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.3.0.cloudera4.jar
chown cloudera-scm:cloudera-scm SPARK2_ON_YARN-2.3.0.cloudera4.jar
# cd /opt/cloudera/parcel-repo/
wget http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha
mv SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha1 SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel.sha
wget http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel
wget http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/manifest.json
chown cloudera-scm:cloudera-scm SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel*
chown cloudera-scm:cloudera-scm manifest.json
