Ambari2.6.2 和HDP2.6.5搭建Hadoop集群-04_Ambari2.6.2 安装HDP2.6.5

本节我们开始正式构建HDP的软件环境

一、服务器环境配置

1.服务器列表
2.配置SSH免密登陆
3.更改hosts
4.配置防火墙
5.配置阿里yum源(可选)
6.NTP时钟同步
7.关闭SELinux、配置umask 值

通过前面的三章以上内容均已完成（即基本的软件安装环境已经具备）

二、软件环境

1.安装mysql数据库

在安装Ambari时，Ambari默认的数据库是ProstgreSQL，这里选择使用MySql。但Centos 7 默认支持的是MariaDB数据库. MariaDB是MySQL的一个分支，主要由开源社区在维护。安装过程中先将Centos7默认安装的MariaDB数据库删除，再重新安装MySql

安装步骤：

软件下载（mysql的rpm包有很多文件，只需要安装Server 和 Client）

由于操作系统是用CentOS7.5 根据官网查询匹配的Mysql 5.7版本为 5.7.26

https://dev.mysql.com/downloads/file/?id=485654

image.png

数据库安装
删除Centos默认安装的MariaDB

[root@node1 ambari]# rpm -qa|grep mariadb
mariadb-libs-5.5.56-2.el7.x86_64
[root@node1 ambari]# rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64

解压mysql-5.7.26-1.el7.x86_64.rpm-bundle.tar

[root@node1 mysql]# tar -xvf mysql-5.7.26-1.el7.x86_64.rpm-bundle.tar
mysql-community-embedded-devel-5.7.26-1.el7.x86_64.rpm
mysql-community-libs-5.7.26-1.el7.x86_64.rpm
mysql-community-embedded-5.7.26-1.el7.x86_64.rpm
mysql-community-test-5.7.26-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.26-1.el7.x86_64.rpm
mysql-community-common-5.7.26-1.el7.x86_64.rpm
mysql-community-devel-5.7.26-1.el7.x86_64.rpm
mysql-community-client-5.7.26-1.el7.x86_64.rpm
mysql-community-server-5.7.26-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm

安装Mysql【注意：执行有依赖顺序】

[root@node1 mysql]# rpm -ivh mysql-community-common-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-common-5.7.26-1.e################################# [100%]
[root@node1 mysql]# rpm -ivh mysql-community-libs-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-libs-5.7.26-1.el7################################# [100%]
[root@node1 mysql]# rpm -ivh mysql-community-client-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-client-5.7.26-1.e################################# [100%]
[root@node1 mysql]# rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
error: Failed dependencies:
    libaio.so.1()(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64
    libaio.so.1(LIBAIO_0.1)(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64
    libaio.so.1(LIBAIO_0.4)(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64

【此处报错：】这个报错的意思是需要安装libaio包 libaio-0.3.109-13.el7.x86_64.rpm

[root@node1 tmp]# yum -y install wget
[root@node1 tmp]# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libaio-0.3.109-13.el7.x86_64.rpm
[root@node1 tmp]# rpm -ivh libaio-0.3.109-13.el7.x86_64.rpm

再次执行安装server(执行成功)

[root@node1 mysql]# rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-server-5.7.26-1.e################################# [100%]

数据库初始化
为了保证数据库目录为与文件的所有者为 mysql 登陆用户，如果你是以 root 身份运行 mysql 服务，需要执行下面的命令初始化

mysqld --initialize --user=mysql

如果是以 mysql 身份运行，则可以去掉 --user 选项。
另外 --initialize 选项默认以“安全”模式来初始化，则会为 root 用户生成一个密码并将该密码标记为过期，登陆后你需要设置一个新的密码，而使用 --initialize-insecure 命令则不使用安全模式，则不会为 root 用户生成一个密码。
这里演示使用的 --initialize 初始化的，会生成一个 root 账户密码，密码在log文件里，红色区域的就是自动生成的密码

[root@node1 mysql]# cat /var/log/mysqld.log
2019-06-12T07:58:54.568405Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-06-12T07:58:55.838148Z 0 [Warning] InnoDB: New log files created, LSN=45790
2019-06-12T07:58:56.155262Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2019-06-12T07:58:56.317281Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: ed9a8884-8ce7-11e9-aafe-1c1b0df90270.
2019-06-12T07:58:56.356702Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2019-06-12T07:58:56.358045Z 1 [Note] A temporary password is generated for root@localhost: n?kifcur4sUe

启动mysql

[root@node1 mysql]# systemctl start mysqld.service

使用临时密码登陆系统【A temporary password is generated for root@localhost: n?kifcur4sUe】

[root@node1 mysql]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.26

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

修改root用户密码

mysql> show databases;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

以前的 password()函数将会被抛弃，官方建议使用下面的命令来修改密码（避免使用简单密码否则也会报错）

ALTER USER 'root'@'localhost' IDENTIFIED BY 'nihao2019@mysql'; 此处自行设置

安装目录及功能
mysql安装完成后，文件分布在以下几个目录:
/var/lib/mysql 数据库文件,目录下的 ibdata1 用于存放数据，索引等，是MYSQL的最主要的数据。ib_filelog是数据库日志文件，采用二进制存储。
/usr/share/mysql 命令及配置文件目录
/usr/bin mysqladmin、mysqldump 等命令
在linux命令行下，可使用 whereis mysql 命令查看其安装目录

[root@node1 mysql]# whereis mysql
mysql: /usr/bin/mysql /usr/lib64/mysql /usr/share/mysql /usr/share/man/man1/mysql.1.gz

开机自启动

[root@node1 mysql]# vi /etc/rc.local
#添加service mysqld start

2.安装mysql-java驱动

命令：yum install -y mysql-connector-java （安装mysql驱动)
检查：cd /usr/share/java (安装完成后，切换至java目录，如下图，则ok)

image.png

cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar
vi /etc/ambari-server/conf/ambari.properties
添加server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
[非常重要]

三、修改yum源，实现离线安装

1、安装httpd服务（主服务器）

[root@node1 ~]# yum install -y httpd

[root@node1 ~]# systemctl start httpd

[root@node1 ~]# chkconfig httpd on
Note: Forwarding request to 'systemctl enable httpd.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

2、将上面下载的三个包放到/var/www/html目录下（主服务器）

[root@node1 tmp]# cd /var/www/html/
[root@node1 html]# mkdir ambari
拷贝文件到ambari下面
[root@node1 html]# cd ambari/
[root@node1 ambari]# ls
ambari-2.6.2.2-centos7.tar.gz  HDP-2.6.5.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@node1 ambari]# tar -zxvf ambari-2.6.2.2-centos7.tar.gz
[root@node1 ambari]# tar -zxvf HDP-2.6.5.0-centos7-rpm.tar.gz 
[root@node1 ambari]# mkdir HDP-UTILS
[root@node1 ambari]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@node1 ambari]# rm -rf ambari-2.6.2.2-centos7.tar.gz  HDP-2.6.5.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@node1 ambari]# ls
ambari  HDP  HDP-UTILS

现在可以通过访问http://192.168.12.191/ambari/查看是否能成功访问

image.png

3、制作本地源

（1）安装本地源制作相关工具（主服务器）

[root@node1 ambari]# yum install yum-utils createrepo yum-plugin-priorities -y
[root@node1 ambari]# createrepo  ./
Spawning worker 0 with 67 pkgs
Spawning worker 1 with 67 pkgs
Spawning worker 2 with 66 pkgs
Spawning worker 3 with 66 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

（2）修改文件里面的源地址（主服务器）

[root@node1 ambari]# cp ambari/centos7/2.6.2.2-1/ambari.repo ambari/centos7/2.6.2.2-1/ambari.repo.bak
[root@node1 ambari]# vi ambari/centos7/2.6.2.2-1/ambari.repo
#VERSION_NUMBER=2.6.2.2-1
[ambari-2.6.2.2]
name=ambari Version - ambari-2.6.2.2
baseurl=http://192.168.12.191/ambari/ambari/centos7/2.6.2.2-1
gpgcheck=1
gpgkey=http://192.168.12.191/ambari/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[root@node1 ambari]# cp ambari/centos7/2.6.2.2-1/ambari.repo /etc/yum.repos.d/


[root@node1 2.6.5.0-292]# cp hdp.repo hdp.repo.bak
[root@node1 2.6.5.0-292]# vi hdp.repo
#VERSION_NUMBER=2.6.5.0-292
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0
baseurl=http://192.168.12.191/ambari/HDP/centos7/2.6.5.0-292
gpgcheck=1
gpgkey=http://192.168.12.191/ambari/HDP/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.12.191/ambari/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://192.168.12.191/ambari/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[root@node1 2.6.5.0-292]# cp hdp.repo /etc/yum.repos.d/

上面就创建好了主机上的文件，然后可以通过下面命令清楚一下yum的缓存就可以了

[root@node1 ambari]# yum clean all
[root@node1 ambari]# yum makecache
[root@node1 ambari]# yum repolist

(3) 将文件分发到其他机器

[root@node1 yum.repos.d]# cd /etc/yum.repos.d
[root@node1 yum.repos.d]# scp ambari.repo root@node2:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   285.9KB/s   00:00
[root@node1 yum.repos.d]# scp ambari.repo root@node3:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   308.5KB/s   00:00
[root@node1 yum.repos.d]# scp ambari.repo root@node4:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   273.3KB/s   00:00
[root@node1 yum.repos.d]# scp ambari.repo root@node5:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   309.9KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node2:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   697.2KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node3:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   557.2KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node4:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   499.9KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node5:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   527.9KB/s   00:00
[root@node1 yum.repos.d]#

测试http://192.168.12.191/ambari/ambari/centos7/2.6.2.2-1/ 是否可以访问，如果可以则没有问题

四、安装ambari-server

ambari-server有两种数据库安装模式

4.1默认postgresql数据库的安装方式

这种不推荐生产环境使用
（1）、安装和配置ambari-server

[root@master yum.repos.d]# yum -y install ambari-server
[root@master yum.repos.d]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_161
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? n
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database ... OK


About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
ambari-admin-2.6.0.0.267.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

（2）启动ambari

[root@master ~]# ambari-server start

成功启动后在浏览器输入Ambari地址：
http://192.168.12.191:8080

4.2第三方数据库的安装方式

生产使用Mysql
（1）安装mysql
（2）登录mysql，执行下面的语句
[root@node1 share]# mysql -uroot -p

CREATE DATABASE ambari;  
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost' IDENTIFIED BY 'bigdata'; 
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%' IDENTIFIED BY 'bigdata';
FLUSH PRIVILEGES;  
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql; 
show tables;  
quit; 

CREATE DATABASE hive;  
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost' IDENTIFIED BY 'hive'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive';
FLUSH PRIVILEGES;

（3）mysql与ambari-server的连接
ambari-server 安装

[root@node1 java]# yum install ambari-server

ambari-server 初始化

ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_121
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Invalid characters in password. Use only alphanumeric or _ or - characters
Enter Database Password (bigdata):
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
....ambari-admin-2.6.2.2.1.jar
.......
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

（4）启动Ambari

[root@node1 java]# ambari-server start

登陆http://192.168.12.191:8080 admin/admin[初始]

image.png

五、安装配置部署HDP集群

1、登录
登录界面，默认管理员账户登录，账户：admin 密码：admin
2、安装向导

（1）配置集群的名字为hadoop
（2）选择版本并修改为本地源地址

image.png

3）安装配置
选择要安装的服务器和上传上面教程所拷贝出来的秘钥文件id_rsa

image.png

（4）安装ambari的agent，同时检查系统问题

image.png

如果这里出了问题，请检查上面所有的步骤有没有遗漏和未设置的参数。同时在重新修改了配置以后，最好是重置ambari-server来重新进行安装,确保每一步无警告再执行下一步。

[root@master ~]# # ambari-server stop    
[root@master ~]# # ambari-server reset   #重置命令
[root@master ~]# # ambari-server setup   #重新设置
[root@master ~]# # ambari-server start

（5）选择要安装的服务

image.png

（6）选择分配服务

image.png

最后安装完成

image.png

我的上面有一些警告和关闭的服务，不用担心。大体上就是这样的安装流程。
对于HDP的安装个人感觉比CDH使用CM的安装方式要繁琐很多。过程中出现错误需要重新安装时需要删除干净相关文件和依赖，在重新安装时Ambari会对已存在的包、文件、用户进行检测、可根据提示完善删除操作。