Ambari部署

Ambari简介：

Apache Ambari是一种基于Web的工具，支持Apache Hadoop集群的供应、管理和监控。Ambari目前已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeper、Sqoop和Hcatalog等。
Apache Ambari 支持HDFS、MapReduce、Hive、Pig、Hbase、Zookeper、Sqoop和Hcatalog等的集中管理。也是5个顶级hadoop管理工具之一。

Ambari界面预览

部署环境：CentOS 6/7

部署步骤：

一、环境准备

1.关闭防火墙

CentOS 6 :

chkconfig iptables off
service iptables stop
chkconfig --list iptables

CentOS 7：

systemctl disable firewalld.service
systemctl stop firewalld.service

2.关闭selinux

setenforce 0   (临时关闭)
vi /etc/selinux/config
SELINUX=disabled    (永久关闭，重启后生效)

3.修改主机名

CentOS 6:

vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=namenode-1

Centos 7:

hostnamectl set-hostname namenode-1

4.配置本地域名解析（以下为示例）

192.168.5.1  namenode-1
192.168.5.2  datanode-1
192.168.5.3  web1
192.168.5.4  web2
192.168.5.5  datanode-2
192.168.5.6  datanode-3
192.168.5.7  datanode-4
192.168.5.8  ambari

5.配置SSH（Ambari节点可无密码访问其他主机）

ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

ssh-copy-id -i /root/.ssh/id_rsa.pub root@namenode-1
... ...

6.修改YUM源

这里使用阿里云镜像已保证安装速度

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bk
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo
yum makecache

7.安装JDK

HADOOP环境需要的JAVA为1.7以上版本的JDK，这里使用1.8版本的JAVA
上传1.8版本的JDK包
rpm -ivh jdk-8u131-linux-x64.rpm
配置环境变量
vim /etc/profile  （文件末尾添加）
export JAVA_HOME=/usr/java/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
验证是否安装成功
java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

8.安装时间同步服务（NTP），确保集群内部时间统一

CentOS 6:

yum install -y ntp
chkconfig --list ntpd
chkconfig ntpd on
service ntpd start

CentOS 7:

yum install -y ntp
chkconfig --list ntpd
systemctl is-enabled ntpd
systemctl enabled ntpd
systemctl start ntpd

9.关闭THP服务（如果不关闭transparent_hugepage，Ambari在检查环境是也会进行警告，HDFS性能会因为THP服务未关闭受到严重的影响）

永久关闭（重启后生效）：

CentOS 6:

vi /etc/grub.conf
添加 transparent_hugepage=never
vi /etc/rc.local
添加：
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
  echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
exit 0

CentOS 7:

vi /etc/default/grub
在有CMDLINE一行添加 transparent_hugepage=never
grub2-mkconfig -o /boot/grub2/grub.cfg
systemctl disable tuned

临时关闭（不需要重启）：

CentOS 6:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled 
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled always madvise [never]

CentOS 7:

echo never > /sys/kernel/mm/transparent_hugepage/enabled 
echo never > /sys/kernel/mm/transparent_hugepage/defrag

检查方法：

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
有 [never]则表示THP被禁用

10.配置UMASK，设定用户创建目录的初始权限

umask 0022

11.禁止离线更新

vi /etc/yum/pluginconf.d/refresh-packagekit
添加enbaled=0

12.备份克隆（略）

以上的环境配置是最基本的，每个节点都需要的配置。可以做一备份节点，用于将来创建子节点克隆使用，减少重复工作。当然克隆后还有小的调整，比如要重新配置hostname，还有SSH配置过程里，分发主节点里配置好的authorized_keys到各从节点，还是在克隆所有子节点后再操作。

二、安装Ambari集群

1.制作本地源

使用官方默认源的话十分缓慢，因此建议使用本地源进行安装（本地源文件较大、下载较为缓慢，酌情考虑）

1).配置HTTP服务（yum实质上是使用http协议进行下载安装的）

chkconfig httpd on
service httpd start  /  systemctl start httpd

2).安装本地源制作相关工具

yum -y install yum-utils createrepo yum-plugin-priorities
vi /etc/yum/pluginconf.d/priorities.conf
添加gpgcheck=0

3).下载Ambari和HDP（下载时间1M带宽预计需要14小时左右，网络不稳定时容易断线）

CentOS 6：

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos6.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos6-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6/HDP-UTILS-1.1.0.21-centos6.tar.gz

CentOS 7:

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

注：以下将按照CentOS 6进行部署演示，CentOS 7请依据实际情况进行改动

4).创建本地源（我在这里将源文件解压到了/var/www/html目录中）

tar zxvf /opt/ambari-2.5.0.3-centos6.tar.gz -C /var/www/html
tar zxvf /opt/HDP-2.6.0.3-centos6-rpm.tar.gz -C /var/www/html
tar zxvf /opt/HDP-UTILS-1.1.0.21-centos6.tar.gz -C /var/www/html

cd /var/www/html/
createrepo  ./

wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/ambari.repo -O /etc/yum.repos.d/ambari.repo

修改ambari.repo,配置为本地源

vi /etc/yum.repos.d/ambari.repo
[ambari-2.5.0.3]
name=ambari Version - ambari-2.5.0.3
baseurl=http://192.168.5.8/ambari/centos6
gpgcheck=1
gpgkey=http://192.168.5.8/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1

priority=1

vi /etc/yum.repos.d/HDP.repo
[HDP-2.6]
name=HDP-2.6
baseurl=http://192.168.5.8/HDP/centos6/

path=/
enabled=1
gpgcheck=0

vi /etc/yum.repos.d/HDP-UTILS.repo
[HDP-UTILS-1.1.0.21]
name=HDP-UTILS-1.1.0.21
baseurl=http://192.168.50.38/HDP-UTILS-1.1.0.20/

path=/
enabled=1
gpgcheck=0

验证是否生效（如果不确定是否正确可以尝试浏览器访问，例如 http://192.168.5.8/ambari/centos6

yum clean all
yum makecache
yum repolist

2.安装数据库

Ambari使用的默认数据库是PostgreSQL,用于存储安装元数据，可以使用自己安装的MySQL作为Ambari元数据库（我是使用了源码安装的方法，不做过多赘述，这里只做Yum安装步骤）

yum -y install mysql mysql-server
chkconfig mysqld on
service mysqld start

3.安装Ambari

yum install -y ambari-server

执行yum install ambari-server成功后,针对mysql数据库再做一些工作：

将mysql-connector-java.jar复制到/usr/share/java目录下（在下载的文件中有mysql-connector-java.jar，具体位置为hdp-utils/mysql-connector-java-source/mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar）

mkdir /usr/share/java
cp /root/hdp-utils/mysql-connector-java-source/mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar /usr/share/java/mysql-connector-java.jar
cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar

编辑Ambari.properties

vi /etc/ambari-server/conf/ambari.properties
添加server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

在mysql中分别创建数据库ambari,hive,oozie和其相应用户，创建相应的表（如果不愿意将hive和oozie放在和ambari相同的数据库里，可以另外建立数据库）

CREATE DATABASE ambari;
use ambari;
grant all on *.* to ambari@'localhost' identified by 'password';
grant all on *.* to ambari@'hostname' identified by 'password';
grant all on *.* to ambari@'%' identified by 'password';

CREATE DATABASE hive;
use hive;
grant all on *.* to hive@'localhost' identified by 'password';
grant all on *.* to hive@'hostname' identified by 'password';
grant all on *.* to hive@'%' identified by 'password';

CREATE DATABASE oozie;
use oozie;
grant all on *.* to oozie@'localhost' identified by 'password';
grant all on *.* to oozie@'hostname' identified by 'password';
grant all on *.* to oozie@'%' identified by 'password';

4.配置Ambari

执行ambari-server setup

ambari-server setup
下面是配置执行流程，按照提示操作
（1） 提示是否自定义设置。输入：y
Customize user account for ambari-server daemon [y/n] (n)? y
（2）ambari-server 账号。
Enter user account for ambari-server daemon (root):
如果直接回车就是默认选择root用户
如果输入已经创建的用户就会显示：
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...
（3）检查防火墙是否关闭
Adjusting ambari-server permissions and ownership...
Checking firewall...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)?
直接回车
（4）设置JDK。输入：3
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
如果上面选择3自定义JDK,则需要设置JAVA_HOME。输入：/usr/java/jdk1.8.0_131
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_131
Validating JDK on Ambari Server...done.
Completing setup...
（5）数据库配置。选择：y
Configuring database...
Enter advanced database configuration [y/n] (n)? y
（6）选择数据库类型。输入：3
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
==============================================================================
Enter choice (3): 3
（7）设置数据库的具体配置信息，根据实际情况输入，如果和括号内相同，则可以直接回车。如果想重命名，就输入。
Hostname (localhost):hdp131.cancer.com
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Re-Enter password: 
（8）将Ambari数据库脚本导入到数据库
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql 
Proceed with configuring remote database connection properties [y/n] (y)?  
如果使用自己定义的数据库，必须在启动Ambari服务之前导入Ambari的sql脚本。这一点在安装Ambari环节提到过。

5.启动Ambari

ambari-server start

成功后浏览器访问
http://AMBARI_IP:8080
默认用户名admin，密码admin