3.6 Zookeeper安装及环境配置

3.6.1 Zookeeper安装

安装文件解压到/opt/zookeeper路径下。

[root@master ~]# tar  -xvf   zookeeper-3.4.10.tar.gz
[root@master ~]# mv  zookeeper-3.4.10  /opt/zookeeper
[root@master ~]# mv  zookeeper-3.4.10  zookeeper3.4

在zookeeper路径下创建数据文件和日志文件。

[root@master ~]# mkdir   /opt/zookeeper/data  
[root@master ~]# mkdir   /opt/zookeeper/log
# 在/opt/zookeeper/data目录下创建myid文件
[root@master ~]# touch myid
[root@master ~]# vi myid
# 设定值为1
1

修改zoo.cfg文件。

[root@master ~]# cd /opt/zookeeper/zookeeper3.4/conf
# 创建zoo.cfg文件
[root@master conf]# cp zoo_sample.cfg zoo.cfg
[root@master conf]# vi zoo.cfg

具体修改如下所示：

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
#dataDir=/tmp/zookeeper
# the port at which the clients will connect
# 默认端口是2181，但本机端口被占用，遂变更为2381
clientPort=2381
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
quorumListenOnAllIPs=ture
# 新增内容
dataDir=/opt/zookeeper/data
dataLogDir=/opt/zookeeper/log
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888

将Zookeeper传递给slave1和slave2，/opt/zookeeper/data路径下的myid与zoo.cfg内容须保持一致。

[root@master conf]# scp -r /opt/zookeeper root@slave1:/opt
[root@master conf]# scp -r /opt/zookeeper root@slave2:/opt

3.6.2 Zookeeper环境配置

编辑环境变量。

[root@master ~]# vi /etc/profile

修改内容如下：

export  ZK_HOME=/opt/zookeeper/zookeeper3.4
PATH=$PATH:${JAVA_HOME}/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:${ZK_HOME}/bin:$PATH

使修改生效。

[root@master ~]# source /etc/profile

3.6.3 Zookeeper启动

完成Zookeeper的安装配置后，在每一台机器上启动。注：Zookeeper遵循选举制
slave1

[root@slave1 ~]# cd /opt/zookeeper/zookeeper3.4/bin
[root@slave1 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/zookeeper3.4/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@slave1 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/zookeeper3.4/bin/../conf/zoo.cfg
Mode: leader

slave2

[root@slave2 ~]#  cd /opt/zookeeper/zookeeper3.4/bin
[root@slave2 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/zookeeper3.4/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@slave2 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/zookeeper3.4/bin/../conf/zoo.cfg
Mode: follower

master

[root@master ~]# cd /opt/zookeeper/zookeeper3.4/bin
[root@master bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/zookeeper3.4/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 3944.
[root@master bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/zookeeper3.4/bin/../conf/zoo.cfg
Mode: follower

3.7 HBase安装及环境配置

3.7.1 HBase安装

将HBase安装包加载至/opt/HBase路径下，并将其解压到当前路径。

[root@master ~]# mkdir /opt/HBase
[root@master ~]# cd /opt/HBase
[root@master HBase]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring hbase-2.0.2-bin.tar.gz...
  100%  150220 KB    10730 KB/sec    00:00:14       0 Errors 
[root@master ~]# tar  -xvf   hbase-2.0.2-bin.tar.gz  

3.7.2 配置文件修改

切换到/opt/hbase/hbase-2.0.2/conf路径。

在这里插入图片描述

修改hbase-env.sh，在文件中添加以下配置。

export JAVA_HOME=/opt/java/jdk1.8.0_181
export HADOOP_HOME=/opt/hadoop/hadoop-2.7.2
export HBASE_HOME=/opt/HBase/hbase-2.0.2
export HBASE_CLASSPATH=/opt/hadoop/hadoop-2.7.2/etc/hadoop
export HBASE_PID_DIR=/root/hbase/pids
export HBASE_MANAGES_ZK=false

修改hbase-site.xml，在文件中添加以下配置。

<configuration>
<property>
 <name>hbase.rootdir</name>
 <value>hdfs://master:9000/hbase</value>
 <description>The directory shared byregion servers.</description>
</property>
 <!-- hbase端口 -->
<property>
 <name>hbase.zookeeper.property.clientPort</name>
 <value>2181</value>
</property>
<!-- 超时时间 -->
<property>
 <name>zookeeper.session.timeout</name>
 <value>120000</value>
</property>
<!--防止服务器时间不同步出错 -->
<property>
<name>hbase.master.maxclockskew</name>
<value>150000</value>
</property>
<!-- 集群主机配置 -->
<property>
 <name>hbase.zookeeper.quorum</name>
 <value>master,slave3,slave4</value>
</property>
<!--   路径存放 -->
<property>
 <name>hbase.tmp.dir</name>
 <value>/root/hbase/tmp</value>
</property>
<!-- true表示分布式 -->
<property>
 <name>hbase.cluster.distributed</name>
 <value>true</value>
</property>
  <!-- 指定master -->
  <property>
    <name>hbase.master</name>
    <value>master:60000</value>
  </property>
</configuration>

修改regionservers，指定hbase的主从。

master
slave1
slave2

将这些配置传输到其他机器上。

scp -r /opt/HBase root@slave1:/opt
scp -r /opt/HBase root@slave2:/opt

配置环境变量。

[root@master ~]# vi /etc/profile

配置文件内容如下所示：

export HBASE_HOME=/opt/HBase/hbase-2.0.2
export PATH=$PATH:${JAVA_HOME}/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:${ZK_HOME}/bin:${HBASE_HOME}/bin:$PATH

使配置文件生效。

[root@master ~]# source  /etc/profile

查看安装结果。

[root@master ~]# hbase version
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/HBase/hbase-2.0.2/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase 2.0.2
Source code repository git://ve0524/home/stack/hbase.git revision=1cfab033e779df840d5612a85277f42a6a4e8172
Compiled by stack on Tue Aug 28 20:50:40 PDT 2018
From source with checksum 80b9ac6ea66f2b5cf6bc3ce886f3fc67

3.7.3 启动HBase

Hbase是基于hadoop提供的分布式文件系统的，所以启动Hbase之前，先确保hadoop在正常运行，另外Hbase还依赖于zookkeeper，本来我们可以用hbase自带的zookeeper，但是我们上面的配置启用的是我们自己的zookeeper集群，所以在启动hbase前，还要确保zokeeper已经正常运行。Hbase可以只在hadoop的某个namenode节点上安装，也可以在所有的hadoop节点上安装，但是启动的时候只需要在一个节点上启动就行了，本例中，我在master、slave1、slave2都安装了Hbase，启动的时候只需要在master上启动就OK。

[root@master ~]# cd /opt/HBase/hbase-2.0.2/bin 
[root@master bin]# start-hbase.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/HBase/hbase-2.0.2/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
running master, logging to /opt/HBase/hbase-2.0.2/logs/hbase-root-master-master.out
slave4: running regionserver, logging to /opt/HBase/hbase-2.0.2/logs/hbase-root-regionserver-slave4.out
slave3: running regionserver, logging to /opt/HBase/hbase-2.0.2/logs/hbase-root-regionserver-slave3.out

bug
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/HBase/hbase-2.0.2/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
solution
发生jar包冲突了,移除其中一个即可。
[root@master bin]# cd /opt/HBase/hbase-2.0.2/lib
[root@master lib]# rm slf4j-log4j12-1.7.25.jar
rm：是否删除普通文件 “slf4j-log4j12-1.7.25.jar”？yes
[root@master lib]# cd /opt/HBase/hbase-2.0.2/bin
[root@master bin]# start-hbase.sh
running master, logging to /opt/HBase/hbase-2.0.2/logs/hbase-root-master-master.out
slave3: running regionserver, logging to /opt/HBase/hbase-2.0.2/logs/hbase-root-regionserver-slave3.out
slave4: running regionserver, logging to /opt/HBase/hbase-2.0.2/logs/hbase-root-regionserver-slave4.out
[root@master bin]# jps
9027 HMaster
9141 Jps
3544 ResourceManager
3944 QuorumPeerMain
3849 Master
3387 SecondaryNameNode
3196 NameNode

状态信息查看，访问IP：16030或者16010。

在这里插入图片描述

此外也可以通过命令行查看状态。

# 切换路径
[root@master bin]# cd /opt/HBase/hbase-2.0.2/bin
# 进入hbase命令窗口
[root@master bin]# ./hbase  shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 2.0.2, r1cfab033e779df840d5612a85277f42a6a4e8172, Tue Aug 28 20:50:40 PDT 2018
Took 0.0268 seconds
# 状态查看                                                                                                                                                                                                                      
hbase(main):001:0> status
1 active master, 0 backup masters, 3 servers, 0 dead, 0.6667 average load
Took 1.2047 seconds  
# 退出                                                                                                                                                                                                                    
hbase(main):002:0> exit

bug

打开webUI界面后，提示The RegionServer is initializing!信息。

solution

查看hdfs safe mode

[root@master bin]# hadoop dfsadmin -safemode get

退出hdfs safe mode

[root@master bin]# hadoop dfsadmin -safemode leave

3.8 Hive安装及环境配置

Hive可以选择只安装在master节点上，而不必每一台都部署。

3.8.1 Mysql安装

由于mysql是Hive默认的元数据，所以需先行安装Mysql。

# 检查是否已安装mysql
[root@master ~]# rpm -qa | grep -i mysql
mysql-community-libs-5.6.42-2.el7.x86_64
mysql-community-server-5.6.42-2.el7.x86_64
mysql-community-release-el7-5.noarch
mysql-community-client-5.6.42-2.el7.x86_64
mysql-community-common-5.6.42-2.el7.x86_64
# 如果需要卸载
[root@master ~]# rpm -e --nodeps mysql-community-libs-5.6.42-2.el7.x86_64
[root@master ~]# rpm -e --nodeps mysql-community-server-5.6.42-2.el7.x86_64
[root@master ~]# rpm -e --nodeps mysql-community-release-el7-5.noarch
[root@master ~]# rpm -e --nodeps mysql-community-client-5.6.42-2.el7.x86_64
[root@master ~]# rpm -e --nodeps mysql-community-common-5.6.42-2.el7.x86_64
# 清理文件夹，必须通过该方式才能彻底卸载
[root@master ~]# find / -name mysql
/etc/selinux/targeted/active/modules/100/mysql
/etc/selinux/targeted/tmp/modules/100/mysql
/var/lib/mysql
/var/lib/mysql/mysql
/usr/lib64/mysql
/usr/share/mysql
/opt/hive/apache-hive-2.3.4-bin/scripts/metastore/upgrade/mysql
[root@master ~]# rm -rf /usr/share/mysql
[root@master ~]# rm -rf /var/lib/mysql
[root@master ~]# rm -rf /var/lib/mysql/mysql
# 安装mysql
[root@master ~]# rpm -qa | grep mysql
[root@master ~]# wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
--2018-12-06 16:46:05--  http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
正在解析主机 repo.mysql.com (repo.mysql.com)... 23.220.145.218
正在连接 repo.mysql.com (repo.mysql.com)|23.220.145.218|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：6140 (6.0K) [application/x-redhat-package-manager]
正在保存至: “mysql-community-release-el7-5.noarch.rpm”

100%[===============================================================================================================================================================================================>] 6,140       --.-K/s 用时 0.003s  

2018-12-06 16:46:06 (1.81 MB/s) - 已保存 “mysql-community-release-el7-5.noarch.rpm” [6140/6140])

[root@master ~]# sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
准备中...                          ################################# [100%]
正在升级/安装...
   1:mysql-community-release-el7-5    ################################# [100%]
[root@master ~]# sudo yum install mysql-server
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.163.com
 * updates: mirrors.aliyun.com
base                                                                                                                                                                                                              | 3.6 kB  00:00:00     
extras                                                                                                                                                                                                            | 3.4 kB  00:00:00     
mysql-connectors-community                                                                                                                                                                                        | 2.5 kB  00:00:00     
mysql-tools-community                                                                                                                                                                                             | 2.5 kB  00:00:00     
mysql56-community                                                                                                                                                                                                 | 2.5 kB  00:00:00     
updates                                                                                                                                                                                                           | 3.4 kB  00:00:00     
(1/3): mysql-connectors-community/x86_64/primary_db                                                                                                                                                               |  29 kB  00:00:00     
(2/3): mysql-tools-community/x86_64/primary_db                                                                                                                                                                    |  48 kB  00:00:00     
(3/3): mysql56-community/x86_64/primary_db                                                                                                                                                                        | 209 kB  00:00:01     
正在解决依赖关系
--> 正在检查事务
---> 软件包 mysql-community-server.x86_64.0.5.6.42-2.el7 将被 安装
--> 正在处理依赖关系 mysql-community-common(x86-64) = 5.6.42-2.el7，它被软件包 mysql-community-server-5.6.42-2.el7.x86_64 需要
--> 正在处理依赖关系 mysql-community-client(x86-64) >= 5.6.10，它被软件包 mysql-community-server-5.6.42-2.el7.x86_64 需要
--> 正在处理依赖关系 perl(DBI)，它被软件包 mysql-community-server-5.6.42-2.el7.x86_64 需要
--> 正在检查事务
---> 软件包 mysql-community-client.x86_64.0.5.6.42-2.el7 将被 安装
--> 正在处理依赖关系 mysql-community-libs(x86-64) >= 5.6.10，它被软件包 mysql-community-client-5.6.42-2.el7.x86_64 需要
---> 软件包 mysql-community-common.x86_64.0.5.6.42-2.el7 将被 安装
---> 软件包 perl-DBI.x86_64.0.1.627-4.el7 将被 安装
--> 正在处理依赖关系 perl(RPC::PlServer) >= 0.2001，它被软件包 perl-DBI-1.627-4.el7.x86_64 需要
--> 正在处理依赖关系 perl(RPC::PlClient) >= 0.2000，它被软件包 perl-DBI-1.627-4.el7.x86_64 需要
--> 正在检查事务
---> 软件包 mariadb-libs.x86_64.1.5.5.56-2.el7 将被 取代
---> 软件包 mysql-community-libs.x86_64.0.5.6.42-2.el7 将被 舍弃
---> 软件包 perl-PlRPC.noarch.0.0.2020-14.el7 将被 安装
--> 正在处理依赖关系 perl(Net::Daemon) >= 0.13，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要
--> 正在处理依赖关系 perl(Net::Daemon::Test)，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要
--> 正在处理依赖关系 perl(Net::Daemon::Log)，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要
--> 正在处理依赖关系 perl(Compress::Zlib)，它被软件包 perl-PlRPC-0.2020-14.el7.noarch 需要
--> 正在检查事务
---> 软件包 perl-IO-Compress.noarch.0.2.061-2.el7 将被 安装
--> 正在处理依赖关系 perl(Compress::Raw::Zlib) >= 2.061，它被软件包 perl-IO-Compress-2.061-2.el7.noarch 需要
--> 正在处理依赖关系 perl(Compress::Raw::Bzip2) >= 2.061，它被软件包 perl-IO-Compress-2.061-2.el7.noarch 需要
---> 软件包 perl-Net-Daemon.noarch.0.0.48-5.el7 将被 安装
--> 正在检查事务
---> 软件包 perl-Compress-Raw-Bzip2.x86_64.0.2.061-3.el7 将被 安装
---> 软件包 perl-Compress-Raw-Zlib.x86_64.1.2.061-4.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

=========================================================================================================================================================================================================================================
 Package                                                         架构                                           版本                                                     源                                                         大小
=========================================================================================================================================================================================================================================
正在安装:
 mysql-community-libs                                            x86_64                                         5.6.42-2.el7                                             mysql56-community                                         2.0 M
      替换  mariadb-libs.x86_64 1:5.5.56-2.el7
 mysql-community-server                                          x86_64                                         5.6.42-2.el7                                             mysql56-community                                          59 M
为依赖而安装:
 mysql-community-client                                          x86_64                                         5.6.42-2.el7                                             mysql56-community                                          20 M
 mysql-community-common                                          x86_64                                         5.6.42-2.el7                                             mysql56-community                                         257 k
 perl-Compress-Raw-Bzip2                                         x86_64                                         2.061-3.el7                                              base                                                       32 k
 perl-Compress-Raw-Zlib                                          x86_64                                         1:2.061-4.el7                                            base                                                       57 k
 perl-DBI                                                        x86_64                                         1.627-4.el7                                              base                                                      802 k
 perl-IO-Compress                                                noarch                                         2.061-2.el7                                              base                                                      260 k
 perl-Net-Daemon                                                 noarch                                         0.48-5.el7                                               base                                                       51 k
 perl-PlRPC                                                      noarch                                         0.2020-14.el7                                            base                                                       36 k

事务概要
=========================================================================================================================================================================================================================================
安装  2 软件包 (+8 依赖软件包)

总下载量：82 M
Is this ok [y/d/N]: y
Downloading packages:
警告：/var/cache/yum/x86_64/7/mysql56-community/packages/mysql-community-common-5.6.42-2.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY                                              ] 231 kB/s | 259 kB  00:06:03 ETA 
mysql-community-common-5.6.42-2.el7.x86_64.rpm 的公钥尚未安装
(1/10): mysql-community-common-5.6.42-2.el7.x86_64.rpm                                                                                                                                                            | 257 kB  00:00:01     
(2/10): mysql-community-client-5.6.42-2.el7.x86_64.rpm                                                                                                                                                            |  20 MB  00:00:07     
警告：/var/cache/yum/x86_64/7/base/packages/perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID f4a80eb5: NOKEY                                                         ] 2.6 MB/s |  20 MB  00:00:23 ETA 
perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64.rpm 的公钥尚未安装
(3/10): perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64.rpm                                                                                                                                                            |  32 kB  00:00:00     
(4/10): perl-Compress-Raw-Zlib-2.061-4.el7.x86_64.rpm                                                                                                                                                             |  57 kB  00:00:00     
(5/10): perl-IO-Compress-2.061-2.el7.noarch.rpm                                                                                                                                                                   | 260 kB  00:00:00     
(6/10): perl-Net-Daemon-0.48-5.el7.noarch.rpm                                                                                                                                                                     |  51 kB  00:00:00     
(7/10): perl-PlRPC-0.2020-14.el7.noarch.rpm                                                                                                                                                                       |  36 kB  00:00:00     
(8/10): perl-DBI-1.627-4.el7.x86_64.rpm                                                                                                                                                                           | 802 kB  00:00:00     
(9/10): mysql-community-libs-5.6.42-2.el7.x86_64.rpm                                                                                                                                                              | 2.0 MB  00:00:24     
(10/10): mysql-community-server-5.6.42-2.el7.x86_64.rpm                                                                                                                                                           |  59 MB  00:02:21     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
总计                                                                                                                                                                                                     567 kB/s |  82 MB  00:02:28     
从 file:/etc/pki/rpm-gpg/RPM-GPG-KEY-mysql 检索密钥
导入 GPG key 0x5072E1F5:
 用户ID     : "MySQL Release Engineering <mysql-build@oss.oracle.com>"
 指纹       : a4a9 4068 76fc bd3c 4567 70c8 8c71 8d3b 5072 e1f5
 软件包     : mysql-community-release-el7-5.noarch (installed)
 来自       : file:/etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
是否继续？[y/N]：y
从 file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 检索密钥
导入 GPG key 0xF4A80EB5:
 用户ID     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 指纹       : 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 软件包     : centos-release-7-5.1804.el7.centos.x86_64 (@anaconda)
 来自       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
是否继续？[y/N]：y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
警告：RPM 数据库已被非 yum 程序修改。
** 发现 2 个已存在的 RPM 数据库问题， 'yum check' 输出如下：
icedtea-web-1.7.1-1.el7.x86_64 有缺少的需求 java-1.8.0-openjdk
jline-1.0-8.el7.noarch 有缺少的需求 java >= ('0', '1.5', None)
  正在安装    : mysql-community-common-5.6.42-2.el7.x86_64                                                                                                                                                                          1/11 
  正在安装    : mysql-community-libs-5.6.42-2.el7.x86_64                                                                                                                                                                            2/11 
  正在安装    : mysql-community-client-5.6.42-2.el7.x86_64                                                                                                                                                                          3/11 
  正在安装    : perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64                                                                                                                                                                          4/11 
  正在安装    : 1:perl-Compress-Raw-Zlib-2.061-4.el7.x86_64                                                                                                                                                                         5/11 
  正在安装    : perl-IO-Compress-2.061-2.el7.noarch                                                                                                                                                                                 6/11 
  正在安装    : perl-Net-Daemon-0.48-5.el7.noarch                                                                                                                                                                                   7/11 
  正在安装    : perl-PlRPC-0.2020-14.el7.noarch                                                                                                                                                                                     8/11 
  正在安装    : perl-DBI-1.627-4.el7.x86_64                                                                                                                                                                                         9/11 
  正在安装    : mysql-community-server-5.6.42-2.el7.x86_64                                                                                                                                                                         10/11 
  正在删除    : 1:mariadb-libs-5.5.56-2.el7.x86_64                                                                                                                                                                                 11/11 
warning: file /usr/lib64/mysql/plugin/mysql_clear_password.so: remove failed: No such file or directory
warning: file /usr/lib64/mysql/plugin/dialog.so: remove failed: No such file or directory
warning: file /usr/lib64/mysql/libmysqlclient.so.18.0.0: remove failed: No such file or directory
  验证中      : mysql-community-libs-5.6.42-2.el7.x86_64                                                                                                                                                                            1/11 
  验证中      : mysql-community-common-5.6.42-2.el7.x86_64                                                                                                                                                                          2/11 
  验证中      : perl-Net-Daemon-0.48-5.el7.noarch                                                                                                                                                                                   3/11 
  验证中      : mysql-community-server-5.6.42-2.el7.x86_64                                                                                                                                                                          4/11 
  验证中      : mysql-community-client-5.6.42-2.el7.x86_64                                                                                                                                                                          5/11 
  验证中      : perl-IO-Compress-2.061-2.el7.noarch                                                                                                                                                                                 6/11 
  验证中      : 1:perl-Compress-Raw-Zlib-2.061-4.el7.x86_64                                                                                                                                                                         7/11 
  验证中      : perl-DBI-1.627-4.el7.x86_64                                                                                                                                                                                         8/11 
  验证中      : perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64                                                                                                                                                                          9/11 
  验证中      : perl-PlRPC-0.2020-14.el7.noarch                                                                                                                                                                                    10/11 
  验证中      : 1:mariadb-libs-5.5.56-2.el7.x86_64                                                                                                                                                                                 11/11 

已安装:
  mysql-community-libs.x86_64 0:5.6.42-2.el7                                                                         mysql-community-server.x86_64 0:5.6.42-2.el7                                                                        

作为依赖被安装:
  mysql-community-client.x86_64 0:5.6.42-2.el7     mysql-community-common.x86_64 0:5.6.42-2.el7     perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7     perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7     perl-DBI.x86_64 0:1.627-4.el7    
  perl-IO-Compress.noarch 0:2.061-2.el7            perl-Net-Daemon.noarch 0:0.48-5.el7              perl-PlRPC.noarch 0:0.2020-14.el7               

替代:
  mariadb-libs.x86_64 1:5.5.56-2.el7                                                                                                                                                                                                     

完毕！

启动并查看mysql服务状态。

# 启动服务
[root@master ~]# service mysqld start
Redirecting to /bin/systemctl start mysqld.service
# 查看mysql连接状态
[root@master ~]# service mysqld status
Redirecting to /bin/systemctl status mysqld.service
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2018-11-16 20:01:52 CST; 44s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 20743 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 20766 (mysqld)
   Status: "SERVER_OPERATING"
    Tasks: 38
   CGroup: /system.slice/mysqld.service
           └─20766 /usr/sbin/mysqld
11月 16 20:01:45 master systemd[1]: Starting MySQL Server...
11月 16 20:01:52 master systemd[1]: Started MySQL Server.
# 设置免密登陆
[root@master ~]# mysqladmin -u root -p password ‘123456’
# 输入密码时候直接选择回车
Enter password: 
Warning: Using a password on the command line interface can be insecure.
# 进入mysql
[root@master ~]# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.6.42 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> grant all privileges on . to ‘root’@’%’ identified by ‘123456’   # 授权法更改远程连接权限
    -> flush privileges  # 刷新
    -> exit  #退出

外部验证连接。

在这里插入图片描述

当图标变绿时，则表明连接完成。

在这里插入图片描述

3.8.2 Hive安装

将HBase安装包加载至/opt/hive路径下，并将其解压到当前路径。

[root@master ~]# mkdir /opt/hive
[root@master ~]# cd /opt/hive
[root@master HBase]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring apache-hive-2.3.4-bin...
  100%  272180 KB    10820 KB/sec    00:00:25       0 Errors 
[root@master ~]# tar  -xvf   apache-hive-2.3.4-bin  

3.8.2 Hive环境配置

设置环境变量。

[root@master ~]# vi /etc/profile
export HIVE_HOME=/opt/hive/apache-hive-2.3.4-bin
export HIVE_CONF_DIR=${HIVE_HOME}/conf
export PATH=$PATH:${JAVA_HOME}/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:${HIVE_HOME}/bin:${ZK_HOME}/bin:${HBASE_HOME}/bin:$PATH
[root@master ~]# source /etc/profile

建立必需的文件夹。

# 建立root目录下需要的文件夹
[root@master ~]# mkdir /root/hive
[root@master ~]# mkdir /root/hive/warehouse
# hadoop下新建文件目录
[root@master ~]# $HADOOP_HOME/bin/hadoop fs -mkdir -p /root/hive/
[root@master ~]# $HADOOP_HOME/bin/hadoop fs -mkdir -p /root/hive/warehouse
# 为新建文件目录授权读写能力
[root@master ~]# $HADOOP_HOME/bin/hadoop fs -chmod 777 /root/hive/
[root@master ~]# $HADOOP_HOME/bin/hadoop fs -chmod 777 /root/hive/warehouse
# 检验创建状态
[root@master ~]# $HADOOP_HOME/bin/hadoop fs -ls /root/
[root@master ~]# $HADOOP_HOME/bin/hadoop fs -ls /root/hive/

修改hive-site.xml。

# 创建hive-site.xml
[root@master ~]# cd /opt/hive/apache-hive-2.3.4-bin/conf
[root@master conf]# cp hive-default.xml.template hive-site.xml
[root@master conf]# vi  hive-site.xml

添加内容如下：

<!-- 指定HDFS中的hive仓库地址 -->  
  <property>  
    <name>hive.metastore.warehouse.dir</name>  
    <value>/root/hive/warehouse</value>  
  </property>  
<property>
    <name>hive.exec.scratchdir</name>
    <value>/root/hive</value>
  </property>
  <!-- 该属性为空表示嵌入模式或本地模式，否则为远程模式 -->  
  <property>  
    <name>hive.metastore.uris</name>  
    <value></value>  
  </property>  
<!-- 指定mysql的连接 -->
 <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true</value>
    </property>
<!-- 指定驱动类 -->
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
   <!-- 指定用户名 -->
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <!-- 指定密码 -->
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
    <property>
   <name>hive.metastore.schema.verification</name>
   <value>false</value>
    <description>
    </description>
 </property>

将配置文件中所有的${system:java.io.tmpdir}更改为 /opt/hive/tmp (如果没有该文件则创建)，并将此文件夹赋予读写权限，将${system:user.name}更改为 root。

   <!-- 代码较长，需耐心查找，也可借助相关文本替代命令进行处理，仅展示部分 -->
  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/opt/hive/tmp/root</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/opt/hive/tmp/${hive.session.id}_resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>

修改 hive-env.sh。

[root@master ~]# cd /opt/hive/apache-hive-2.3.4-bin/conf
[root@master conf]# cp hive-env.sh.template hive-env.sh
[root@master conf]# vi hive-env.sh
# 添加以下内容
export  HADOOP_HOME=/opt/hadoop/hadoop-2.7.2
export  HIVE_CONF_DIR=/opt/hive/apache-hive-2.3.4-bin/conf
export  HIVE_AUX_JARS_PATH=/opt/hive/apache-hive-2.3.4-bin/lib

添加数据驱动包

首先从mysql官网下载对应的JDBC安装包。

在这里插入图片描述

在这里插入图片描述

[root@master ~]# cd /opt/hive/apache-hive-2.3.4-bin/lib
# 下载mysql 的 jdbc连接驱动jar包，放到hive的lib目录下
[root@master lib]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring mysql-connector-java-5.1.47-bin.jar...
  100%     983 KB     983 KB/sec    00:00:01       0 Errors 

3.8.2 Hive启动

# 切换到hive文件目录
[root@master lib]# cd /opt/hive/apache-hive-2.3.4-bin/bin
[root@master bin]# pwd
/opt/hive/apache-hive-2.3.4-bin/bin
# 初始化hive
[root@master bin]# schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       root
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: java.sql.SQLException : Access denied for user 'root'@'master' (using password: YES)
SQL Error code: 1045
Use --verbose for detailed stacktrace.
*** schemaTool failed ***

bug01
发生jar包冲突。
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-
solution
[root@master bin]# rm /opt/hive/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar
rm：是否删除普通文件 “/opt/hive/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar”？y
bug02
用户权限不足。
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: java.sql.SQLException : Access denied for user ‘root’@‘master’ (using password: YES)
SQL Error code: 1045
solution
[root@master bin]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.6.42 MySQL Community Server (GPL)
Copyright © 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select host,user from user;
±----------±-----+
| host | user |
±----------±-----+
| % | root |
| 127.0.0.1 | root |
| ::1 | root |
| localhost | |
| master | |
| master | root |
±----------±-----+
6 rows in set (0.00 sec)
mysql> show grants for root;
ERROR 1290 (HY000): The MySQL server is running with the --skip-grant-tables option so it cannot execute this statement
mysql> flush privileges;
Query OK, 0 rows affected (0.06 sec)
mysql> show grants for root;
±-------------------------------------------------------------------------------------------------------------------------------+
| Grants for root@% |
±-------------------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON . TO ‘root’@’%’ IDENTIFIED BY PASSWORD ‘*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9’ WITH GRANT OPTION |
±-------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
Query OK, 0 rows affected (0.01 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)