1. 借鉴
Hadoop集群添加新节点
为什么Hive里，要用mysql?
centos7下安装mysql5.7（rpm）
error: Failed dependencies: perl(Data::Dumper) is needed by MySQL-server-5.6.46-1.el7.x8
如何在DOCKER容器中运行MySQL服务
rpm 完全卸载mysql
Tez 0.9安装部署+hive on tez配置 + Tez-UI
2. 开始
我们的节点规划如下：
| hive01[172.173.16.22] |
|---|
镜像准备
hive是依托于hadoop的，所以hive机器上需要hadoop环境
- docker hub 下载
docker pull caiserkaiser/hadoop:2.7.2
创建自定义网络
docker network create -d bridge --subnet "172.173.16.0/24" --gateway "172.173.16.1" datastore_net
启动容器
docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hadoop:2.7.2
下载并配置hive
下载hive
-
拷贝到容器内
docker cp ~/Downloads/apache-hive-1.2.2-bin.tar.gz e2d21b78fdb0:/opt/envs -
解压并重命名
mkdir hive-1.2.2 && tar -zxvf apache-hive-1.2.2-bin.tar.gz -C hive-1.2.2 --strip-components 1
配置
hive-env.sh
-
a. 备份
cp /opt/envs/hive-1.2.2/conf/hive-env.sh.template /opt/envs/hive-1.2.2/conf/hive-env.sh -
b. 编辑hive-env.sh
vi /opt/envs/hive-1.2.2/conf/hive-env.sh -
c. 配置
①. 配置 HADOOP_HOME
export HADOOP_HOME=/opt/envs/hadoop-2.7.2②. 配置 HIVE_CONF_DIR
export HIVE_CONF_DIR=/opt/envs/hive-1.2.2/conf
安装which
不安装which，启动datanode时会报找不到文件的错误
yum install which
安装mysql
为啥需要mysql
[借鉴引用]
你在哪路径下，执行hive指令，就在哪路径下生成metastore_db建一套数据库文件，这样是极其不合适的，公司里每个人若不一样，则会显得非常混杂。导致员工之间无法公用交流，也不便于管理。
-
a .
重要如果安装了mysql，先删除，再重新安装rpm -qa | grep -i mysql rpm -ev --nodeps xxx find / -name mysql find / -name my.cnf rm -rf xxx/mysql rm -rf /etc/my.cnf -
b. 下载mysql
MySQL-client-5.6.48-1.el7.x86_64.rpm MySQL-server-5.6.48-1.el7.x86_64.rpm -
c. 拷贝到容器
docker cp ~/Downloads/MySQL-client-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs docker cp ~/Downloads/MySQL-server-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs -
d. 安装mysql服务端
① 安装服务端
rpm -ivh MySQL-server-5.6.48-1.el7.x86_64.rpm② 出现问题:
/usr/bin/perl is needed by MySQL-server-5.6.48-1.el7.x86_64 libaio.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64 libaio.so.1(LIBAIO_0.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64 libaio.so.1(LIBAIO_0.4)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64 libnuma.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64 libnuma.so.1(libnuma_1.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64 libnuma.so.1(libnuma_1.2)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64 net-tools is needed by MySQL-server-5.6.48-1.el7.x86_64 perl(Data::Dumper) is needed by MySQL-server-5.6.48-1.el7.x86_64③ 安装以下依赖
yum install -y perl perl-devel autoconf libaio numactl net-tools iproute④ server安装完成[看到以下内容证明完成了]
A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER ! You will find that password in '/root/.mysql_secret'. You must change that password on your first connect, no other statement but 'SET PASSWORD' will be accepted. See the manual for the semantics of the 'password expired' flag. Also, the account for the anonymous user has been removed. In addition, you can run: /usr/bin/mysql_secure_installation which will also give you the option of removing the test database. This is strongly recommended for production servers. See the manual for more instructions. Please report any problems at http://bugs.mysql.com/ The latest information about MySQL is available on the web at http://www.mysql.com Support MySQL by buying support/licenses at http://shop.mysql.com New default config file was created as /usr/my.cnf and will be used by default by the server when you start it. You may edit this file to change server settings -
e. 查看随机生成的密码
cat /root/.mysql_secret -
f. 启动mysql
① 启动
/etc/init.d/mysql start 或 service mysql start② 启动成功日志如下：
Starting MySQL.Logging to '/var/lib/mysql/e2d21b78fdb0.err'. SUCCESS! -
g. 安装mysql客户端
rpm -ivh MySQL-client-5.6.48-1.el7.x86_64.rpm -
h. 连接mysql，使用root用户
mysql -uroot -p -
i. 修改mysql root用户密码
set password=password('密码'); quit; -
j. 修改用户表，将host表的host列修改为任意ip可访问，并删除其他host（否则在后面启动时会报没有权限）
use mysql; update user set host = '%' where host = 'localhost'; delete from user where host = 'e2d21b78fdb0'; delete from user where Host='127.0.0.1'; delete from user where Host='::1'; flush privileges; -
k. 安装mysql-connector-java.jar
② 将jar包拷贝到 hive-1.2.2/lib 下
配置Metastore到Mysql
-
在hive-1.2.2/conf下创建hive-site.xml
vi hive-site.xml -
配置如下:
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://hive01:3306/metastore?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>密码</value> <description>password to use against metastore database</description> </property> <!-- 显示当前数据库，显示查询表的头信息 --> <property> <name>hive.cli.print.header</name> <value>true</value> </property> <property> <name>hive.cli.print.current.db</name> <value>true</value> </property> </configuration>
配置运行日志便于查询
-
备份
cp /opt/envs/hive-1.2.2/conf/hive-log4j.properties.template /opt/envs/hive-1.2.2/conf/hive-log4j.properties -
修改
vi /opt/envs/hive-1.2.2/conf/hive-log4j.properties -
修改hive.log.dir属性
hive.log.dir=/opt/envs/hive-1.2.2/logs
保存为镜像并移除容器
docker commit e2d21b78fdb0 caiser/hive:1.2.2
docker rm e2d21b78fdb0
启动容器
docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hive:1.2.2 bin/bash
配置ssh免密登录
-
进入容器
docker exec -it hive01 /bin/bash -
到~/.ssh目录下生成秘钥
ssh-keygen -t rsa -
拷贝秘钥到hadoop01，hadoop02和hadoop03
a.[如果没开启]开启ssh服务[ps -ef | grep ssh]
/usr/sbin/sshd -D &b. 拷贝秘钥到hadoop01，hadoop02，hadoop03
ssh-copy-id hadoop01 ssh-copy-id hadoop02 ssh-copy-id hadoop03 -
hadoop01,hadoop02和hadoop03依次执行
ssh-copy-id hive01
向hadoop集群中添加节点
-
动态
① 在hive01机器的hadoop/sbin目录下启动datanode：hadoop-daemon.sh start datanode② 在hive01机器的hadoop/sbin目录下启动nodemanager
yarn-daemon.sh start nodemanager③ jps查看datanode和nodemanager是否已启动
④ 回到namenode节点打印集群信息，或网页登录50070端口查看节点数量hdfs dfsadmin -report -
静态
① 停止hadoop集群
② 保证各节点hadoop的配置文件内容一致
③ 分别配置4个节点hadoop的salves文件hadoop01 hadoop02 hadoop03 hive01④ 重启hadoop集群
启动hive
-
启动
bin/hive -
看到这种就是成了
Logging initialized using configuration in file:/opt/envs/hive-1.2.2/conf/hive-log4j.properties hive (default)>
