hive测试&&生产环境搭建

测试环境搭建

安装

https://hive.apache.org/downloads.html
因为需要使用Phoenix Storage Handler,所以选择支持hadoop2.x.y的 1.2+版本,hive1.2.2

配置环境变量

~/.bash_profile 配置hadoop&&hive变量

export HADOOP_HOME="/Users/titengjiang/Documents/DeveloperTool/hadoop/hadoop-2.7.5"
export PATH="$HADOOP_HOME/bin:$PATH"
export HIVE_HOME="/Users/titengjiang/Documents/DeveloperTool/hive/apache-hive-1.2.2-bin"
export PATH="$HIVE_HOME/bin:$PATH"

配置hive

mv hive-default.xml.template hive-site.xml

hive-site.xml增加如下配置

  <property>
    <name>system:java.io.tmpdir</name>
    <value>/Users/titengjiang/Documents/DeveloperTool/hive/local/tmp</value>
  </property>
  <property>
    <name>system:user.name</name>
    <value>${user.name}</value>
  </property>

解决如下异常

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

创建warehouse

  $ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
  $ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

初始化mysql metastore

create schema metastore_db default character set utf8;
CREATE USER hive@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON metastore_db.* TO hive@'%' identified by 'hive' ;
GRANT ALL PRIVILEGES ON metastore_db.* TO hive@'localhost identified by 'hive';
 flush privileges;

配置metastore,使用推荐的mysql替换derby

  <!-- 设置元数据存放的数据库地址、名字 -->
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore_db?useSSL=false</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/dbName?ssl=true for postgres database.
    </description>
  </property>
<!--Driver class名字-->
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
<!--连接使用的用户名-->
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>
  <!--连接使用的密码-->
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive</value>
    <description>password to use against metastore database</description>
  </property>

拷贝mysql-connection-version.jar 到hive lib下并初始化metastore_db

注意mysql-connection的版本选择5.1.34最好，较低版本可能会遇到错误

$HIVE_HOME/bin/schematool -dbType mysql -initSchema

如果出现上面的错误,检查下mysql 用户权限，连接之类的配置

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>

配置日志

hive-log4j.properties

hive.log.dir=/Users/username/Documents/DeveloperTool/hive/local/logs/${user.name} //日志路径改成指定路径

启动

$HIVE_HOME/bin/hiveserver2
hive --service metastore //启动metatore 提供hcatalog streaming写入端口 9083默认

常见错误

For direct MetaStore DB connections, we don’t support retries at the client level

create table years (year string, event string) row format delimited fields terminated by '\t';
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)

修改mysql-connection版本

最后编辑于：2018.01.20 13:09:27