hive on tez

编译 tez

下载 apache-tez-0.9.2-src 并解压到/opt下

设置环境变量

#安装tez-ui的时候，node 会被自动下载到的目录，所以必须设置到path环境变量中弄过，不然后面的命令会报找不到node
export PATH=${PATH}:/opt/apache-tez-0.9.2-src/tez-ui/src/main/webapp/node
#bower global安装后的软连接路径
export PATH=${PATH}:/opt/apache-tez-0.9.2-src/tez-ui/src/main/webapp/bin
# ./yarn global add  bower@1.8.4
#在编译tez-ui的时候，bower 很可能安装不上，报错找不到bower命令，此时需要手动安装bower，因为front-maven-plugin已经自动安装的node 和 yarn包管理工具。yarn包管理命令在/opt/apache-tez-0.9.2-src/tez-ui/src/main/webapp/node/yarn/dist/bin下，所以可以通过yarn global add  bower@x.x.x 来安装，具体版本好，在tez-ui下的package.json中。
export PATH=${PATH}:/opt/.config/yarn/global/node_modules/bower/bin

1.环境要求

ProtocolBuffer2.5.0
JDK8 or later 
Maven 3 or later

可参考官方的编译与安装步骤
http://tez.apache.org/install.html

1.下载源码包

直接使用0.9.2的下载src
http://mirror.bit.edu.cn/apache/tez/0.9.2/apache-tez-0.9.2-src.tar.gz
在这里插入图片描述

2.安装protobuf

安装gcc, gcc-c++, make, build
yum install gcc gcc-c++ libstdc++-devel make build
安装ProtocolBuffer2.5.0
https://github.com/google/protobuf/releases/tag/v2.5.0
wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
./configure
make && make install
测试安装是否成功
[root@hadoop protobuf-2.5.0]# protoc --version
libprotoc 2.5.0

3.源码根目录pom.xml中添加相关依赖与集成hadoop版本

3.1修改集成hadoop版本

方式一: 修改跟目录/tez/apache-tez-0.9.2-src的pom.xml
<hadop.version> 2.6.0-cdh5.16.2<hadoop.version>
方式二: 直接在maven编译命令中加 
 -Dhaddop.version=2.6.0-cdh5.16.2
本地使用第二种方式

3.2由于集成的是cdh版本 需要在repository中加上cdh仓库

    <repository>
        <id>cloudera</id>
        <name>cloudera Repository</name>
        <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
       <snapshots>
              <enabled>false</enabled>
       </snapshots>
    </repository>

    <pluginRepository>
        <id>cloudera</id>
        <name>cloudera Repository</name>
        <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
    </pluginRepository>

3.3如果是用国内服务器编译,可以将本地maven中添加阿里云依赖

博主是安装在/usr/local下
cd /usr/local/maven/conf
vim settings.xml
<mirrors>
   <mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>central</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
   </mirror>
   <mirror>
     <id>CN</id>
     <name>OSChina Central</name>
     <url>http://maven.oschina.net/content/groups/public/</url>
     <mirrorOf>central</mirrorOf>
   </mirror>
   <mirror>
     <id>alimaven</id>
     <mirrorOf>central</mirrorOf>
     <name>aliyun maven</name>
     <url>https://maven.aliyun.com/nexus/content/repositories/central/</url>
   </mirror>
   <mirror>
     <id>jboss-public-repository-group</id>
     <mirrorOf>central</mirrorOf>
     <name>JBoss Public Repository Group</name>
     <url>https://repository.jboss.org/nexus/content/groups/public</url>
   </mirror>
   <mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>*</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    </mirror>
  </mirrors>

3.4 为了能够顺利编译tez-ui,安装nodejs与bower

a. nodejs安装
  $> wget https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.xz
  $> tar xf  node-v10.9.0-linux-x64.tar.xz
  $> ln -s /usr/software/nodejs/bin/npm /usr/local/bin/
  $> ln -s /usr/software/nodejs/bin/node /usr/local/bin/
  $> node -v  ##查看版本

b. bower安装
$> npm install -g bower

npm太慢， 淘宝npm镜像使用方法
https://blog.csdn.net/qq_37674858/article/details/80065701

npm --registry https://registry.npm.taobao.org install express

npm install -g bower

ln -s /usr/local/node/node-v6.17.1-linux-x64/bin/bower /usr/local/bin/bower

# 能够执行出命令
[root@hadoop003 /]# bower --allow-root -v
1.8.8

4.编译及报错

4.1在源码包的根目录执行编译命令

mvn clean package -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dhadoop.version=2.6.0-cdh5.16.2

4.2 报错

（1）报错1：在JobContextImpl.java中找不到userClassesTakesPrecedence的实现

[INFO] tez-mapreduce ...................................... FAILURE [  0.575 s]
ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project tez-mapreduce: Compilation failure
[ERROR] /tez/apache-tez-0.9.2-src/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapreduce/JobContextImpl.java:[58,8] org.apache.tez.mapreduce.hadoop.mapreduce.JobContextImpl is not abstract and does not override abstract method userClassesTakesPrecedence() in org.apache.hadoop.mapreduce.JobContext
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :tez-mapreduce

解决：
 改正：修改/tez/apache-tez-0.9.2-src/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/mapred/JobContextImpl.java文件：追加下面代码
     @Override
     public boolean userClassesTakesPrecedence() {
       return getJobConf().getBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, false);
     } 
    并导入类
     import org.apache.tez.mapreduce.hadoop.MRJobConfig;
在这里插入图片描述

（2）tez-ui编译报错

如果是使用root用户编译tez-ui会报错
报错日志：
'bower install' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
解决：
a.在其他用户下编译
b.修改pom文件 , 允许root用户编译
改正：修改apache-tez-0.9.2-src/tez-ui/pom.xml文件，在pom包中搜索bower，修改如下
<execution>
  <phase>generate-resources</phase>
  <id>bower install</id>
  <configuration>
    <arguments>install</arguments>
    <arguments>--allow-root</arguments>
  </configuration>
  <goals>
    <goal>bower</goal>
  </goals>
</execution>
关键是增加--allow-root ，默认情况下 bower intall命令是不允许root用户执行的。

(3) HDFS Truncate截断功能目前发布在Hadoop 2.7.0以及以上的版本中，低于此版本调用fs.truncate()会报错

[INFO] tez-protobuf-history-plugin ........................ FAILURE [  0.441 s]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project tez-protobuf-history-plugin: Compilation failure
[ERROR] /tez/apache-tez-0.9.2-src/tez-plugins/tez-protobuf-history-plugin/src/test/java/org/apache/tez/dag/history/logging/proto/TestDagManifestFileScanner.java:[157,17] cannot find symbol
[ERROR]   symbol:   method truncate(org.apache.hadoop.fs.Path,long)
[ERROR]   location: variable fs of type org.apache.hadoop.fs.FileSystem
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :tez-protobuf-history-plugin

这个报错出现在 TestDagManifestFileScanner中，这个是测试类。 maven中加了-DskipTests不够，还要加-Dmaven.test.skip=true ，在编译中已经加好了。
-Dmaven.test.skip=true  不但跳过单元测试的运行，也跳过测试代码的编译
-DskipTests=true  跳过单元测试，但是会继续编译

参考资料：
Hive on Tez
https://my.oschina.net/guol/blog/915416?p=1

5.编译完成

INFO] Reactor Summary for tez 0.9.2:
[INFO] 
[INFO] tez ................................................ SUCCESS [  0.997 s]
[INFO] hadoop-shim ........................................ SUCCESS [  1.264 s]
[INFO] tez-api ............................................ SUCCESS [  4.649 s]
[INFO] tez-build-tools .................................... SUCCESS [  0.052 s]
[INFO] tez-common ......................................... SUCCESS [  0.367 s]
[INFO] tez-runtime-internals .............................. SUCCESS [  0.612 s]
[INFO] tez-runtime-library ................................ SUCCESS [  1.922 s]
[INFO] tez-mapreduce ...................................... SUCCESS [  0.787 s]
[INFO] tez-examples ....................................... SUCCESS [  0.257 s]
[INFO] tez-dag ............................................ SUCCESS [  3.390 s]
[INFO] tez-tests .......................................... SUCCESS [  0.394 s]
[INFO] tez-ext-service-tests .............................. SUCCESS [  0.442 s]
[INFO] tez-ui ............................................. SUCCESS [ 22.191 s]
[INFO] tez-plugins ........................................ SUCCESS [  0.018 s]
[INFO] tez-protobuf-history-plugin ........................ SUCCESS [  0.321 s]
[INFO] tez-yarn-timeline-history .......................... SUCCESS [  0.163 s]
[INFO] tez-yarn-timeline-history-with-acls ................ SUCCESS [  0.096 s]
[INFO] tez-history-parser ................................. SUCCESS [ 10.582 s]
[INFO] tez-aux-services ................................... SUCCESS [  4.762 s]
[INFO] tez-tools .......................................... SUCCESS [  0.016 s]
[INFO] tez-perf-analyzer .................................. SUCCESS [  0.015 s]
[INFO] tez-job-analyzer ................................... SUCCESS [  0.267 s]
[INFO] tez-javadoc-tools .................................. SUCCESS [  0.091 s]
[INFO] hadoop-shim-impls .................................. SUCCESS [  0.017 s]
[INFO] hadoop-shim-2.7 .................................... SUCCESS [  0.065 s]
[INFO] tez-dist ........................................... SUCCESS [ 16.737 s]
[INFO] Tez ................................................ SUCCESS [  0.016 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:10 min
[INFO] Finished at: 2020-07-28T16:50:02+08:00
[INFO] ------------------------------------------------------------------------

在根目录的tez-dist/target中可以看到编译好的包
/tez/apache-tez-0.9.2-src/tez-dist/target

在这里插入图片描述

6.安装tez（可用）

本地安装在hive，使得hive能够自由切换mr与tez两种引擎

6.1 解压tez-0.0.2.tar.gz ，并上传至hdfs目录（可任意指定）

6.2配置tez-site.xml

默认配置之文件 可以下载bin包解压后看到
在这里插入图片描述
在/hive/apache-hive-2.3.3-bin/conf目录下，具体配置含义和调优参考官方文档
touch  tez-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

<property>
    <name>tez.lib.uris</name>
    <value>${fs.defaultFS}/tez/tez-0.9.2,${fs.defaultFS}/tez/tez-0.9.2/lib</value>
</property>

<property>
    <name>tez.lib.uris.classpath</name>
   <value>${fs.defaultFS}/tez/tez-0.9.2,${fs.defaultFS}/tez/tez-0.9.2/lib</value>
</property>
<property>
     <name>tez.use.cluster.hadoop-libs</name>
     <value>true</value>
</property>
<property>
     <name>tez.history.logging.service.class</name>
     <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>

</configuration>

6.3配置hive-env.sh

# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/hadoop/hadoop

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/hive/hive/conf
export TEZ_HOME=/tez/tez-0.9.2
# tez-site.xml所在目录
export TEZ_CONF_DIR=$HIVE_CONF_DIR
#将jar包添加到 hive环境中，也可直接将所有jar拷贝到hive lib下
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
   export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
  export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done

# 使用hive的aux_jar的方式添加
# /hive/hive/lib/hive-exec-1.1.0-cdh5.16.2.jar 前面必须要加一个jar ,无论是什么都行，不然会报错Expected authority at index 7
export HIVE_AUX_JARS_PATH=/hadoop/hadoop/share/hadoop/mapreduce1/hadoop-core-2.6.0-mr1-cdh5.16.2.jar $TEZ_JARS

7.测试使用

7.1开启hive的metastore

<property>
                <name>hive.metastore.uris</name>
                 <value>thrift://hadoop03:9083,thrift://hadoop02:9083,thrift://hadoop01:9083</value>
                 <description>对外提供接口，如sparksql连接hive</description>
 </property>

7.2 启动metastore

nohup hive --service metastore & 

# 如果不开启,会报错
hive (default)> show tables;
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

7.3 使用Tez报错汇总

报错1
hadoop默认使用的是容量调度器
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1595946044525_0012_1_00, diagnostics=[Vertex vertex_1595946044525_0012_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: empt initializer failed, vertex=vertex_1595946044525_0012_1_00 [Map 1], java.lang.IllegalArgumentException: Illegal Capacity: -1044

将默认容量调度器改成公平调度器(可不改)  
https://blog.csdn.net/weixin_38586230/article/details/106073251
原因是资源被抢占或者资源不足
vim tez-site.xml  这个不是重点
<property>
        <name>tez.am.max.app.attempts</name>
        <value>5</value>
        <description>am自己失败的最大重试次数，默认是2次。这里并不是说am自己挂了，只是因为一些系统原因导致失联>了</description>
</property>

<property>
        <name>tez.am.task.max.failed.attempts</name>
        <value>10</value>
        <description>任务中attempts失败的最大重试次数，默认跟yarn一样是4次,在不稳定集群可以设置大一点</description>
</property>

vim hive-site.xml
  <property>
    <name>hive.tez.container.size</name>
    <value>-1</value>
    <description>By default Tez will spawn containers of the size of a mapper. This can be used to overwrite.</description>
  </property>

可以调大,测试调成4G,

报错2

    killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: test initializer failed
    https://blog.csdn.net/hqwang4/article/details/72861969
    解决办法：
        将hadoop-core-2.6.0-mr1-cdh5.16.2.jar 包添加到HIVE_CLASS中，如下：
    export HIVE_AUX_JARS_PATH=/hadoop/hadoop/share/hadoop/mapreduce1/hadoop-core-2.6.0-mr1-cdh5.16.2.jar 

    如果还报错,则将hadoop-core-2.6.0-mr1-cdh5.16.2.jar拷贝到/hive/hive/auxlib中

报错3
java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/Serializer
https://www.jianshu.com/p/45c95a51a8c2

    将/hive/hive/auxlib中的jar包重名令
    mv hive-exec-1.1.0-cdh5.16.2-core.jar hive-exec-1.1.0-cdh5.16.2-core.jar.bak

7.4 案例

set hive.execution.engine=tez;

hive (default)> select count(1) from kylin_account;
Query ID = root_20200729103333_5c9d45b7-8e1d-4dfb-9268-f769551783af
Total jobs = 1
Launching Job 1 out of 1

Status: Running (Executing on YARN cluster with App id application_1595989425725_0003)

--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ..........   SUCCEEDED      1          1        0        0       0       0
Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0
--------------------------------------------------------------------------------
VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 5.32 s     
--------------------------------------------------------------------------------
OK
_c0
10000

set hive.execution.engine=mr;

hive (default)> set hive.execution.engine=mr;
hive (default)> select count(1) from kylin_account;
Automatically selecting local only mode for query
Query ID = root_20200729103434_005ecfee-d25d-419f-897f-21bed5029db7
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Job running in-process (local Hadoop)
2020-07-29 10:34:52,565 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_local895650320_0001
MapReduce Jobs Launched: 
Stage-Stage-1:  HDFS Read: 400012 HDFS Write: 109149264 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
_c0
10000

7.5 TezUI安装

7.5.1

tar -zxvf apache-tomcat-8.5.31.tar.gz -C /opt/modules/
cd /opt/module/tomcat-8.5.59/conf
vi server.xml 
# 随便改别和现有占用的端口冲突就行
image.png
image.png
# 这块配个环境变量
vi /etc/profile
export TOMCAT_HOME=/opt/module/tomcat-8.5.59
@#启动
${TOMCAT-HOME}/bin/startup.sh
# 关闭
${TOMCAT-HOME}/bin/shutdown.sh

  • 登录ip:端口


    image.png
image.png
  • 出现上面那个进程不一定说明成功,下面这个异常是我部署时候出现的坑
Oct 17, 2020 10:30:09 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact [localhost:8005]. Tomcat may not be running.
Oct 17, 2020 10:30:09 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop: 
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:504)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:388)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
  • 注:上面这个异常时关闭Tomcat出现的,发现开启tomcat后关不掉或者Tomcat的webUI界面连不上
  • 解决思路:jps看看Tomcat的进程bootstrap是不是没有了,我在部署时候发现Tomcat进程可以启动,但是过几秒就进程闪退(自动退出,卡掉了),原因就是在部署Tomcat文件夹下配置文件的时候有一个端口后配混了,观看此文的兄弟们要仔细了
# 需要修改的是这个端口,这个才是咱们web访问的
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
# 而不是这个
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <!-- Security listener. Documentation at /docs/config/listeners.html
  <Listener className="org.apache.catalina.security.SecurityListener" />
  -->
  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

7.5.2将tez-ui部署在tomcat


#在webapps下创建tez-ui目录
mkdir ${TOMCAT-HOME}/webapps/tez-ui/

#进入文件
cd ${TOMCAT-HOME}/webapps/tez-ui/

#解压war包,可以官方编译好的,也可用自己编译的
# 要是有小伙伴编译不出来就用官方的吧,省事, 这块我编译这个用来好久才成功,卡在bower 这,不一定允许root那个有了就能编译上,跟系统中依赖bower的版本也有关系,
我后来都是手动安装的bower安装依赖的yarn和npm,原因是系统本身安装别的程序时候自动装了一些别的工具比如npm,他在安装bower时候就去找系统默认的那个,而不走你正常的bower所需环境的工具,卸载固有的,安装版本要和brower的版本对上的工具,并指定ln -s 到编译能检测到直接引用的地方就可
版本这块去tez-ui的pom.xml里去看吧
jar -xvf    /opt/modules/tez-0.9.2/tez-ui-0.9.2.war

#编辑配置文件，第26行修改localhost
vim config/configs.env
改下面这俩,把这俩配置打开
    timeline: "http://端口:8188",
    rm: "http://端口:8088",

7.5.3 修改yarn配置文件,就安装我这个来,全是tez-ui的timeline官网基本配置

vi yarn-site.xml

 <!-- 向client标示Timeline Services是否是启用状态。如果启用，application使用的TimelineClient library会发送entitiy和event到Timeline server。默认为false-->
 <property>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.timeline-service.hostname</name>
  <value>192.168.111.111</value>
</property>
<property>
  <name>yarn.timeline-service.http-cross-origin.enabled</name>
  <value>true</value>
</property>
<property>
  <name> yarn.resourcemanager.system-metrics-publisher.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.timeline-service.generic-application-history.enabled</name>
  <value>true</value>
</property>
<property>
  <description>Address for the Timeline server to start the RPC server.</description>
  <name>yarn.timeline-service.address</name>
  <value>${yarn.timeline-service.hostname}:10201</value>
</property>
<property>
  <description>The http address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.address</name>
  <value>${yarn.timeline-service.hostname}:8188</value>
</property>
<property>
  <description>The https address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.https.address</name>
  <value>${yarn.timeline-service.hostname}:2191</value>
</property>
<property>
  <name>yarn.timeline-service.handler-thread-count</name>
  <value>24</value>
</property>

7.5.4tez-site.xml添加配置

        <property>
                <!--自己安装的tomcat的端口号-->
                <name>tez.tez-ui.history-url.base</name>
                <value>http://192.168.111.111:8080/tez-ui/</value>
        </property>
        <property>
                <name>tez.runtime.convert.user-payload.to.history-text</name>
                <value>true</value>
        </property>
        <property>
                <name>tez.task.generate.counters.per.io</name>
                <value>true</value>
        </property>

7.5.5启动时间轴

yarn-daemon.sh start timelineserver

  • hive跑个sql


    image.png
image.png

image.png

image.png
  • 装完了就是一个字,爽

8.安装tez (这种安装方式测试不通过)

参考安装
https://my.oschina.net/guol/blog/915416

8.1 创建hdfs的tez目录,并将文件上传

hdfs hdfs -mkdir /tez
hfds dfs -put tez-0.9.2.tar.gz /tez/

8.2 配置tez-site.xml

默认配置之文件 可以下载bin包解压后看到

在$HIVE_HOME/conf目录下，具体配置含义和调优参考官方文档
touch  tez-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

<property>
    <name>tez.lib.uris</name>
    <value>${fs.defaultFS}/tez/tez-0.9.2.tar.gz</value>
</property>

<property>
     <name>tez.use.cluster.hadoop-libs</name>
     <value>true</value>
</property>
</configuration>

(3)本地tez包目录  
mkdir /tez/tez-0.9.2
tar  -zxvf   tez-0.9.2-minimal.tar.gz  -C /tez/tez-0.9.2

(4)在hadoop.env.sh中指定HADOOP_CLASSPATH

# tez-site.xml所在的目录
export TEZ_CONF_DIR=/hive/hive/conf
# tez的jar所在目录
export TEZ_JARS=/tez/tez-0.9.2
export HADOOP_CLASSPATH=$HADOOP_CLASSPAT:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*

备注：以上只是安装tez的一种方式，这种安装方式测试报错, 建议还是使用安装在hive-env.sh中的方式,这种方式对hadoop的侵入性更小.

注:本文总结了cscn和简书一些写的比较好和精的大佬们的文章差不多都搜了还有一些国外的文章也看一点在编译tez-ui那,站在巨人的肩膀上看世界,在此感谢,大佬们文章链接就不打了太多了,某度和某goo搜hive-on-te编译就能出来.

©著作权归作者所有,转载或内容合作请联系作者
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。