个人博客地址:Tez 0.10.1安装 | 一张假钞的真实世界
具体安装步骤参照官网安装手册即可。此处只对官网手册进行补充。
从官网下载apache-tez-0.10.1-bin.tar.gz进行安装未成功,出现下面的异常。最终按照官网源代码编译的方式安装测试成功。
环境
- Hadoop 3.2.2
- Tez 0.10.1
二进制包安装异常
- 异常1:
2021-08-18 18:51:15,362 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED
org.apache.hadoop.service.ServiceStateException: java.lang.NoClassDefFoundError: org/eclipse/jetty/servlet/ServletContextHandlerat org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)at org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1879)at org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:1943)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2626)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2622)at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2409)
Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/servlet/ServletContextHandlerat org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:515)at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:521)at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:526)at org.apache.tez.dag.app.web.WebUIService.serviceStart(WebUIService.java:92)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1800)at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1821)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.servlet.ServletContextHandlerat java.net.URLClassLoader.findClass(URLClassLoader.java:381)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)at java.lang.ClassLoader.loadClass(ClassLoader.java:357)... 7 more
- 异常2:
2021-08-18 16:18:36,244 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED
org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)at org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1879)at org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:1943)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2626)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2622)at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2409)
Caused by: java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:569)at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:550)at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:117)at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:425)at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:341)at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:432)at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:428)at org.apache.tez.dag.app.web.WebUIService.serviceStart(WebUIService.java:94)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1800)at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1821)
安装说明
pom.xml
修改以下配置与使用的Hadoop版本一致:
<hadoop.version>3.2.2</hadoop.version>
修改以下配置与${HADOOP_HOME}/share/hadoop/common/lib/guava-27.0-jre.jar
版本一致。
<guava.version>27.0-jre</guava.version>
tez-site.xml
需要注意两点:
- 与官网不同,
tez.use.cluster.hadoop-libs
配置为true。 - 该配置文件需要分发到Hadoop集群的所有节点。位置:
${HADOOP_HOME}/etc/hadoop
目录下。
完整内容如下:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration><property><name>tez.lib.uris</name><value>${fs.defaultFS}/lib/tez/tez-0.10.1.tar.gz</value><type>string</type></property><property><name>tez.use.cluster.hadoop-libs</name><value>true</value></property>
</configuration>