Tez学习笔记@安装

安装前置条件

Tez 0.8.3及以上版本需要使用Hadoop 2.6.0及以上版本；

Tez 0.9.0及以上版本需要使用Hadoop 2.7.0及以上版本；

Tez安装

上传tar.gz包到HDFS

hadoop fs -mkdir-p /tez

hadoop fs -puttez.tar.gz /tez

检查是否上传成功：

hadoop fs -ls/tez/

 

在$HADOOP_HOME/etc/hadoop目录下增加tez-site.xml文件，配置内容如下：

<configuration>

<property>

<name>tez.lib.uris</name>

<value>hdfs://node01:9000/tez/tez.tar.gz</value>

</property>

</configuration>

 

hadoop-env.sh中添加tez环境变量

##tez

exportTEZ_HOME=/home/hadoop/public/tez

exportTEZ_CONF_DIR=${HADOOP_HOME}/etc/hadoop

exportHADOOP_CLASSPATH=$HADOOP_CLASSPATH:${TEZ_CONF_DIR}:${TEZ_HOME}/*:${TEZ_HOME}/lib/*

 

mapred-site.xml中修改Mr引擎

<property>

<name>mapreduce.framework.name</name>

<value>yarn-tez</value>

</property>

 

重启Hadoop集群

Tez测试

执行Tez自带的例子orderedwordcount

 

hadoop jar tez-examples-0.8.5.jarorderedwordcount /mr/input /mr/tez

Jobhistory监控界面：

问题解决

tez-examples-0.8.5.jar orderedwordcount例子报错

问题描述

For more detailed output, check applicationtracking page:http://node01:8088/proxy/application_1495800987987_0003/Then,click on links to logs of each attempt.

Diagnostics: Exception fromcontainer-launch.

Container id:container_1495800987987_0003_02_000001

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

       at org.apache.hadoop.util.Shell.runCommand(Shell.java:543)

       at org.apache.hadoop.util.Shell.run(Shell.java:460)

       atorg.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:720)

       atorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)

       at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

       atorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

       at java.util.concurrent.FutureTask.run(FutureTask.java:262)

       atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

       atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

       at java.lang.Thread.run(Thread.java:745)

 

 

Container exited with a non-zero exit code1

Failing this attempt. Failing theapplication.

2017-05-26 20:35:16,481 INFOorg.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:application_1495800987987_0003 State change from FINAL_SAVING to FAILED

2017-05-26 20:35:16,483 WARNorg.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop   OPERATION=Application Finished - FailedTARGET=RMAppManager    RESULT=FAILURE   DESCRIPTION=App failed with state:FAILED       PERMISSIONS=Applicationapplication_1495800987987_0003 failed 2 times due to AM Container forappattempt_1495800987987_0003_000002 exited with  exitCode: 1

For more detailed output, check applicationtracking page:http://node01:8088/proxy/application_1495800987987_0003/Then,click on links to logs of each attempt.

Diagnostics: Exception fromcontainer-launch.

Container id:container_1495800987987_0003_02_000001

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

       at org.apache.hadoop.util.Shell.runCommand(Shell.java:543)

       at org.apache.hadoop.util.Shell.run(Shell.java:460)

       atorg.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:720)

       atorg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)

       atorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

       atorg.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

       at java.util.concurrent.FutureTask.run(FutureTask.java:262)

       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

       atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

       at java.lang.Thread.run(Thread.java:745)

 

 

Container exited with a non-zero exit code1

Failing this attempt. Failing the application.  APPID=application_1495800987987_0003

2017-05-26 20:35:16,486 INFOorg.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary:appId=application_1495800987987_0003,name=OrderedWordCount,user=hadoop,queue=root.hadoop,state=FAILED,trackingUrl=http://node01:8088/cluster/app/application_1495800987987_0003,appMasterHost=N/A,startTime=1495802100615,finishTime=1495802116271,finalStatus=FAILED

2017-05-26 20:35:17,324 INFOorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:Null container completed...

问题解决

需要将${TEZ_HOME}/share/tez.tar.gz文件上传到Hdfs上，并在tez-site.xml文件tez.lib.uris属性中配置。