Tez上线部署记录

前言：

        Tez在Hive 0.13.0版本引入，全称Hive on Tez。顾名思义就是基于Tez之上的运行Hive作业的形式。具体Tez是什么？Tez是一种运算框架，MapReduce是一种运算框架，这么说应该懂了。

更具体的讲，Tez是一种内存运算框架，类似Spark的核心——RDD。

        正如刚才提到的Tez类似RDD的特性，适用于DAG运算，多路并行聚合运算，迭代运算。得益于Tez采用Map,Reduce分离的执行策略，传统意义的MapReduce作业，Map和Reduce不可分离，必须成对出现，缺一不可。这在不需要Map环节的作业中会耗费不少的内存和IO开销——这里的IO指磁盘和内存之间。同时这也拖曳了Map和Reduce之间的并行度。

        然而Tez就完全不一样了，Tez通过把MapReduceWork分解成MapWork和ReduceWork,这个过程中抛弃了不必要的MapWork。

        就像这样：

          可以看到，过程中没有Map了，这样的形式，我们称为MRR。

          更为重要的一点，基于内存模式，任何中间数据不再flush到磁盘，全程在内存当中，IO和CPU之间不再有瓶颈。

适用场景：

          简单点说就是，任何复杂语句，任何长语句。具体一点就是下面场景：

• MRR 多阶段的Reduce操作。
• MPJ 多路Join操作。

使用方法：

         在你的执行语句中加上以下参数即可：

set hive.execution.engine=tez;

set mapreduce.reduce.java.opts=-Xmx1536m -verbose:gc -Xloggc:/opt/hadoop/bfdhadoop26/logs/hadoop/@taskid@.gc;

set mapreduce.map.java.opts=-Xmx1536m -verbose:gc -Xloggc:/opt/hadoop/bfdhadoop26/logs/hadoop/@taskid@.gc;

踩坑记录：

        在运行时，报错：

2015-04-20 10:19:59,437 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1429505171727_0001_02_000001 and exit code: -1

ExitCodeException exitCode=-1:

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

    at org.apache.hadoop.util.Shell.run(Shell.java:455)

    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

    at java.util.concurrent.FutureTask.run(FutureTask.java:262)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

container初始化失败，查看contain目录下的stderr文件，有以下信息：

Conflicting collector combinations in option list; please refer to the release notes for the combinations allowed

Error: Could not create the Java Virtual Machine.

Error: A fatal exception has occurred. Program will exit.

看似回收器冲突，拉取container的launch_container.sh文件来看：

在脚本的执行命令中看到了两个垃圾回收器设置，第二个是我们提交作业设置的，第一次是Tez默认的设置。

在代码tez-branch-0.5.2/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java中：

public static final String TEZ_AM_LAUNCH_CMD_OPTS = TEZ_AM_PREFIX +  "launch.cmd-opts";

  public static final String TEZ_AM_LAUNCH_CMD_OPTS_DEFAULT =

      "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC";

  /**

   * String value. Command line options which will be prepended to {@link

   * #TEZ_TASK_LAUNCH_CMD_OPTS} during the launch of Tez tasks.  This property will typically be configured to

   * include default options meant to be used by all jobs in a cluster. If required, the values can

   * be overridden per job.

   */

  public static final String TEZ_TASK_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS =

      TEZ_TASK_PREFIX + "launch.cluster-default.cmd-opts";

  public static final String TEZ_TASK_LAUNCH_CLUSTER_DEFAULT_CMD_OPTS_DEFAULT =

      "-server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN";

  /**

   * String value. Command line options provided during the launch of Tez Task

   * processes. Its recommended to not set any Xmx or Xms in these launch opts

   * so that Tez can determine them automatically.

   */

  public static final String TEZ_TASK_LAUNCH_CMD_OPTS = TEZ_TASK_PREFIX

      + "launch.cmd-opts";

  public static final String TEZ_TASK_LAUNCH_CMD_OPTS_DEFAULT =

      "-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC";

解决方法就是上面提到的，提交前设置参数。

另一个很诡异的问题：

2015-11-08 15:30:33,550 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1446967358935_0001 failed 2 times due to AM Container for appattempt_1446967358935_0001_000002 exited with  exitCode: -1000

For more detailed output, check application tracking page:http://bjlg-112p14-hadoop04.bfdabc.com:8088/proxy/application_1446967358935_0001/Then, click on links to logs of each attempt.

Diagnostics: Resource hdfs://bfdhadoop26/tmp/hive/hadoop/_tez_session_dir/dbc39c11-6869-4b7e-9833-172e40912c88/lib changed on src filesystem (expected 1446967815233, was 1446967815339

java.io.IOException: Resource hdfs://bfdhadoop26/tmp/hive/hadoop/_tez_session_dir/dbc39c11-6869-4b7e-9833-172e40912c88/lib changed on src filesystem (expected 1446967815233, was 1446967815339

        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:255)

        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)

        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)

        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)

        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Failing this attempt. Failing the application.

AMcontainer直接launch不了，没有任何日志。查看了nodemanager,resourcemanager,hive.log全都是上面这个错。

查看yarn-common包的源码：

private Path copy(Path sCopy, Path dstdir) throws IOException {

  FileSystem sourceFs = sCopy.getFileSystem(conf);

  Pattern p = Pattern.compile("_tez_session_dir");

  Matcher m = p.matcher(sCopy.toString());

  Path dCopy = new Path(dstdir, "tmp_"+sCopy.getName());

  FileStatus sStat = sourceFs.getFileStatus(sCopy);

  if (sStat.getModificationTime() != resource.getTimestamp()) {

    if (m.find() != false)

        LOG.warn("This job is Tez.and not check the ts. for " + sCopy.toString());

    else

      throw new IOException("Resource " + sCopy +

        " changed on src filesystem (expected " + resource.getTimestamp() +

        ", was " + sStat.getModificationTime());

  }

  if (resource.getVisibility() == LocalResourceVisibility.PUBLIC) {

    if (!isPublic(sourceFs, sCopy, sStat, statCache)) {

      throw new IOException("Resource " + sCopy +

          " is not publicly accessable and as such cannot be part of the" +

          " public cache.");

    }

  }

  FileUtil.copy(sourceFs, sStat, FileSystem.getLocal(conf), dCopy, false,

      true, conf);

  return dCopy;

}

Tez作业在分发Map,Reduce作业时，会在container launch之前，从hdfs上下载jar包等资源文件。就是我们常说的localize resource阶段。在下载之前，会校验resource文件是否被修改。

校验的方式是比较文件的时间戳是否一样。而且很奇怪的是文件一样，时间戳会不一样。（此处原由还没有搞清楚）

所以我简单做了下处理，判断如果是tez任务，如果时间戳不同，不抛出异常。程序执行正常。

遗留问题，还待研究：

1.Tez作业resource分发原理。

2.Tez session以及在Hive Cli模式下会话超时。