hive on tez Caused by: java.lang.OutOfMemoryError: Java heap space

昨天进行两个hive表关联导出数据，不幸的是爆出如下错误：

Status: Failed
Vertex failed, vertexName=Map 5, vertexId=vertex_1443634917922_0008_1_05, diagnostics=[Task failed, taskId=task_1443634917922_0008_1_05_000006, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
    at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
    at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
    at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
    at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
    at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space

仔细观看日志发现上面错误信息，主要是因为内存不足，有意思的是我整个服务器都在给它跑，怎么会内存不足呢？后来查阅资料和咨询我一个同事吴哥发现，这个内存不足是值java堆内存，好把，既然内存不足，那我就看看hive给的默认内存是多少

hive>SET hive.tez.container.size;

hive.tez.container.size=6144;

hive>SET hive.tez.java.opts;

hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps

尴尬了，确实有点小，那么容器我给它20G，java.opt给80%容器试试，反正服务器内存大

SET hive.tez.container.size=20480;
SET hive.tez.java.opts=-Xmx16384m;

解决了，真是天天踩坑。