MapReduce job Shuffle 过程的ERROR
来源:互联网 发布:英伟达驱动游戏优化 编辑:程序博客网 时间:2024/06/03 17:12
1.错误描述
error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#43
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1550)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:414)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:343)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:166)
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1550)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:414)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:343)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:166)
2.问题解决
从error的stack信息可以看出是mapreduce 的job shuffle过程中发生的OOM,map的输出数据在reduce节点load的过多导致的OOM,所以将reduce的shuffle.input 由默认的0.7 改为0.6,该参数的变化需要根据实际情况确定。
conf.set("mapreduce.reduce.shuffle.input.buffer.percent", "0.6");
3.相关参数
- mapreduce.reduce.shuffle.input.buffer.percent :
the percentage of the reducer's heap memory to be allocated for the circular buffer to store the intermedite outputs copied from multiple mappers.
mapreduce.reduce.shuffle.memory.limit.percent
:
the maximum percentage of the above memory buffer that a single shuffle (output copied from single Map task) should take. The shuffle's size above this size will not be copied to the memory buffer, instead they will be directly written to the disk of the reducer.
- mapreduce.reduce.shuffle.merge.percent:
the threshold percentage by where the in-memory merger thread will run to merge the available shuffle contents on the memory buffer into a single file and immediately spills the merged file into the disk.
0 0
- MapReduce job Shuffle 过程的ERROR
- MapReduce的shuffle过程
- MapReduce的shuffle过程
- MapReduce的shuffle过程
- mapreduce的shuffle过程
- mapreduce的shuffle过程
- MapReduce的Shuffle过程
- 详解MapReduce的Shuffle过程
- MapReduce的shuffle过程详解
- Hadoop MapReduce的shuffle过程
- MapReduce的Shuffle过程介绍
- MapReduce的Shuffle过程介绍
- MapReduce的shuffle过程图解
- MapReduce的Shuffle过程介绍
- MapReduce的Shuffle过程详解
- 《MapReduce:详细介绍Shuffle的执行过程》
- mapreduce的执行流程以及shuffle过程
- Shuffle过程是MapReduce的核
- 商城列表效果的实现(三)整理
- HDU 1016 Prime Ring Problem
- JS-事件类型(鼠标事件中event对象的独有属性)
- STM32系列第29篇--DS18B20
- Android百度地图定位收索取周边在列表中展示并选择
- MapReduce job Shuffle 过程的ERROR
- OKhttp的回调的代码
- 【Android】Toast控件使用
- lightoj 1369 - Answering Queries 数学
- 多态经典面试题解析
- 杭电1012
- 剑指offer 66 机器人的运动范围
- SpringMvc 上传excel(注解和非注解两种方式)
- Vim插件