hadoop在对数据压缩时出现的问题

来源:互联网 发布:微信小程序域名未备案 编辑:程序博客网 时间:2024/06/05 00:16

最近在对数据做压缩时,报了数组太大,导致内存不够的错误:

2012-12-19 13:15:03,866 FATAL org.apache.hadoop.mapred.Child: Error running child :

java.lang.OutOfMemoryError: allocLargeObjectOrArray: [I, size 3600016at org.apache.hadoop.io.compress.bzip2.CBZip2OutputStream$Data.<init>(CBZip2OutputStream.java:2074)
at org.apache.hadoop.io.compress.bzip2.CBZip2OutputStream.init(CBZip2OutputStream.java:747)
at org.apache.hadoop.io.compress.bzip2.CBZip2OutputStream.<init>(CBZip2OutputStream.java:637)
at org.apache.hadoop.io.compress.bzip2.CBZip2OutputStream.<init>(CBZip2OutputStream.java:594)
at org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionOutputStream.internalReset(BZip2Codec.java:186)
at org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionOutputStream.write(BZip2Codec.java:205)
at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:86)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:761)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:848)
at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:265)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:198)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)


报此错误是在reduce阶段出现。我尝试着修改了reduce数(加大),之前(hive-site.xml)设置是10,后来加到25,最后设置成40,就运行OK了。

解决方式:设定reduce数(具体数字需要自己测试下,根据数据量的不同来判断,我这次是用我们能用到最大的数据容量做动态分区加bz压缩)

总数据量在130GB左右。

 

另外我们尝试过不用压缩,也是能正常运行的。

说明使用压缩的时候,我们还有些问题没有更好的解决掉,后续会专门研究其压缩的使用。

原创粉丝点击