hive-05-Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

来源:互联网 发布:tv霸网络电视破解版 编辑:程序博客网 时间:2024/05/16 08:08

hive命令行里执行了一句话:

select * from person  join zhanghao   on person.zjhm = zhanghao.zjhm limit 100 ; 

就是两个表做连接查询,数据量大小一个是3千万,一个是3亿
结果报错

hive> select * from person  join zhanghao   on person.zjhm = zhanghao.zjhm limit 100 ;  WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.Query ID = root_20171121152644_c3525e56-ea74-474f-8d14-c1d18f3b3f81Total jobs = 1SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/hzjs/spark-2.1.1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/hzjs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]2017-11-21 15:27:11,779 main ERROR Unable to locate appender "INFO,DRFA" for logger config "root"2017-11-21 15:27:15     Starting to launch local task to process map join;      maximum memory = 4776263682017-11-21 15:27:39     Processing rows:        200000  Hashtable size: 199999  Memory usage:   275244704       percentage:     0.5762017-11-21 15:27:50     Processing rows:        300000  Hashtable size: 299999  Memory usage:   351999776       percentage:     0.7372017-11-21 15:28:01     Processing rows:        400000  Hashtable size: 399999  Memory usage:   373283344       percentage:     0.7822017-11-21 15:28:49     Processing rows:        500000  Hashtable size: 499999  Memory usage:   439679208       percentage:     0.921Execution failed with exit status: 3Obtaining error informationTask failed!Task ID:  Stage-4Logs:FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

解决方法:
set hive.mapjoin.localtask.max.memory.usage=0.99
hive.mapjoin.localtask.max.memory.usage
•说明:本地任务可以使用内存的百分比 •默认值: 0.90,如果你的localtask mapjoin 表很小可以试试,但彻底解决需要
set hive.auto.convert.join=false;关闭自动mapjoin 但这个参数用的时候一定要注意,
如果你的sql 很长join会常多,关闭mapjoin任务数会成10倍激增,contener满了任务同样会非常之慢,set hive.auto.convert.join=false;一定要用在localtask级别这种超轻量及的job上。

阅读全文
0 0
原创粉丝点击