hive-05-Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
来源:互联网 发布:tv霸网络电视破解版 编辑:程序博客网 时间:2024/05/16 08:08
hive命令行里执行了一句话:
select * from person join zhanghao on person.zjhm = zhanghao.zjhm limit 100 ;
就是两个表做连接查询,数据量大小一个是3千万,一个是3亿
结果报错
hive> select * from person join zhanghao on person.zjhm = zhanghao.zjhm limit 100 ; WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.Query ID = root_20171121152644_c3525e56-ea74-474f-8d14-c1d18f3b3f81Total jobs = 1SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/hzjs/spark-2.1.1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/hzjs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]2017-11-21 15:27:11,779 main ERROR Unable to locate appender "INFO,DRFA" for logger config "root"2017-11-21 15:27:15 Starting to launch local task to process map join; maximum memory = 4776263682017-11-21 15:27:39 Processing rows: 200000 Hashtable size: 199999 Memory usage: 275244704 percentage: 0.5762017-11-21 15:27:50 Processing rows: 300000 Hashtable size: 299999 Memory usage: 351999776 percentage: 0.7372017-11-21 15:28:01 Processing rows: 400000 Hashtable size: 399999 Memory usage: 373283344 percentage: 0.7822017-11-21 15:28:49 Processing rows: 500000 Hashtable size: 499999 Memory usage: 439679208 percentage: 0.921Execution failed with exit status: 3Obtaining error informationTask failed!Task ID: Stage-4Logs:FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
解决方法:
set hive.mapjoin.localtask.max.memory.usage=0.99
hive.mapjoin.localtask.max.memory.usage
•说明:本地任务可以使用内存的百分比 •默认值: 0.90,如果你的localtask mapjoin 表很小可以试试,但彻底解决需要
set hive.auto.convert.join=false;关闭自动mapjoin 但这个参数用的时候一定要注意,
如果你的sql 很长join会常多,关闭mapjoin任务数会成10倍激增,contener满了任务同样会非常之慢,set hive.auto.convert.join=false;一定要用在localtask级别这种超轻量及的job上。
阅读全文
0 0
- hive-05-Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- hive运行错误Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- Hive报错:FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- 一次hive reduce oom 处理:Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTas
- hive Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException
- FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(me
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
- Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeExcept
- 程序中执行hive sql出错[FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask]
- Hive连接Mysql---FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
- hive创建表失败:FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
- Zabbix企业微信告警最新版
- 前端调试工具(ionic)
- Sublime Text 3 全程详细图文原创教程
- java 中 string 和 boolean 类型相互转换
- 谷歌浏览器中添加网页授权回调域名时无保存按钮
- hive-05-Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- 编写函数,其功能是求3个整数的最大值和最小值
- 微信投票怎么刷票之微信投票怎么免费刷票的办法
- Vmware虚拟机三种网络模式详解
- VS Winform开发实践纪要
- Swift基础 下标
- 稀疏矩阵乘法
- onclick时间动态传递参数,拼接字符串参数
- 绝地求生个人挑战到底