hiveserver 占用内存过大的问题
来源:互联网 发布:网络女强与传统女强 编辑:程序博客网 时间:2024/04/30 11:21
今天为了求解hiveserver占用内存过大的问题,特地加了hive在apache的邮件列表,讨论半天。特别说的是 里面的人确实很热情啊 ,外国人做事确实很认真,讨论帖发的时候都狠详细。
粘出一些记录:
Did you update your JDK in last time? A java-dev told me that could bea issue in JDK _26(https://forums.oracle.com/forums/thread.jspa?threadID=2309872), somedevs report a memory decrease when they use GC - flags. I'm quite notsure, sounds for me to far away.The stacks have a lot waitings, but I see nothing special.- Alex2011/12/12 王锋 <wfeng1982@163.com>:>> The hive log:>> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121840_767713480.txt> 8159.581: [GC [PSYoungGen: 1927208K->688K(2187648K)]> 9102425K->7176256K(9867648K), 0.0765670 secs] [Times: user=0.36 sys=0.00,> real=0.08 secs]> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121841_451939518.txt> 8219.455: [GC [PSYoungGen: 1823477K->608K(2106752K)]> 8999046K->7176707K(9786752K), 0.0719450 secs] [Times: user=0.66 sys=0.01,> real=0.07 secs]> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121842_1930999319.txt>> Now we have 3 hiveservers and I set the concurrent job num to 4,but the Mem> still be so large .I'm mad, God>> have other suggestions ?>> 在 2011-12-12 17:59:52,"alo alt" <wget.null@googlemail.com>> 写道:>>When you start a high-load hive query can you watch the stack-traces?>>Its possible over the webinterface:>>http://jobtracker:50030/stacks>>>>- Alex>>>>>>2011/12/12 王锋 <wfeng1982@163.com>>>>>>> hiveserver will throw oom after several hours .>>>>>>>>> At 2011-12-12 17:39:21,"alo alt" <wget.null@googlemail.com> wrote:>>>>>> what happen when you set xmx=2048m or similar? Did that have any negative effects for running queries?>>>>>> 2011/12/12 王锋 <wfeng1982@163.com>>>>>>>>> I have modify hive jvm args.>>>> the new args is -Xmx15000m -XX:NewRatio=1 -Xms2000m .>>>>>>>> but the memory used by hiveserver is still large.>>>>>>>>>>>>>>>>>>>>>>>> At 2011-12-12 16:20:54,"Aaron Sun" <aaron.sun82@gmail.com> wrote:>>>>>>>> Not from the running jobs, what I am saying is the heap size of the Hadoop really depends on the number of files, directories on the HDFS. Remove old files periodically or merge small files would bring in some performance boost.>>>>>>>> On the Hive end, the memory consumed also depends on the queries that are executed. Monitor the reducers of the Hadoop job, and my experiences are that reduce part could be the bottleneck here.>>>>>>>> It's totally okay to host multiple Hive servers on one machine.>>>>>>>> 2011/12/12 王锋 <wfeng1982@163.com>>>>>>>>>>> is the files you said the files from runned jobs of our system? and them can't be so much large.>>>>>>>>>> why is the cause of namenode. what are hiveserver doing when it use so large memory?>>>>>>>>>> how do you use hive? our method using hiveserver is correct?>>>>>>>>>> Thanks.>>>>>>>>>> 在 2011-12-12 14:27:09,"Aaron Sun" <aaron.sun82@gmail.com> 写道:>>>>>>>>>> Not sure if this is because of the number of files, since the namenode would track each of the file and directory, and blocks.>>>>> See this one. http://www.cloudera.com/blog/2009/02/the-small-files-problem/>>>>>>>>>> Please correct me if I am wrong, because this seems to be more like a hdfs problem which is actually irrelevant to Hive.>>>>>>>>>> Thanks>>>>> Aaron>>>>>>>>>> 2011/12/11 王锋 <wfeng1982@163.com>>>>>>>>>>>>>>>>>>> I want to know why the hiveserver use so large memory,and where the memory has been used ?>>>>>>>>>>>> 在 2011-12-12 10:02:44,"王锋" <wfeng1982@163.com> 写道:>>>>>>>>>>>>>>>>>> The namenode summary:>>>>>>>>>>>>>>>>>>>>>>>> the mr summary>>>>>>>>>>>>>>>>>> and hiveserver:>>>>>>>>>>>>>>>>>> hiveserver jvm args:>>>>>> export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=1 -Xms15000m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParallelGC -XX:ParallelGCThreads=20 -XX:+UseParall>>>>>> elOldGC -XX:-UseGCOverheadLimit -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps">>>>>>>>>>>> now we using 3 hiveservers in the same machine.>>>>>>>>>>>>>>>>>> 在 2011-12-12 09:54:29,"Aaron Sun" <aaron.sun82@gmail.com> 写道:>>>>>>>>>>>> how's the data look like? and what's the size of the cluster?>>>>>>>>>>>> 2011/12/11 王锋 <wfeng1982@163.com>>>>>>>>>>>>>>> Hi,>>>>>>>>>>>>>> I'm one of engieer of sina.com. We have used hive ,hiveserver several months. We have our own tasks schedule system .The system can schedule tasks running with hiveserver by jdbc.>>>>>>>>>>>>>> But The hiveserver use mem very large, usally large than 10g. we have 5min tasks which will be running every 5 minutes.,and have hourly tasks .total num of tasks is 40. And we start 3 hiveserver in one linux server,and be cycle connected .>>>>>>>>>>>>>> so why Memory of hiveserver using so large and how we do or some suggestion from you ?>>>>>>>>>>>>>> Thanks and Best Regards!>>>>>>>>>>>>>> Royce Wang
最上面 Alex发现一篇文章
https://forums.oracle.com/forums/thread.jspa?threadID=2309872说是 jdk_1.0.26存在泄露的风险,我们正在使用也正是同一个版本,看这个url里文章说的也是谁也不能确认,而oracle方自然说不由其负责。
I tried with java6u29 and java7 and they work great. Actually on the production server we are running for almost 4 days with java7 and it's stable, no crash, no slowdown, no restart in this period, and with less maximum memory. If it's going to last for a week then I trust it will go on fine.
最后是有人用java6u29 和java7 运行 稳定。特别是java7.明天尝试在hiveserver服务器换用java7试试。
append。。。。
今天改用jdk 7测试 情况基本一致,看来问题并不是 jvm问题。
使用jmap -heap 发现 hiveserver 新生代 并没有去按照ratio设置的 那样,最大容量还是默认的800m,这个对数据分析来说太小了,使用xmn配置新生代,并配置最大新生代大小,而且将gc机制改为cms,目前内存占用稳定在 2.3g左右。
最后的参数 :
export HADOOP_OPTS="$HADOOP_OPTS -Xms5000m -Xmn4000m -XX:MaxNewSize=4000m -Xss128k -XX:MaxHeapFreeRatio=80 -XX:MinHeapFreeRatio=40 -XX:+UseParNewGC -XX:+UseConcMarkSw
eepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:-UseGCOverheadLimit -XX:MaxTenuringThreshold=8 -XX:P
ermSize=800M -XX:MaxPermSize=800M -XX:GCTimeRatio=19 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
- hiveserver 占用内存过大的问题
- firefox和thunderbird的内存占用过大问题
- 飞信内存占用过大问题的临时解决方案
- Android 图片加载导致内存占用过大的问题
- 图片轮播器占用内存过大问题的解决办法
- SVN CPU内存占用过大问题
- PHPExcel解决内存占用过大问题
- 解决linux服务器内存占用过大问题
- SVN CPU内存占用过大问题
- javafx——内存占用过大问题
- [ZT]MSSQL占用内存过大的解决方法
- MSSQL占用内存过大的解决方法
- PHPExcel占用内存过大
- 解决迅雷占用系统资源过大的问题
- win7 64位的 svchost.exe 占用内存过大的问题
- 解决Firefox占用内存“过大”的问题(FireFox about:config介绍)
- 【转载】SQL Server占用内存过大导致服务器死机问题的解决
- PHPExcel解决内存占用过大问题-设置单元格对象缓存
- Linux系统引导流程(开启系统、运行级别、服务、故障)
- 绿镜头补丁
- 基于Servlet网页验证码的生成
- 2011-12-11 三多身上有你的影子,树身上可能也有你的影子
- 中文编码、unicode、全角
- hiveserver 占用内存过大的问题
- 让Android设备永不锁屏
- Java平台的企业版
- 在Eclipse中运行第一个MapReduce程序
- 隐藏/显示div
- oracle开启10046的方法
- 嵌入式软件工程师应聘知識點
- 构建Oracle高可用环境HA rac:企业级高可用数据库架构、实战与经验总结
- 使用qt编写mel的UI