hiveserver2服务异常停止java.lang.OutOfMemoryError: Java heap space
来源:互联网 发布:师洋的淘宝店铺是 编辑:程序博客网 时间:2024/06/02 05:21
2017-01-02 15:34:04,461 WARN [IPC Client (188363230) connection to nn01/192.168.1.4:8020 from hive]: ipc.Client (Client.java:run(955)) - Unexpected error reading responses on connection Thread[IPC Client (188363230) connection to nn01.100.cloud/172.16.8.4:8020 from hive,5,main]
java.lang.OutOfMemoryError: Java heap space
at com.google.protobuf.CodedInputStream.<init>(CodedInputStream.java:573)
at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:199)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)
2017-01-02 15:34:03,365 FATAL [Thread-51]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(91)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
at org.apache.thrift.transport.TSocket.<init>(TSocket.java:78)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:114)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:121)
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:86)
at java.lang.Thread.run(Thread.java:745)
2017-01-02 15:34:29,186 INFO [Thread-50]: server.HiveServer2 (HiveServer2.java:stop(266)) - Shutting down HiveServer2
2017-01-02 15:34:29,186 INFO [Thread-50]: thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2017-01-02 15:34:29,188 INFO [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at nn02.100.cloud/192.168.1.5
heapsize太小了,可以适当的调大些
Root cause, the heapsize of hadoop opts is too small, need to increase the size in hive-env.sh as bold:
继续分析查看下进程:
观察2天后发现新生代和老年代都已经爆满,永久代几乎爆满,jvm一直在做full GC,但是没有效果,各个内存区都被占用了。
就日志来看,老年代和持久代的内存只要上去了,就再没有下降过,而当这两块区域饱和后,新生代也出现了这种现象。
java.lang.OutOfMemoryError: Java heap space
at com.google.protobuf.CodedInputStream.<init>(CodedInputStream.java:573)
at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:199)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)
2017-01-02 15:34:03,365 FATAL [Thread-51]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(91)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
at org.apache.thrift.transport.TSocket.<init>(TSocket.java:78)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:114)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:121)
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:86)
at java.lang.Thread.run(Thread.java:745)
2017-01-02 15:34:29,186 INFO [Thread-50]: server.HiveServer2 (HiveServer2.java:stop(266)) - Shutting down HiveServer2
2017-01-02 15:34:29,186 INFO [Thread-50]: thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2017-01-02 15:34:29,188 INFO [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at nn02.100.cloud/192.168.1.5
分析及解决:
java.lang.OutOfMemoryError: Java heap space内存溢出报错
heapsize太小了,可以适当的调大些
Root cause, the heapsize of hadoop opts is too small, need to increase the size in hive-env.sh as bold:
if [ "$SERVICE" = "cli" ]; then if [ -z "$DEBUG" ]; then export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit" else export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit" fifi# The heap size of the jvm stared by hive shell script can be controlled via:#export HADOOP_HEAPSIZE=2048
继续分析查看下进程:
监控语句如下:
jstat -gcutil [进程号] 5000 --每5秒打印一次jvm各个内存区的状态
另外找了一个监控系统内存的语句:
vmstat | sed -n '/[0-9]/p'
观察2天后发现新生代和老年代都已经爆满,永久代几乎爆满,jvm一直在做full GC,但是没有效果,各个内存区都被占用了。
就日志来看,老年代和持久代的内存只要上去了,就再没有下降过,而当这两块区域饱和后,新生代也出现了这种现象。
是有什么资源没释放吧?一看代码,果然,有的连接没有关闭!
如果还是有问题,hive0.14版本,自身也存在着bug,可以考虑进行定期重启hiveserver2服务,别笑实际工作中很多大的公司都是这样做的。
参考:
http://blog.csdn.net/gklifg/article/details/50418109
http://blog.csdn.net/lulynn/article/details/46455097
0 0
- hiveserver2服务异常停止java.lang.OutOfMemoryError: Java heap space
- java.lang.OutOfMemoryError: Java heap space异常
- 异常:java.lang.OutOfMemoryError: Java heap space
- java.lang.OutOfMemoryError: Java heap space异常
- hiveserver2 java.lang.OutOfMemoryError: PermGen space问题解决
- how-to: resolve "java.lang.OutOfMemoryError: Java heap space" during using beeline && hiveserver2
- java.lang.OutOfMemoryError: Java heap space异常处理
- Eclipse下解决java.lang.OutOfMemoryError: Java heap space异常
- Caused by: java.lang.OutOfMemoryError: Java heap space 异常
- Caused by: java.lang.OutOfMemoryError: Java heap space 异常
- myeclipse+tomcat:java.lang.OutOfMemoryError: Java heap space异常
- java.lang.OutOfMemoryError: Java heap space异常解决
- tomcat启动出现异常:java.lang.OutOfMemoryError: Java heap space
- java.lang.OutOfMemoryError: Java heap space异常解决方法。
- idea编译时报java.lang.OutOfMemoryError: Java heap space异常
- tomcat中java.lang.OutOfMemoryError: Java heap space异常处理
- java.lang.OutOfMemoryError: Java heap space 异常解决
- java.lang.OutOfMemoryError: Java heap space异常处理
- 一个简单的linux命令——whereis
- 获取EXE或者DLL的版本信息
- jQuery中AJAX,$.post()与$.get()的区别
- Android viewpager滑动指示器,滑动卡片,
- 不经过docker hub 直接将别处的image 拷贝到相应机器
- hiveserver2服务异常停止java.lang.OutOfMemoryError: Java heap space
- Android adb shell启动应用程序的方法
- Java中String的常用方法
- hprof文件打开和转换方式
- Spring与Maven两种占位符的替换时机
- sentry使用方法
- 设计对于开发是很重要的
- java web几种开发模式
- 静默获取用户基本信息的一种方法