dubbo Thread pool is EXHAUSTED-故障排除

整个排除故障过程使用了jstack命令

故障现象描述

dubbo Thread pool is EXHAUSTED

dubbo线程池耗尽

dubbo日志如下：

16:54:34,026 WARN [New I/O server worker #1-4] -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.8.64.57:20880, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 5823 (completed: 5623), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.8.64.57:20880!, dubbo version: 2.5.3, current host: 10.8.64.5716:54:34,027 WARN [New I/O server worker #1-4] -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.8.64.57:20880, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 5823 (completed: 5623), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.8.64.57:20880!, dubbo version: 2.5.3, current host: 10.8.64.5716:54:34,027 WARN [New I/O server worker #1-4] -  [DUBBO] An exception was thrown by a user handler while handling an exception event ([id: 0x7434600d, /10.8.64.57:59102 => /10.8.64.57:20880] EXCEPTION: com.alibaba.dubbo.remoting.ExecutionException: class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process received event .), dubbo version: 2.5.3, current host: 10.8.64.57com.alibaba.dubbo.remoting.ExecutionException: class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process caught event .at com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler.caught(AllChannelHandler.java:67)at com.alibaba.dubbo.remoting.transport.AbstractChannelHandlerDelegate.caught(AbstractChannelHandlerDelegate.java:44)at com.alibaba.dubbo.remoting.transport.AbstractChannelHandlerDelegate.caught(AbstractChannelHandlerDelegate.java:44)at com.alibaba.dubbo.remoting.transport.AbstractPeer.caught(AbstractPeer.java:127)at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.exceptionCaught(NettyHandler.java:112)at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.exceptionCaught(NettyCodecAdapter.java:165)at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:148)at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)at java.lang.Thread.run(Thread.java:619)Caused by: java.util.concurrent.RejectedExecutionException: Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.8.64.57:20880, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 5823 (com

下面是分析调试过程：

查看jstack 历史dump文件(card-service是一个采用了Dubbo框架的服务程序)
/data/service/card-service_20161104/logs/dump/20170829171658/jstack-93064.dump

如果现在dubbo刚刚挂掉。
可以根据card-service的pid来获取jstack日志
==
获取dubbo服务card-service的jstack日志的步骤如下：
1. 去程序源码找到这个dubbo服务的端口为20880
2. 根据20880查找对应的pid，发现pid为99668
[root@NYSJHL64-57 20170829171658]# netstat -ntlp |grep 20880
tcp        0      0 0.0.0.0:20880               0.0.0.0:*                   LISTEN      99668/java
3.        jstack 'pid' > xxxx.log
(xxx.log是你希望生成的jstack dump文件)
jstack 99668 > card-service_down.log




------
现在查看这个
jstack-93064.dump   或者xxxlog

"DubboServerHandler-10.8.64.57:20880-thread-111" daemon prio=10 tid=0x00007fd574193800 nid=0x16cf1 waiting for monitor entry [0x00007fd691887000..0x00007fd691888810]   java.lang.Thread.State: BLOCKED (on object monitor)at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)- waiting to lock <0x00007fd6c9fa4ba8> (a java.lang.Object)at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:250)at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:318)at sun.reflect.GeneratedMethodAccessor222.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:550)at $Proxy49.createSession(Unknown Source)at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:457)at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)at com.frank.searchClient.home.CardIndexerHome.createOrUpdate(CardIndexerHome.java:72)at com.frank.card.service.provider.UserCardProvider.createOrUpdateIndex(UserCardProvider.java:616)at com.frank.card.service.provider.UserCardProvider.updateCardNormal(UserCardProvider.java:432)at com.frank.card.worker.UpdateCardWorker.doJob(UpdateCardWorker.java:85)at com.frank.card.service.CardServiceImpl.updateCard(CardServiceImpl.java:300)at com.alibaba.dubbo.common.bytecode.Wrapper15.invokeMethod(Wrapper15.java)at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.filter.AccessLogFilter.invoke(AccessLogFilter.java:199)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:112)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)at java.lang.Thread.run(Thread.java:619)

关键点java.lang.Thread.State: BLOCKED (on object monitor)
dubbo的线程被activemq连接的线程所阻塞BLOCKED


由此推测测试环境的activemq运行遇到性能瓶颈（比如内存耗尽，或者cpu耗尽等）
因为activemq是轻量级中间件， cpu很少耗尽。    估计是内存参数设置太小， 内存被耗尽。
查看 下。 果然
-Xmx256m -Xms256m     256M内存显然太小。 改成下面这样的

set ACTIVEMQ_OPTS=-Xms1G-Xmx1G

改成1G内存， 大功告成，dubbo不再down了。

附：

jstack是jdk自带的一个命令

$JAVA_HOME/bin/jstack

例如

/opt/j2sdk/bin/jstack