凌晨lvs连接不上realserver 异常分析

现象：凌晨3点多几秒，，，， lvs VIP连接realserver的心跳多次连不上，，，导致lvs拒绝为realserver服务。（所有的http服务器都连不上）

分析：所有的机器都连不上，，说明说是缓存或者数据库或者存储的问题，，，（因为是所有http应用都不能提供服务）

分析了凌晨3点跑的定时任务，没有慢查或者比较耗资源的sql， 排查数据库的问题

分析了凌晨3点redis，mongo 的日志，发现运行正常，，，，，

找不到头绪。。。。。。

oy想了一个办法，在凌晨3点整到   3点1分，，， 每隔2s跑一次jstack，把日志保持下来

这个方法有效果啊，，，，

第二天开始分析jstack

根据昨晚jstack打出的线程堆栈分析，发现有 264个线程卡在 locked <0x0000000600d68cd0> (a com.mchange.v2.resourcepool.BasicResourcePool) 语句
也就是说，有264个线程获取不到连接


"http-nio-80-exec-166" daemon prio=10 tid=0x00007fdca00b3800 nid=0xa3b3 in Object.wait() [0x00007fda6e6a7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x0000000600d68cd0> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:66)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:271)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:69)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:56)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:259)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:132)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:104)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:98)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:62)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:354)
at com.sun.proxy.$Proxy15.selectOne(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159)
at com.newtaxi.biz.common.dao.impl.AndroidAPNDaoImpl.getPushMessage(AndroidAPNDaoImpl.java:45)
at com.newtaxi.biz.common.api.impl.TaxiApnServiceImpl.getAndroidPushMessage(TaxiApnServiceImpl.java:62)
at sun.reflect.GeneratedMethodAccessor1590.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
at com.sun.proxy.$Proxy19.getAndroidPushMessage(Unknown Source)
at com.taxi.handler.user.AndroidPushMessageHandler.handle(AndroidPushMessageHandler.java:25)
at com.taxi.handler.user.AndroidPushMessageHandler.handle(AndroidPushMessageHandler.java:19)
at com.taxi.filter.RequestHandleFilter.doFilter(RequestHandleFilter.java:125)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.taxi.filter.SignatureProvingFilter.doFilter(SignatureProvingFilter.java:110)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.taxi.filter.AuthorizeFilter.doFilter(AuthorizeFilter.java:145)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.taxi.web.filter.LogRequestFilter.doFilter(LogRequestFilter.java:83)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1686)
- locked <0x000000072d666b88> (a org.apache.tomcat.util.net.NioChannel)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

"http-nio-80-exec-156" daemon prio=10 tid=0x00007fdc940a5000 nid=0xa3a6 in Object.wait() [0x00007fda6f3b4000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
- locked <0x0000000600d68cd0> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:66)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:271)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:69)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:56)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:259)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:132)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:104)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:98)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:62)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:354)
at com.sun.proxy.$Proxy15.selectOne(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159)
at com.newtaxi.biz.common.dao.impl.AndroidAPNDaoImpl.getPushMessage(AndroidAPNDaoImpl.java:45)
at com.newtaxi.biz.common.api.impl.TaxiApnServiceImpl.getAndroidPushMessage(TaxiApnServiceImpl.java:62)
at sun.reflect.GeneratedMethodAccessor1590.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
at com.sun.proxy.$Proxy19.getAndroidPushMessage(Unknown Source)
at com.taxi.handler.user.AndroidPushMessageHandler.handle(AndroidPushMessageHandler.java:25)
at com.taxi.handler.user.AndroidPushMessageHandler.handle(AndroidPushMessageHandler.java:19)
at com.taxi.filter.RequestHandleFilter.doFilter(RequestHandleFilter.java:125)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.taxi.filter.SignatureProvingFilter.doFilter(SignatureProvingFilter.java:110)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.taxi.filter.AuthorizeFilter.doFilter(AuthorizeFilter.java:145)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.taxi.web.filter.LogRequestFilter.doFilter(LogRequestFilter.java:83)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1686)
- locked <0x000000072db30ab8> (a org.apache.tomcat.util.net.NioChannel)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)


   Locked ownable synchronizers:
- <0x0000000617d04f28> (a java.util.concurrent.ThreadPoolExecutor$Worker)




可见在凌晨3点出几秒，，，com.newtaxi.biz.common.api.impl.TaxiApnServiceImpl.getAndroidPushMessage 读数据库获取连接被卡住了，
导致tomcat 连接池耗尽，，，lvs  VIP的心跳包连接不通.


那么为什么是凌晨3点出现这个问题呢？




该接口调用是客户端发起的，，，发起的时间是凌晨1点到5点（整点），就是说客户端有可能在1点或2点，3点，4点，5点调用这个请求
看看客户端那边的随机数算法


Random random = new Random(80000);
            final int nextPullHour = (random.nextInt() % 5 + 1);
            Calendar calendar = Calendar.getInstance();
            calendar.setTime(new Date((System.currentTimeMillis() + 24 * 60 * 60 * 1000)));
            calendar.set(Calendar.HOUR_OF_DAY, nextPullHour);
            calendar.set(Calendar.MINUTE, 0);
            calendar.set(Calendar.SECOND, 0);


我们拿这个代码在客户端同学的机器做测试，跑20万随机数试验，，发现跑出来的结果分布不均匀
（即使是均匀的，这个算法也是要调整的，，，，最好按秒随机）

04-02 16:42:20.350: D/zheng(24869): num1 = 39894
04-02 16:42:20.350: D/zheng(24869): num2 = 19954
04-02 16:42:20.350: D/zheng(24869): num3 = 20000
04-02 16:42:20.350: D/zheng(24869): num4 = 19818
04-02 16:42:20.350: D/zheng(24869): num5 = 19869       




那么基本可以断定是随机数算法导致，大量请求在凌晨3点  发起，导致http服务几乎瘫痪     

=========================================== 除了以上问题外，，，，我们怀疑凌晨3点，，数据库可能也有其他任务在跑，，，

所有把mysql的slowlog打开，，，，