Mysql空闲连接超时自动断开问题记录

Mysql数据库空闲连接默认8小时后会自动断开连接，此时由于业务处理使用C3P0连接池，业务在一个长时间休眠恢复后(其实是一个每天的定时任务)出现mysql 读写socket异常。异常栈信息如下：

com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:** BEGIN NESTED EXCEPTION **java.io.EOFExceptionMESSAGE: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.STACKTRACE:java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1997)    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)    at com.mysql.jdbc.Connection.execSQL(Connection.java:3283)    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1332)    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1467)    at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418)    at com.temp.datamigration.tool.DatasourceUtil.executMysqlQuery(DatasourceUtil.java:188)    at com.temp.datamigration.processor.recordprocessor.datacloud.UserSelfDefineProcessor.initUserSelfDefineSdsMode(UserSelfDefineProcessor.java:106)    at com.temp.datamigration.processor.recordprocessor.datacloud.UserSelfDefineProcessor.init(UserSelfDefineProcessor.java:79)    at com.temp.datamigration.processor.WearProcessor.initDataCloudProfileProcessors(WearProcessor.java:176)    at com.temp.datamigration.processor.WearProcessor.init(WearProcessor.java:156)    at com.temp.datamigration.bootstrap.Bootstrap.startMerge(Bootstrap.java:155)    at com.temp.datamigration.bootstrap.Bootstrap.access$000(Bootstrap.java:43)    at com.temp.datamigration.bootstrap.Bootstrap$1.run(Bootstrap.java:125)    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)    at java.lang.Thread.run(Thread.java:745)

或者如下

com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:** BEGIN NESTED EXCEPTION **java.net.SocketExceptionMESSAGE: Broken pipeSTACKTRACE:java.net.SocketException: Broken pipe        at java.net.SocketOutputStream.socketWrite0(Native Method)        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)        at java.net.SocketOutputStream.write(SocketOutputStream.java:159)        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2744)        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1612)        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)        at com.mysql.jdbc.Connection.execSQL(Connection.java:3283)        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1332)        at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1467)        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418)        at com.temp.datamigration.tool.DatasourceUtil.executMysqlQuery(DatasourceUtil.java:188)        at com.temp.datamigration.processor.WearProcessor.obtailUserToBeMerged(WearProcessor.java:100)        at com.temp.datamigration.processor.WearProcessor.init(WearProcessor.java:131)        at com.temp.datamigration.bootstrap.Bootstrap.startMerge(Bootstrap.java:156)        at com.temp.datamigration.bootstrap.Bootstrap.access$000(Bootstrap.java:43)        at com.temp.datamigration.bootstrap.Bootstrap$1.run(Bootstrap.java:126)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)        at java.lang.Thread.run(Thread.java:745)

mysql server gone away
mysql wait_timeout变量说明
C3P0连接可用测试配置说明

下面是我们代码中初始化C3P0 DataSource的代码

ComboPooledDataSource dataSource = new ComboPooledDataSource();dataSource.setDriverClass("com.mysql.jdbc.Driver");dataSource.setJdbcUrl(mysqlConfig.getJdbcUrl());dataSource.setUser(mysqlConfig.getUsername());dataSource.setPassword(mysqlConfig.getPasswd());dataSource.setMinPoolSize(5);dataSource.setAcquireIncrement(5);dataSource.setMaxPoolSize(maxPoolSize);dataSource.setInitialPoolSize(2);dataSource.setAcquireRetryDelay(400);

即未设置连接任何Connection Testing 配置，都使用默认值
preferredTestQuery 默认值null，没有校验连接可用性的测试语句
testConnectionOnCheckin 默认值false，创建连接时不会校验连接可用性
testConnectionOnCheckout 默认值false，从连接池borrow时不会校验连接可用性
idleConnectionTestPeriod 默认值为0，即不进行空闲测试逐出处理。

而连接池的minPoolSize却被设置为5，连接池默认最小有5个空闲连接。尼玛，这5个连接在长时间(8小时)后被mysql server断连了。

修改方案1：修改mysql wait_timeout值超过24小时。但不能从根本上解决问题，而且一般现网的mysql不能随意修改，该方案不可行。

修改方案2：配置C3P0的Connection Testing ，由连接池自行校验连接可用性。

dataSource.setPreferredTestQuery("select 1");dataSource.setTestConnectionOnCheckout(true);dataSource.setTestConnectionOnCheckin(true);

测试验证：
1)set global wait_timeout=30;—-将mysql的wait_timeout全局变量设置为30s。
2)将业务(定时读取Mysql的任务)定时周期挑战为1分钟。
即可复现和验证该问题。

而现网的Tomcat业务服务器未出现过类似问题，而我们的Tomcat业务服务使用Tomcat dbcp数据库连接池中间件，并设置了validationQuery为”select 1”，但未配置其他Connection Testing。
查看DBCP配置参数说明
testOnBorrow、testOnConnect、testOnReturn、testWhileIdle的默认值也都是False，按道理也应该会存在连接断连的问题的啊。
查看源码发现 org.apache.tomcat.dbcp.dbcp.BasicDataSource.java
testOnBorrow默认值是true。。。。和文档不符。。。