weblogic：java.lang.reflect.InvocationTargetException

2015-02-25 07:36:25:035[ERROR]: java.lang.reflect.InvocationTargetException -CannotCreateTransactionException:Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: Cannot obtain connection: driverURL = jdbc:weblogic:pool:test-racdb, props = {EmulateTwoPhaseCommit=false, connectionPoolID=test-racdb, jdbcTxDataSource=true, LoggingLastResource=false, dataSourceName=test-racdb}.
Nested Exception: java.lang.NullPointerException
        at weblogic.jdbc.wrapper.JDBCWrapperFactory.getWrapper(JDBCWrapperFactory.java:146)
        at weblogic.jdbc.pool.Driver.allocateConnection(Driver.java:227)
        at weblogic.jdbc.pool.Driver.connect(Driver.java:156)
        at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:662)
        at weblogic.jdbc.jts.Driver.connect(Driver.java:128)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:589)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:545)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:538)
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:202)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:335)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy112.receiveBondInfoHandle(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.sapcis.test.common.distributed.BeanMethodJob.execute(BeanMethodJob.java:29)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)


 2015-02-25 07:36:25:037[ERROR]: com.sapcis.test.common.distributed.BeanMethodJob.execute:29

========

####<Feb 25, 2015 7:36:25 AM CST> <Info> <Common> <bja-bapp2> <core_wss> <DefaultQuartzScheduler_Worker-9> <<anonymous>> <> <> <1424820985025> <BEA-000634> <Resource hang detected for pool "test-racdb", group "test2". Waited 10,012 milliseconds where a typical test has been taking 1.> 
####<Feb 25, 2015 7:36:25 AM CST> <Info> <Common> <bja-bapp2> <core_wss> <DefaultQuartzScheduler_Worker-10> <<anonymous>> <> <> <1424820985025> <BEA-000635> <Resource Pool "test-racdb", group "test2" disabled due to number of resource creation or test failures exceeding threshold of 2.> 
####<Feb 25, 2015 7:36:25 AM CST> <Info> <JDBC> <bja-bapp2> <core_wss> <DefaultQuartzScheduler_Worker-10> <<anonymous>> <> <> <1424820985029> <BEA-001128> <Connection for pool "test-racdb" has been closed.

========

Database Connection Testing Semantics

When WebLogic Server tests database connections in a data source, it reserves a connection from the data source, runs a small query on the connection, then returns the connection to the pool in the data source. The server instance tracks statistics on the pool status, including the amount of time a required to complete a connection test, the number of connections waiting for a connection, and the number of connections being tested. The history of recent test connection behavior is used to calculate the amount of time the server instance waits until a connection test is determined to have failed.


If a thread appears to be taking longer than normal to connect, the server instance may delay testing on other threads until the abnormally long-running test completes. If that thread hangs too long in connection testing (10 seconds by default), a pool may declare a DBMS connectivity failure, disable itself, and kill all connections, whether unreserved or in application hands.


This is very rare, and is intended to relieve the otherwise interminable hangs that can be caused by network cable disconnects and other problems that can lock any JVM thread which is doing a call in a socket read that the JVM will be unable to break until the OS TCP limit is hit (typically 10 minutes). If a pool disables itself in this manner, it will periodically try to reconnect to the DBMS (every 5 seconds by default). Once a new connection can be made, the pool will re-enable itself, and subsequent connection requests will be served as normal (with the pool repopulating itself as load requires).


The query used in testing is determined by the value in Test Table Name. If the value is a table name, the query is select 1 from table_name. If Test Table Name includes a full query starting with SQL followed by space and the query, WebLogic Server uses that query when testing database connections.


If a connection fails the test, WebLogic Server closes and recreates the connection, and then tests the new connection.


Details about the semantics of connection testing is discussed in the following sections:


"Connection Testing When Database Connections are Created"
"Periodic Connection Testing"
"Testing Reserved Connections"
"Minimized Connection Test Delay After Database Connectivity Loss"
"Minimized Connection Request Delay After Connection Test Failures"
"Minimizing Connection Request Delay with Seconds to Trust an Idle Pool Connection"
Connection Testing When Database Connections are Created


When connections are created in a data source, WebLogic Server tests each connection using the query defined by the value in Test Table Name. Connections are created when a data source is deployed, either at server startup or when creating a data source, when increasing capacity to meet demand for connections, or when recreating a connection that failed a connection test.


The purpose of this testing is to ensure that new connections are viable and ready for use when an application requests a connection.


Periodic Connection Testing


If Test Frequency is greater than 0, WebLogic Server periodically tests the pooled connections that are not currently reserved by applications. The test is based on the query defined in Test Table Name. If a connection fails the test, WebLogic Server closes the connection, recreates the connection, and tests the new connection before returning it to the pool.


Testing Reserved Connections


When Test Connections On Reserve is enabled, when your application requests a connection from the data source, WebLogic Server tests the connection using the query specified in Test Table Name before giving the connection to the application. The default value is not enabled.


Testing reserved connections can cause a delay in satisfying connection requests, but it makes sure that the connection is viable when the application gets the connection. You can minimize the impact of testing reserved connections by tuning Seconds to Trust an Idle Pool Connection. See Minimizing Connection Request Delay with Seconds to Trust an Idle Pool Connection.


Minimized Connection Test Delay After Database Connectivity Loss


When connectivity to the DBMS is lost, even if only momentarily, some or all of the JDBC connections in a data source typically become defunct. If the data source is configured to test connections on reserve, when an application requests a database connection, WebLogic Server tests the connection, discovers that the connection is dead, and tries to replace it with a new connection to satisfy the request. Ordinarily, when the DBMS comes back online, the refresh process succeeds. However, in some cases and for some modes of failure, testing a dead connection can impose a long delay.


To minimize this delay, WebLogic data sources include logic that considers all connections in the data source as dead after a number of consecutive test failures, and closes all connections in the data source. After all connections are closed, when an application requests a connection, the data source creates a connection without first having to test a dead connection. This behavior minimizes the delay for connection requests following the data source's connection pool flush.


WebLogic Server determines the number of test failures before closing all connections based on the Test Frequency setting for the data source:


If Test Frequency is greater than 0, the number of test failures before closing all connections is set to 2.


If Test Frequency is set to 0 (periodic testing is disabled), the number of test failures before closing all connections is set to 25% of the Maximum Capacity for the data source.


Minimized Connection Request Delay After Connection Test Failures


If your DBMS becomes and remains unavailable, the data source will persistently test and try to replace dead connections while trying to satisfy connection requests. This behavior is beneficial because it enables the data source to react immediately when the database becomes available. However, testing a dead database connection can take as long as the network timeout, and can cause a long delay for clients.


To minimize this delay, the WebLogic data sources include logic that disables the data source after 2 consecutive failures to replace a dead connection. When an application requests a connection from a disabled data source, WebLogic Server throws a PoolDisabledSQLException immediately to notify the client that a connection is not available.


For data sources that are disabled in this manner, WebLogic Server periodically runs a refresh process. The refresh process does the following:


The server instance executes a health check on the database server every 5 seconds. This setting is not configurable.


If the server instance recognizes that the database was recovered, it creates a new database connection and enables the data source.


You can also manually enable the data source using the Administration Console.


Minimized Connection Request Delays After Loss of DBMS Connectivity


If your DBMS becomes and remains unavailable, the data source will persistently test and try to replace dead connections while trying to satisfy connection requests. This behavior is beneficial because it enables the data source to react immediately when the database becomes available. However, testing a dead database connection and trying futilely to replace it can in some cases take as long as the OS network timeout (minutes), and can cause long delays to clients before getting the expected failure message.


Note:
If a data source is added to a multi data source, the multi data source takes over the responsibility of disabling and re-enabling its data sources. By default, a multi data source will check every two minutes (configurable) and re-enable any of its data sources that can re-establish connections.
Minimizing Connection Request Delay with Seconds to Trust an Idle Pool Connection


For some applications that use DBMS connections in a lot of very short cycles (such as reserve-do_one_query-close), the data source's testing of the connection can contribute a significant amount of overhead to each use cycle. To minimize the impact of connection testing, you can set the Seconds To Trust An Idle Pool Connection attribute in the JDBC data source configuration to trust recently-used or recently-tested database connections and skip the connection test.


If Test Reserved Connections is enabled on your data source, when an application requests a database connection, WebLogic Server tests the database connection before giving it to the application. If the request is made within the time specified for Seconds to Trust an Idle Pool Connection, since the connection was tested or successfully used by an application, WebLogic Server skips the connection test before delivering it to an application.


If Test Frequency is greater than 0 for your data source (periodic testing is enabled), WebLogic Server also skips the connection test if the connection was successfully used and returned to the data source within the time specified for Seconds to Trust an Idle Pool Connection.


For instructions to set Seconds to Trust an Idle Pool Connection, see "Configure testing options for a JDBC data source" in the Oracle WebLogic Server Administration Console Help.


Seconds to Trust an Idle Pool Connection is a tuning feature that can improve application performance by minimizing the delay caused by database connection testing, especially during heavy traffic. However, it can reduce the effectiveness of connection testing, especially if the value is set too high. The appropriate value depends on your environment and the likelihood that a connection will become defunct.


Database Connection Testing Configuration Recommendations


You should set connection testing attributes so that they best fit your environment. For example, if your application cannot tolerate database connection failures, you should set Seconds to Trust an Idle Pool Connection to 0 and make sure Test Reserved Connections is enabled so that WebLogic Server will test every connection before giving it to an application. If your application is more sensitive to delays in getting a connection from the data source and can tolerate a possible application failure due to using a dead connection, you should set Seconds to Trust an Idle Pool Connection to a higher number, set Test Frequency to a lower number, and enable Test Reserved Connections.


With these settings, your application will rely more on the data source testing connections in the pool when they are not in use, rather than when an application requests a connection.


Note:
Ultimately, even if WebLogic does its best, a connection may fail in the instant after WebLogic successfully tested it, and just before the application uses it. Therefore, every application should be written to respond appropriately in the case of unexpected exceptions from a dead connection.