weblogic 服务不响应java.net.SocketTimeoutException: Read timed out

  weblogic服务不能访问，后台的报错，重启服务就恢复正常了。

####<2016-1-4 上午08时29分16秒 CST> <Error> <HTTP> <zcglappserver04> <APP02_6004> <[ACTIVE] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1451867356003> <BEA-101017> <[ServletContext@338963353[app:EAR module:web path:/web spec-version:2.5]] Root cause of ServletException.

weblogic.utils.NestedRuntimeException: Cannot parse POST parameters of request: '/web/ontractpayment/invoice/queryNewTaskInfoCountByUserIdAndprocessId.sc'
at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.mergePostParams(ServletRequestImpl.java:2144)
at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.parseQueryParams(ServletRequestImpl.java:2024)
at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.getQueryParams(ServletRequestImpl.java:1918)
at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.getParameter(ServletRequestImpl.java:1995)
at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.access$800(ServletRequestImpl.java:1817)
at weblogic.servlet.internal.ServletRequestImpl.getParameter(ServletRequestImpl.java:804)
at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:169)
at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:169)
at com.mmbpms.action.WorkflowController.queryNewTaskInfoCountByUserIdAndprocessId(WorkflowController.java:143)
at sun.reflect.GeneratedMethodAccessor28557.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at com.common.dao.DataSourceFilter.doFilter(DataSourceFilter.java:46)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)
at com.common.dao.OpenSessionFilter.doFilterInternal(OpenSessionFilter.java:86)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Caused By: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at weblogic.servlet.internal.PostInputStream.read(PostInputStream.java:177)
at weblogic.servlet.internal.ServletInputStreamImpl.read(ServletInputStreamImpl.java:228)

at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.mergePostParams(ServletRequestImpl.java:2118)

Intermittent HTTP error 500 during QA/load tests (文档 ID 1344882.1)

APPLIES TO:

Oracle WebLogic Server - Version 10.3 to 12.1.1.0
Information in this document applies to any platform.
***Checked for relevance on 02-Jul-2014***

SYMPTOMS

Intermittent HTTP error 500 is got on client side. Below stack trace is seen which shows the sockets getting read timeout:

Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at weblogic.servlet.internal.PostInputStream.read(PostInputStream.java:177)
at weblogic.servlet.internal.ServletInputStreamImpl.read(ServletInputStreamImpl.java:228)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:250)
at com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:133)
at com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:531)

CHANGES

Intermittent HTTP error 500 during QA/load tests

CAUSE

Customer code(EJB/webservice) was doing remote socket communication which eventually failed with timeout. That caused HTTP 500 returned to client.

Socket Read Timeout Exception comes from the OS socket when a request is made to an external resource and a thread is allocated to the request synchronous call it keeps the Socket open and waits for the response from the external resource e.g from the server which is serving EJB response to the web services call is that response take longer time than the default read timeout then socket me return the exception.

When we do not manually configure a timeout for each method call involving networking, it will have a potential blocking behavior by the underlying physical socket read/connect characteristic. While waiting infinitely for the response from the other side, the native OS networking layer probably throws an I/O exception.

SOLUTION

Specify the -Dsun.net.client.defaultReadTimeout=30000 (or specify a value as per the response behavior of the remote application) on the WebLogic side so that the Socket which has been opened at weblogic end while an call for the external resource been made.This the socket which is timing out while reading the response from the remote end.So the issue is lying on the remote end which have to respond back to the Weblogic and write the response back to the socket so that that can be passed back by the web services to the caller.

the delay in response time could be due to may factors e.g:-

--Under heavy load high usage of CPU causing the I/O operations to get delayed
--The remote side has a large data to be written back to the socket which is taking more time (As it is intermittent it could be that it is happening for some particular request-response sets and not for all)

the latency added by the Responding remote side is causing this socket timeout in Weblogic End and we are trying to increase the time of the socket timeout so that the socket should not timeout while the response is still not completely written.