glassfish:Resource temporarily unavailable in tsStartJavaThread

         今天测试服务器(用的glassfish3.1.1)在部署应用的时候忽然报出如下异常。随后使用用su和ps都会报出“-bash: fork: retry: Resource temporarily unavailable”。

[ERROR] [ERROR][thread ] Could not start thread Keep-Alive-Timer. Resource temporarily unavailable[ERROR] Exception in thread "Main Thread" java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).[ERROR] Java heap 3G reserved, 64M committed[ERROR] Paged memory=18014398505034060K/18389412K.[ERROR] Your Java heap size might be set too high.[ERROR] Try to reduce the Java heap size using -Xmx:<size> (e.g. "[ERROR]         at java.lang.Thread.start0(Native Method)[ERROR]         at java.lang.Thread.start(Thread.java:597)[ERROR]         at sun.net.www.http.KeepAliveCache$1.run(KeepAliveCache.java:89)[ERROR]         at sun.net.www.http.KeepAliveCache.put(KeepAliveCache.java:75)[ERROR]         at sun.net.www.http.HttpClient.putInKeepAliveCache(HttpClient.java:370)[ERROR]         at sun.net.www.http.HttpClient.finished(HttpClient.java:358)[ERROR]         at sun.net.www.http.ChunkedInputStream.closeUnderlying(ChunkedInputStream.java:196)[ERROR]         at sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:428)[ERROR]         at sun.net.www.http.ChunkedInputStream.readAheadNonBlocking(ChunkedInputStream.java:493)[ERROR]         at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:584)[ERROR]         at sun.net.www.http.ChunkedInputStream.available(ChunkedInputStream.java:698)[ERROR]         at java.io.FilterInputStream.available(FilterInputStream.java:142)[ERROR]         at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:196)[ERROR]         at com.sun.enterprise.util.io.FileUtils.copy(FileUtils.java:915)[ERROR]         at com.sun.enterprise.admin.remote.RemoteResponseManager.<init>(RemoteResponseManager.java:79)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.handleResponse(RemoteAdminCommand.java:1056)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.handleResponse(RemoteAdminCommand.java:1045)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.access$600(RemoteAdminCommand.java:101)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand$1$1.handleReport(RemoteAdminCommand.java:526)[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.processReport(PayloadFilesManager.java:473)[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.access$900(PayloadFilesManager.java:93)[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager$DataRequestType$4.processPart(PayloadFilesManager.java:776)[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.processPartsExtended(PayloadFilesManager.java:624)[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.processParts(PayloadFilesManager.java:643)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand$1.useConnection(RemoteAdminCommand.java:531)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.doHttpCommand(RemoteAdminCommand.java:700)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.doHttpCommand(RemoteAdminCommand.java:543)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.executeRemoteCommand(RemoteAdminCommand.java:482)[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.executeCommand(RemoteAdminCommand.java:419)[ERROR]         at com.sun.enterprise.admin.cli.remote.RemoteCommand.executeCommand(RemoteCommand.java:335)[ERROR]         at com.sun.enterprise.admin.cli.CLICommand.execute(CLICommand.java:264)[ERROR]         at com.sun.enterprise.admin.cli.AsadminMain.executeCommand(AsadminMain.java:306)[ERROR]         at com.sun.enterprise.admin.cli.AsadminMain.main(AsadminMain.java:238)[ERROR] [ERROR][thread ] Could not start thread Thread-0. Resource temporarily unavailable[ERROR] Deployment of /home/javadev/codes/*****************.war failed.[ERROR] For more detail on what might be causing the problem try running maven with the --debug option [ERROR] or setting the maven-glassfish-plugin "echo" property to "true".

         我们先分析了一下异常，看到“Paged memory=18014398505034060K/18389412K.”立马就吓到了，以为谁无聊又在改配置信息呢。在glassfish控制台查看glassfish启动JVM的配置参数，发现没什么异常，对比了其他配置都没有改变。相同的配置在其他测试服务器都能正常运行，初步排除了glassfish和JVM参数配置的问题。
         随后对“-bash: fork: retry: Resource temporarily unavailable”进行分析，查看进程数:

 [javadev@snaqi-test3 ~]$ ps -ef|wc -l 1025

         但在/etc/security/limits.conf用户javadev的配置

    javadev             soft    nproc   1024    javadev             hard    nproc   16384    javadev             soft    nofile  2048    javadev             hard    nofile  65536

        显然用户进程数已经达到限制的极限，将第1行的1024改成2048，问题解决。另外，还可以将第3行2048改大，以防程序出现“too many file opens”的异常。

参考：oracle，ulimit，通过 ulimit 改善系统性能