Weblogic OutOfMemory exception的误解 -- thread limitation

不是所有的OutofMemory exception都是内存问题。。。前几天有个客户的site报了以下错误：

[ERROR][thread ] Could not start thread Timer-72025. Resource temporarily unavailableException in thread "Timer-72024" java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1097).Attempting to allocate 2G bytesThere is insufficient native memory for the JavaRuntime Environment to continue.Possible reasons:  The system is out of physical RAM or swap space  In 32 bit mode, the process size limit was hitPossible solutions:  Reduce memory load on the system  Increase physical memory or swap space  Check if swap backing store is full  Use 64 bit Java on a 64 bit OS  Decrease Java heap size (-Xmx/-Xms)  Decrease number of Java threads  Decrease Java thread stack sizes (-Xss)  Disable compressed references (-XXcompressedRefs=false)                at java.lang.Thread.start0(Native Method)                at java.lang.Thread.start(Thread.java:640)                at java.util.Timer.<init>(Timer.java:137)                at java.util.Timer.<init>(Timer.java:106)                at ......                at java.util.TimerThread.mainLoop(Timer.java:512)                at java.util.TimerThread.run(Timer.java:462)

第一反应以为是内存不足，于是dump了heap回来看，发现系统只占用了80M的内存，远远系统内存绰绰有余，那么为啥会出现这种情况呢？查看了出错的代码，发现代码启动了很多的线程，遂想起以前遇到过类似的问题，weblogic里面启动的线程太多，超出了linux的最大限制，也是会出这个看起来内存泄漏的日志。查看系统的限制：

sc-1@vECE-222 limits.d # ulimit -acore file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size               (blocks, -f) unlimitedpending signals                 (-i) 38610max locked memory       (kbytes, -l) 64max memory size         (kbytes, -m) unlimitedopen files                      (-n) 8192pipe size            (512 bytes, -p) 8POSIX message queues     (bytes, -q) 819200real-time priority              (-r) 0stack size              (kbytes, -s) 10240cpu time               (seconds, -t) unlimitedmax user processes              (-u) 1024virtual memory          (kbytes, -v) unlimitedfile locks                      (-x) unlimited

发现最大的进程数是1024，注意：linux里面计算线程和进程是用一样计算的，这里说的max user processes是包括线程的。

使用以下代码重现问题：

public class ThreadExample {    public static void main(String[] args){        System.out.println(Thread.currentThread().getName());        for(int i=0; i< 2048; i++){            new Thread("" + i){                public void run(){                    System.out.println(Thread.currentThread().getName() + " running");                    try {                        Thread.sleep(60000);                    } catch (InterruptedException e) {                        e.printStackTrace();                    }                    System.out.println(Thread.currentThread().getName() + " ending");                }            }.start();        }    }}

执行和输出：

sc-1@ECE30CP3_Z1 etmdaha # su -p occas -c "/opt/jrockit-jdk/bin/java ThreadExample"Main Thread0 running5 running//836 running[ERROR][thread ] Could not start thread 837. Resource temporarily unavailableException in thread "Main Thread" java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).Attempting to allocate 3G bytesThere is insufficient native memory for the JavaRuntime Environment to continue.Possible reasons:  The system is out of physical RAM or swap space  In 32 bit mode, the process size limit was hitPossible solutions:  <strong>Reduce memory load on the system  Increase physical memory or swap space  Check if swap backing store is full  Use 64 bit Java on a 64 bit OS  Decrease Java heap size (-Xmx/-Xms)  Decrease number of Java threads  Decrease Java thread stack sizes (-Xss)  Disable compressed references (-XXcompressedRefs=false)</strong>        at java.lang.Thread.start0(Native Method)        at java.lang.Thread.start(Thread.java:640)        at ThreadExample.main(ThreadExample.java:6)

linux系统资源有各种限制：内存限制，栈限制，文件数目限制，线程限制，要小心陷进这些问题。

# ulimit -acore file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size               (blocks, -f) unlimitedpending signals                 (-i) 79010max locked memory       (kbytes, -l) 64max memory size         (kbytes, -m) unlimitedopen files                      (-n) 1024pipe size            (512 bytes, -p) 8POSIX message queues     (bytes, -q) 819200real-time priority              (-r) 0stack size              (kbytes, -s) 10240cpu time               (seconds, -t) unlimitedmax user processes              (-u) 1024virtual memory          (kbytes, -v) unlimitedfile locks                      (-x) unlimited

这里有两篇很好的文章：

http://people.redhat.com/alikins/system_tuning.html#threads

http://kavassalis.com/2011/03/linux-and-the-maximum-number-of-processes-threads/