JVM最多能创建多少个线程: unable to create new native thread

转载自：http://www.rigongyizu.com/jvm-max-threads/

有应用报出这样的异常“java.lang.OutOfMemoryError: unable to create new native thread”。甚至机器上执行shell命令也会报”-bash: fork: Resource temporarily unavailable”异常。机器上的其他应用如hadoop也会受影响：

12013-08-21 20:15:48,496 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread

2java.lang.OutOfMemoryError: unable to create new native thread

3        at java.lang.Thread.start0(Native Method)

4        at java.lang.Thread.start(Thread.java:640)

5        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.

6ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:524)

7        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.

8ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:456)

9        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:128)

10        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)

11        at java.lang.Thread.run(Thread.java:662)

122013-08-21 20:15:48,497 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..

一看以为内存不够导致无法创建新的线程，但是观察机器上的内存还有空闲，猜测是哪个地方对线程创建有限制。

首先需要排除操作系统对线程创建数的限制，参考：《JVM中可生成的最大Thread数量》一文，设置操作系统可以支持创建10万个线程：

1echo "100000" > /proc/sys/kernel/threads-max

2echo "100000" > /proc/sys/kernel/pid_max     （默认32768）

3echo "200000" > /proc/sys/vm/max_map_count   （默认65530）

4ulimit -u unlimited   (设置max user processes的值)

当前测试环境为：

1[[e-mail] ~]$ uname -a

2Linux bufer108081.tbc 2.6.32-220.23.2.ali927.el5.x86_64 #1 SMP Mon Jan 28 14:57:06 CST 2013 x86_64 x86_64 x86_64 GNU/Linux

3[[e-mail] ~]$ cat /etc/redhat-release

4Red Hat Enterprise Linux Server release 5.7 (Tikanga)

5[[e-mail] ~]$ java -version

6java version "1.7.0_51"

7Java(TM) SE Runtime Environment (build 1.7.0_51-b13)

8OpenJDK (Alibaba) 64-Bit Server VM (build 24.45-b08-internal, mixed mode)

9[[e-mail] ~]$ ulimit -a

10core file size          (blocks, -c) 0

11data seg size           (kbytes, -d) unlimited

12scheduling priority             (-e) 0

13file size               (blocks, -f) unlimited

14pending signals                 (-i) 387068

15max locked memory       (kbytes, -l) 64

16max memory size         (kbytes, -m) unlimited

17open files                      (-n) 131072

18pipe size            (512 bytes, -p) 8

19POSIX message queues     (bytes, -q) 819200

20real-time priority              (-r) 0

21stack size              (kbytes, -s) 10240

22cpu time               (seconds, -t) unlimited

23max user processes              (-u) unlimited

24virtual memory          (kbytes, -v) unlimited

25file locks                      (-x) unlimited

26[[e-mail] ~/dev/baoniu]$ free -g

27             total       used       free     shared    buffers     cached

28Mem:            47         31         15          0          3         25

29-/+ buffers/cache:          3         44

30Swap:            0          0          0

测试程序见本文最后面。测试结果：突破了网上所说的32000个线程数，成功创建了 10万个线程。
（由于/proc/sys/kernel/pid_max默认为32768，所以网上很多测试程序测试JVM只能创建32000个线程。）

1[[e-mail] ~/dev/baoniu]$ java -Xss128k MaxThreadsMain

2The stack size specified is too small, Specify at least 228k

3Error: Could not create the Java Virtual Machine.

4Error: A fatal exception has occurred. Program will exit.

5[[e-mail] ~/dev/baoniu]$ java -Xss228k MaxThreadsMain

64,000 threads: Time to create 4,000 threads was 0.846 seconds

78,000 threads: Time to create 4,000 threads was 2.425 seconds

812,000 threads: Time to create 4,000 threads was 4.813 seconds

916,000 threads: Time to create 4,000 threads was 7.229 seconds

1020,000 threads: Time to create 4,000 threads was 10.443 seconds

1124,000 threads: Time to create 4,000 threads was 14.480 seconds

1228,000 threads: Time to create 4,000 threads was 19.709 seconds

1332,000 threads: Time to create 4,000 threads was 24.742 seconds

1436,000 threads: Time to create 4,000 threads was 31.181 seconds

1540,000 threads: Time to create 4,000 threads was 36.629 seconds

1644,000 threads: Time to create 4,000 threads was 42.796 seconds

1748,000 threads: Time to create 4,000 threads was 48.659 seconds

1852,000 threads: Time to create 4,000 threads was 55.030 seconds

1956,000 threads: Time to create 4,000 threads was 60.130 seconds

2060,000 threads: Time to create 4,000 threads was 67.419 seconds

2164,000 threads: Time to create 4,000 threads was 73.507 seconds

2268,000 threads: Time to create 4,000 threads was 79.416 seconds

2372,000 threads: Time to create 4,000 threads was 85.261 seconds

2476,000 threads: Time to create 4,000 threads was 92.201 seconds

2580,000 threads: Time to create 4,000 threads was 98.087 seconds

2684,000 threads: Time to create 4,000 threads was 108.263 seconds

2788,000 threads: Time to create 4,000 threads was 114.840 seconds

2892,000 threads: Time to create 4,000 threads was 121.841 seconds

2996,000 threads: Time to create 4,000 threads was 127.714 seconds

30After creating 99,410 threads, java.lang.OutOfMemoryError: unable to create new native thread

31        at java.lang.Thread.start0(Native Method)

32        at java.lang.Thread.start(Thread.java:713)

33        at MaxThreadsMain.addThread(MaxThreadsMain.java:43)

34        at MaxThreadsMain.main(MaxThreadsMain.java:13)

创建9W多个线程后，进程占用内存：VIRT=40.5g RES=4.7g，用free -g查看系统还有9G的空闲(free)内存。

---

JVM最多能启动的线程数参照公式：

1(MaxProcessMemory - JVMMemory – ReservedOsMemory) / (ThreadStackSize) = Number of threads

• MaxProcessMemory : 进程的最大寻址空间
• JVMMemory : JVM内存
• ReservedOsMemory : 保留的操作系统内存，如Native heap，JNI之类，一般100多M
• ThreadStackSize : 线程栈的大小，jvm启动时由Xss指定

MaxProcessMemory：如32位的linux默认每个进程最多申请3G的地址空间，64位的操作系统可以支持到46位（64TB）的物理地址空间和47位（128T）的进程虚拟地址空间（linux 64位CPU内存限制）。

JVM内存：由Heap区和Perm区组成。通过-Xms和-Xmx可以指定heap区大小，通过-XX:PermSize和-XX:MaxPermSize指定perm区的大小(默认从32MB 到64MB，和JVM版本有关)。

线程栈ThreadStackSize：

Java程序中，每个线程都有自己的Stack Space。这个Stack Space的空间是独立分配的，与-Xmx和-Xms指定的堆大小无关。Stack Space用来做方法的递归调用时压入Stack Frame。所以当递归调用太深的时候，就有可能耗尽Stack Space，爆出StackOverflow的错误。对于32位JVM，缺省值为256KB，对于64位JVM，缺省值为512KB。最大值根据平台和特定机器配置的不同而不同。如果超过最大值，那么将报告java/lang/OutOfMemoryError消息。

可见，减少Xss指定的线程栈大小能够启动更多的线程，但是线程总数也受到系统空闲内存和操作系统的限制。

总结下影响Java线程数量的因素：

• Java虚拟机本身：-Xms，-Xmx，-Xss；
• 系统限制：
  /proc/sys/kernel/pid_max，
  /proc/sys/kernel/thread-max，
  max_user_process（ulimit -u），
  /proc/sys/vm/max_map_count。

ps: 最后发现是这台机器上有个应用代码问题创建了过多的线程，达到系统限制，而影响了YARN和其他应用。一般来说，单机线程数过多可以考虑使用线程池或者更多的服务器。

---

附测试程序：

1import java.util.ArrayList;

2import java.util.List;

3 

4public class MaxThreadsMain {

5 

6  public static final int BATCH_SIZE = 4000;

7 

8  public static void main(String... args) throws InterruptedException {

9    List<Thread> threads = new ArrayList<Thread>();

10    try {

11      for (int i = 0; i <= 100 * 1000; i += BATCH_SIZE) {

12        long start = System.currentTimeMillis();

13        addThread(threads, BATCH_SIZE);

14        long end = System.currentTimeMillis();

15        Thread.sleep(1000);

16        long delay = end - start;

17        System.out.printf("%,d threads: Time to create %,d threads was %.3f seconds %n", threads.size(), BATCH_SIZE, delay / 1e3);

18      }

19    } catch (Throwable e) {

20      System.err.printf("After creating %,d threads, ", threads.size());

21      e.printStackTrace();

22    }

23 

24  }

25 

26  private static void addThread(List<Thread> threads, int num) {

27    for (int i = 0; i < num; i++) {

28      Thread t = new Thread(new Runnable() {

29        @Override

30        public void run() {

31          try {

32            while (!Thread.interrupted()) {

33              Thread.sleep(1000);

34            }

35          } catch (InterruptedException ignored) {

36            //

37          }

38        }

39      });

40      t.setDaemon(true);

41      t.setPriority(Thread.MIN_PRIORITY);

42      threads.add(t);

43      t.start();

44    }

45  }

46}

 

附：

MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html  $ Ulimit -U  1024It Explains 1000 Sessions Limitation ;-)Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:  # Cat /Etc/Security/Limits.Conf  Mysql Soft Nofile 10240  Mysql Hard    Nofile 40960  Mysql Soft Nproc 10240  Mysql Hard    Nproc 40960Fixed My Issue! :-))