.net线程池内幕

http://group.jobbole.com/27467/

本文通过对ThreadPool源码的分析讲解揭示.net线程池的内幕，并总结ThreadPool设计的好与不足。

 

线程池的作用

 

线程池，顾名思义，线程对象池。Task和TPL都有用到线程池，所以了解线程池的内幕有助于你写出更好的程序。由于篇幅有限，在这里我只讲解以下核心概念：

 

• 线程池的大小
• 如何调用线程池添加任务
• 线程池如何执行任务

Threadpool也支持操控IOCP的线程，但在这里我们不研究它，和task以及TPL相关的会在其他博客中详解。

 

线程池的大小

不管什么池，总有尺寸，ThreadPool也不例外。ThreadPool提供了4个方法来调整线程池的大小：

• SetMaxThreads
• GetMaxThreads
• SetMinThreads
• GetMinThreads

SetMaxThreads指定线程池最多可以有多少个线程，而GetMaxThreads自然就是获取这个值。SetMinThreads指定线程池中最少存活的线程的数量，而GetMinThreads就是获取这个值。

为何要设置一个最大数量和有一个最小数量呢？原来线程池的大小取决于若干因素，如虚拟地址空间的大小等。比如你的计算机是4g内存，而一个线程的初始堆栈大小为1m，那么你最多能创建4g/1m的线程（忽略操作系统本身以及其他进程内存分配）；正因为线程有内存开销，所以如果线程池的线程过多而又没有被完全使用，那么这就是对内存的一种浪费，所以限制线程池的最大数是很make sense的。

那么最小数又是为啥？线程池就是线程的对象池，对象池的最大的用处是重用对象。为啥要重用线程，因为线程的创建与销毁都要占用大量的cpu时间。所以在高并发状态下，线程池由于无需创建销毁线程节约了大量时间，性能也远远高于多线程。最小数可以让你调整最小的存活线程数量来应对高并发。

 

如何调用线程池添加任务

线程池主要提供了2个方法来调用：QueueUserWorkItem和UnsafeQueueUserWorkItem。

两个方法的代码基本一致，除了attribute不同，QueueUserWorkItem可以被partial trust的代码调用，而UnsafeQueueUserWorkItem只能被full trust的代码调用。

接着看看源代码：

 

C#

public static bool QueueUserWorkItem(WaitCallback callBack){ StackCrawlMark stackMark = StackCrawlMark.LookForMyCaller; return ThreadPool.QueueUserWorkItemHelper(callBack, (object) null, ref stackMark, true);}

1

2

3

4

5

publicstaticboolQueueUserWorkItem(WaitCallbackcallBack)

{

    StackCrawlMarkstackMark=StackCrawlMark.LookForMyCaller;

    returnThreadPool.QueueUserWorkItemHelper(callBack,(object)null,refstackMark,true);

}

 

QueueUserWorkItemHelper首先调用ThreadPool.EnsureVMInitialized()来确保CLR虚拟机初始化（VM是一个统称，不是单指java虚拟机，也可以指CLR的execution engine），紧接着实例化ThreadPoolWorkQueue，最后调用ThreadPoolWorkQueue的Enqueue方法并传入callback和true。

 

C#

[SecurityCritical]public void Enqueue(IThreadPoolWorkItem callback, bool forceGlobal){ ThreadPoolWorkQueueThreadLocals queueThreadLocals = (ThreadPoolWorkQueueThreadLocals) null; if (!forceGlobal) queueThreadLocals = ThreadPoolWorkQueueThreadLocals.threadLocals; if (this.loggingEnabled) FrameworkEventSource.Log.ThreadPoolEnqueueWorkObject((object) callback); if (queueThreadLocals != null) { queueThreadLocals.workStealingQueue.LocalPush(callback); } else { ThreadPoolWorkQueue.QueueSegment comparand = this.queueHead; while (!comparand.TryEnqueue(callback)) { Interlocked.CompareExchange<ThreadPoolWorkQueue.QueueSegment>(ref comparand.Next, new ThreadPoolWorkQueue.QueueSegment(), (ThreadPoolWorkQueue.QueueSegment) null); for (; comparand.Next != null; comparand = this.queueHead) Interlocked.CompareExchange<ThreadPoolWorkQueue.QueueSegment>(ref this.queueHead, comparand.Next, comparand); } } this.EnsureThreadRequested();}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

[SecurityCritical]

public voidEnqueue(IThreadPoolWorkItemcallback,boolforceGlobal)

{

    ThreadPoolWorkQueueThreadLocalsqueueThreadLocals= (ThreadPoolWorkQueueThreadLocals)null;

    if(!forceGlobal)

    queueThreadLocals=ThreadPoolWorkQueueThreadLocals.threadLocals;

    if(this.loggingEnabled)

    FrameworkEventSource.Log.ThreadPoolEnqueueWorkObject((object)callback);

    if(queueThreadLocals!=null)

    {

        queueThreadLocals.workStealingQueue.LocalPush(callback);

    }

    else

    {

        ThreadPoolWorkQueue.QueueSegmentcomparand=this.queueHead;

        while(!comparand.TryEnqueue(callback))

        {

            Interlocked.CompareExchange<ThreadPoolWorkQueue.QueueSegment>(refcomparand.Next,newThreadPoolWorkQueue.QueueSegment(),(ThreadPoolWorkQueue.QueueSegment)null);

            for(;comparand.Next!=null;comparand=this.queueHead)

            Interlocked.CompareExchange<ThreadPoolWorkQueue.QueueSegment>(refthis.queueHead,comparand.Next,comparand);

        }

    }

    this.EnsureThreadRequested();

}

 

ThreadPoolWorkQueue主要包含2个“queue”(实际是数组)，一个为QueueSegment（global work queue），另一个是WorkStealingQueue(local work queue)。两者具体的区别会在Task/TPL里讲解，这里暂不解释。

由于forceGlobal是true，所以执行到了comparand.TryEnqueue(callback)，也就是QueueSegment.TryEnqueue。comparand先从队列的头(queueHead)开始enqueue，如果不行就继续往下enqueue，成功后再赋值给queueHead。

让我们来看看QueueSegment的源代码：

 

C#

public QueueSegment(){ this.nodes = new IThreadPoolWorkItem[256];}public bool TryEnqueue(IThreadPoolWorkItem node){ int upper; int lower; this.GetIndexes(out upper, out lower); while (upper != this.nodes.Length) { if (this.CompareExchangeIndexes(ref upper, upper + 1, ref lower, lower)) { Volatile.Write<IThreadPoolWorkItem>(ref this.nodes[upper], node); return true; } } return false;}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

publicQueueSegment()

{

    this.nodes=newIThreadPoolWorkItem[256];

}

 

public boolTryEnqueue(IThreadPoolWorkItemnode)

{

    intupper;

    intlower;

    this.GetIndexes(outupper,outlower);

    while(upper!=this.nodes.Length)

    {

        if(this.CompareExchangeIndexes(refupper,upper+1,reflower,lower))

        {

            Volatile.Write<IThreadPoolWorkItem>(refthis.nodes[upper],node);

            returntrue;

        }

    }

    returnfalse;

}

 

这个所谓的global work queue实际上是一个IThreadPoolWorkItem的数组，而且限死256，这是为啥？难道是因为和IIS线程池(也只有256个线程）对齐？使用interlock和内存写屏障volatile.write来保证nodes的正确性，比起同步锁性能有很大的提高。最后调用EnsureThreadRequested，EnsureThreadRequested会调用QCall把请求发送至CLR，由CLR调度ThreadPool。

 

线程池如何执行任务

线程被调度后通过ThreadPoolWorkQueue的Dispatch方法来执行。

C#

internal static bool Dispatch(){ThreadPoolWorkQueue threadPoolWorkQueue = ThreadPoolGlobals.workQueue;int tickCount = Environment.TickCount;threadPoolWorkQueue.MarkThreadRequestSatisfied();threadPoolWorkQueue.loggingEnabled = FrameworkEventSource.Log.IsEnabled(EventLevel.Verbose, (EventKeywords) 18);bool flag1 = true;IThreadPoolWorkItem callback = (IThreadPoolWorkItem) null;try{ThreadPoolWorkQueueThreadLocals tl = threadPoolWorkQueue.EnsureCurrentThreadHasQueue();while ((long) (Environment.TickCount - tickCount) < (long) ThreadPoolGlobals.tpQuantum){try{}finally{bool missedSteal = false;threadPoolWorkQueue.Dequeue(tl, out callback, out missedSteal);if (callback == null)flag1 = missedSteal;elsethreadPoolWorkQueue.EnsureThreadRequested();}if (callback == null)return true;if (threadPoolWorkQueue.loggingEnabled)FrameworkEventSource.Log.ThreadPoolDequeueWorkObject((object) callback);if (ThreadPoolGlobals.enableWorkerTracking){bool flag2 = false;try{try{}finally{ThreadPool.ReportThreadStatus(true);flag2 = true;}callback.ExecuteWorkItem();callback = (IThreadPoolWorkItem) null;}finally{if (flag2)ThreadPool.ReportThreadStatus(false);}}else{callback.ExecuteWorkItem();callback = (IThreadPoolWorkItem) null;}if (!ThreadPool.NotifyWorkItemComplete())return false;}return true;}catch (ThreadAbortException ex){if (callback != null)callback.MarkAborted(ex);flag1 = false;}finally{if (flag1)threadPoolWorkQueue.EnsureThreadRequested();}return true;}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

internalstaticboolDispatch()

{

ThreadPoolWorkQueuethreadPoolWorkQueue= ThreadPoolGlobals.workQueue;

int tickCount=Environment.TickCount;

threadPoolWorkQueue.MarkThreadRequestSatisfied();

threadPoolWorkQueue.loggingEnabled=FrameworkEventSource.Log.IsEnabled(EventLevel.Verbose,(EventKeywords)18);

boolflag1=true;

IThreadPoolWorkItem callback=(IThreadPoolWorkItem)null;

try

{

ThreadPoolWorkQueueThreadLocalstl=threadPoolWorkQueue.EnsureCurrentThreadHasQueue();

while ((long)(Environment.TickCount-tickCount)<(long)ThreadPoolGlobals.tpQuantum)

{

try

{

}

finally

{

boolmissedSteal=false;

threadPoolWorkQueue.Dequeue(tl,outcallback,outmissedSteal);

if(callback==null)

flag1 =missedSteal;

else

threadPoolWorkQueue.EnsureThreadRequested();

}

if (callback==null)

returntrue;

if (threadPoolWorkQueue.loggingEnabled)

FrameworkEventSource.Log.ThreadPoolDequeueWorkObject((object)callback);

if (ThreadPoolGlobals.enableWorkerTracking)

{

bool flag2=false;

try

{

try

{

}

finally

{

ThreadPool.ReportThreadStatus(true);

flag2=true;

}

callback.ExecuteWorkItem();

callback =(IThreadPoolWorkItem)null;

}

finally

{

if (flag2)

ThreadPool.ReportThreadStatus(false);

}

}

else

{

callback.ExecuteWorkItem();

callback=(IThreadPoolWorkItem)null;

}

if(!ThreadPool.NotifyWorkItemComplete())

return false;

}

return true;

}

catch (ThreadAbortExceptionex)

{

if (callback!=null)

callback.MarkAborted(ex);

flag1 =false;

}

finally

{

if (flag1)

threadPoolWorkQueue.EnsureThreadRequested();

}

returntrue;

}

 

while语句判断如果执行时间少于30ms会不断继续执行下一个callback。这是因为大多数机器线程切换大概在30ms，如果该线程只执行了不到30ms就在等待中断线程切换那就太浪费CPU了，浪费可耻啊！

Dequeue负责找到需要执行的callback：

 

C#

public void Dequeue(ThreadPoolWorkQueueThreadLocals tl, out IThreadPoolWorkItem callback, out bool missedSteal){callback = (IThreadPoolWorkItem) null;missedSteal = false;ThreadPoolWorkQueue.WorkStealingQueue workStealingQueue1 = tl.workStealingQueue;workStealingQueue1.LocalPop(out callback);if (callback == null){for (ThreadPoolWorkQueue.QueueSegment comparand = this.queueTail; !comparand.TryDequeue(out callback) && comparand.Next != null && comparand.IsUsedUp(); comparand = this.queueTail)Interlocked.CompareExchange<ThreadPoolWorkQueue.QueueSegment>(ref this.queueTail, comparand.Next, comparand);}if (callback != null)return;ThreadPoolWorkQueue.WorkStealingQueue[] current = ThreadPoolWorkQueue.allThreadQueues.Current;int num = tl.random.Next(current.Length);for (int length = current.Length; length > 0; --length){ThreadPoolWorkQueue.WorkStealingQueue workStealingQueue2 = Volatile.Read<ThreadPoolWorkQueue.WorkStealingQueue>(ref current[num % current.Length]);if (workStealingQueue2 != null && workStealingQueue2 != workStealingQueue1 && workStealingQueue2.TrySteal(out callback, ref missedSteal))break;++num;}}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

publicvoidDequeue(ThreadPoolWorkQueueThreadLocalstl,outIThreadPoolWorkItemcallback,outboolmissedSteal)

{

callback=(IThreadPoolWorkItem)null;

missedSteal =false;

ThreadPoolWorkQueue.WorkStealingQueueworkStealingQueue1= tl.workStealingQueue;

workStealingQueue1.LocalPop(outcallback);

if(callback==null)

{

for(ThreadPoolWorkQueue.QueueSegmentcomparand=this.queueTail;!comparand.TryDequeue(outcallback)&&comparand.Next!=null&&comparand.IsUsedUp();comparand=this.queueTail)

Interlocked.CompareExchange<ThreadPoolWorkQueue.QueueSegment>(refthis.queueTail,comparand.Next,comparand);

}

if (callback!=null)

return;

ThreadPoolWorkQueue.WorkStealingQueue[]current=ThreadPoolWorkQueue.allThreadQueues.Current;

intnum=tl.random.Next(current.Length);

for (intlength=current.Length;length>0;--length)

{

ThreadPoolWorkQueue.WorkStealingQueueworkStealingQueue2= Volatile.Read<ThreadPoolWorkQueue.WorkStealingQueue>(refcurrent[num%current.Length]);

if(workStealingQueue2!=null&&workStealingQueue2!=workStealingQueue1&&workStealingQueue2.TrySteal(outcallback,refmissedSteal))

break;

++num;

}

}

因为我们把callback添加到了global work queue，所以这里的local work queue(workStealingQueue)找不到callback，所以只能在global work queue查找，local work queue会在task里讲解。先从global work queue的起始位置查找，直到尾部。所以global work quque里的callback是FIFO的执行顺序。

 

C#

public bool TryDequeue(out IThreadPoolWorkItem node){int upper;int lower;this.GetIndexes(out upper, out lower);while (lower != upper){// ISSUE: explicit reference operation// ISSUE: variable of a reference typeint& prevUpper = @upper;// ISSUE: explicit reference operationint newUpper = ^prevUpper;// ISSUE: explicit reference operation// ISSUE: variable of a reference typeint& prevLower = @lower;// ISSUE: explicit reference operationint newLower = ^prevLower + 1;if (this.CompareExchangeIndexes(prevUpper, newUpper, prevLower, newLower)){SpinWait spinWait = new SpinWait();while ((node = Volatile.Read<IThreadPoolWorkItem>(ref this.nodes[lower])) == null)spinWait.SpinOnce();this.nodes[lower] = (IThreadPoolWorkItem) null;return true;}}node = (IThreadPoolWorkItem) null;return false;}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

publicboolTryDequeue(outIThreadPoolWorkItemnode)

{

intupper;

int lower;

this.GetIndexes(outupper,outlower);

while (lower!=upper)

{

// ISSUE: explicit reference operation

// ISSUE: variable of a reference type

int&prevUpper=@upper;

// ISSUE: explicit reference operation

int newUpper=^prevUpper;

// ISSUE: explicit reference operation

// ISSUE: variable of a reference type

int&prevLower=@lower;

// ISSUE: explicit reference operation

intnewLower=^prevLower+1;

if (this.CompareExchangeIndexes(prevUpper,newUpper,prevLower,newLower))

{

SpinWait spinWait=newSpinWait();

while((node=Volatile.Read<IThreadPoolWorkItem>(refthis.nodes[lower]))==null)

spinWait.SpinOnce();

this.nodes[lower]=(IThreadPoolWorkItem)null;

return true;

}

}

node=(IThreadPoolWorkItem)null;

return false;

}

使用自旋锁和内存读屏障来避免内核态和用户态的切换，提高了获取callback的性能。如果还是没有callback，那么就从所有的local work queue里随机选取一个，然后在该local work queue里“偷取”一个任务(callback)。

拿到callback后执行callback.ExecuteWorkItem()，通知完成。

 

总结

ThreadPool提供了方法调整线程池最少活跃的线程来应对不同的并发场景。ThreadPool带有2个work queue，一个golbal一个local。执行时先从local找任务，接着去global，最后才会去随机选取一个local偷一个任务，其中global是FIFO的执行顺序。Work queue实际上是数组，使用了大量的自旋锁和内存屏障来提高性能。但是在偷取任务上，是否可以考虑得更多，随机选择一个local太随意。首先要考虑偷取的队列上必须有可执行任务；其次可以选取一个不在调度中的线程的local work queue，这样降低了自旋锁的可能性，加快了偷取的速度；最后，偷取的时候可以考虑像golang一样偷取别人queue里一半的任务，因为执行完偷到的这一个任务之后，下次该线程再次被调度到还是可能没任务可执行，还得去偷取别人的任务，这样既浪费CPU时间，又让任务在线程上分布不均匀，降低了系统吞吐量！

另外，如果禁用log和ETW trace，可以使ThreadPool的性能更进一步