tc malloc

Overview

TCMalloc assigns each thread a thread-local cache. Small allocationsare satisfied from the thread-local cache. Objects are moved fromcentral data structures into a thread-local cache as needed, andperiodic garbage collections are used to migrate memory back from athread-local cache into the central data structures.

TCMalloc treates objects with size <= 32K ("small" objects)differently from larger objects. Large objects are allocateddirectly from the central heap using a page-level allocator(a page is a 4K aligned region of memory). I.e., a large objectis always page-aligned and occupies an integral number of pages.

A run of pages can be carved up into a sequence of small objects, eachequally sized. For example a run of one page (4K) can be carved upinto 32 objects of size 128 bytes each.

Small Object Allocation

Each small object size maps to one of approximately 170 allocatablesize-classes. For example, all allocations in the range 961 to 1024bytes are rounded up to 1024. The size-classes are spaced so thatsmall sizes are separated by 8 bytes, larger sizes by 16 bytes, evenlarger sizes by 32 bytes, and so forth. The maximal spacing (for sizes>= ~2K) is 256 bytes.

A thread cache contains a singly linked list of free objects per size-class.

When allocating a small object: (1) We map its size to thecorresponding size-class. (2) Look in the corresponding free list inthe thread cache for the current thread. (3) If the free list is notempty, we remove the first object from the list and return it. Whenfollowing this fast path, TCMalloc acquires no locks at all. Thishelps speed-up allocation significantly because a lock/unlock pairtakes approximately 100 nanoseconds on a 2.8 GHz Xeon.

If the free list is empty: (1) We fetch a bunch of objects from acentral free list for this size-class (the central free list is sharedby all threads). (2) Place them in the thread-local free list. (3)Return one of the newly fetched objects to the applications.

If the central free list is also empty: (1) We allocate a run of pagesfrom the central page allocator. (2) Split the run into a set ofobjects of this size-class. (3) Place the new objects on the centralfree list. (4) As before, move some of these objects to thethread-local free list.

Large Object Allocation

A large object size (> 32K) is rounded up to a page size (4K) andis handled by a central page heap. The central page heap is again anarray of free lists. For i < 256, thekth entry is a free list of runs that consist ofk pages. The 256th entry is a free list ofruns that have length >= 256 pages:

An allocation for k pages is satisfied by looking in thekth free list. If that free list is empty, we look inthe next free list, and so forth. Eventually, we look in the lastfree list if necessary. If that fails, we fetch memory from thesystem (using sbrk, mmap, or by mapping in portions of /dev/mem).

If an allocation for k pages is satisfied by a runof pages of length > k, the remainder of therun is re-inserted back into the appropriate free list in thepage heap.

Spans

The heap managed by TCMalloc consists of a set of pages. A run ofcontiguous pages is represented by a Span object. A spancan either be allocated, or free. If free, the spanis one of the entries in a page heap linked-list. If allocated, it iseither a large object that has been handed off to the application, ora run of pages that have been split up into a sequence of smallobjects. If split into small objects, the size-class of the objectsis recorded in the span.

A central array indexed by page number can be used to find the span towhich a page belongs. For example, span a below occupies 2pages, span b occupies 1 page, span c occupies 5pages and span d occupies 3 pages.

A 32-bit address space can fit 2^20 4K pages, so this central arraytakes 4MB of space, which seems acceptable. On 64-bit machines, weuse a 3-level radix tree instead of an array to map from a page numberto the corresponding span pointer.

Deallocation

When an object is deallocated, we compute its page number and look it upin the central array to find the corresponding span object. The span tellsus whether or not the object is small, and its size-class if it issmall. If the object is small, we insert it into the appropriate freelist in the current thread's thread cache. If the thread cache nowexceeds a predetermined size (2MB by default), we run a garbagecollector that moves unused objects from the thread cache into centralfree lists.

If the object is large, the span tells us the range of pages coveredby the object. Suppose this range is [p,q]. We alsolookup the spans for pages p-1 and q+1. Ifeither of these neighboring spans are free, we coalesce them with the[p,q] span. The resulting span is inserted into theappropriate free list in the page heap.

Central Free Lists for Small Objects

As mentioned before, we keep a central free list for each size-class.Each central free list is organized as a two-level data structure:a set of spans, and a linked list of free objects per span.

An object is allocated from a central free list by removing thefirst entry from the linked list of some span. (If all spanshave empty linked lists, a suitably sized span is first allocatedfrom the central page heap.)

An object is returned to a central free list by adding it to thelinked list of its containing span. If the linked list length nowequals the total number of small objects in the span, this span is nowcompletely free and is returned to the page heap.

Garbage Collection of Thread Caches

A thread cache is garbage collected when the combined size of allobjects in the cache exceeds 2MB. The garbage collection thresholdis automatically decreased as the number of threads increases so thatwe don't waste an inordinate amount of memory in a program with lotsof threads.

We walk over all free lists in the cache and move some number ofobjects from the free list to the corresponding central list.

The number of objects to be moved from a free list is determined usinga per-list low-water-mark L. L records theminimum length of the list since the last garbage collection. Notethat we could have shortened the list by L objects at thelast garbage collection without requiring any extra accesses to thecentral list. We use this past history as a predictor of futureaccesses and move L/2 objects from the thread cache freelist to the corresponding central free list. This algorithm has thenice property that if a thread stops using a particular size, allobjects of that size will quickly move from the thread cache to thecentral free list where they can be used by other threads.