Spark BytesToBytesMap分析

标签（空格分隔）： spark

---

• 在Tungsten-sort base中使用了bytetobytesMap的数据结构来实现序列化的排序。BytesToBytesMap是spark使用java实现的一种数据结构，在spark-core的unsafe.map下。
• 使用unsafe类的主要目的是直接对内存进行操作，比如分配内存，收回内存等。
• BytesToBytesMap主要包含两个变量，一个是用于记录record地址和hashcode的LongArray，还有一种是记录数据的dataPages。
• 可以看到一条record需要在LongArray中占用两个位置。1<<30是小于Integer.MAX_VALUE的最大2的power。因此BytesToBytesMap的最大容量为1<<29.
• 内部类Location用于寻址和插入。
• 插入一条记录的过程如下
  
  • 首先根据记录的这条record的keyBase keyOffset keyLength（这些在申请内存时会得到？没太弄懂）生成hashcode

   public Location lookup(Object keyBase, long keyOffset, int keyLength) {    safeLookup(keyBase, keyOffset, keyLength, loc,      Murmur3_x86_32.hashUnsafeWords(keyBase, keyOffset, keyLength, 42));    return loc;

• 调用safeLookUp

public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {    assert(longArray != null);    if (enablePerfMetrics) {      numKeyLookups++;    }    int pos = hash & mask;    int step = 1;    while (true) {      if (enablePerfMetrics) {        numProbes++;      }      if (longArray.get(pos * 2) == 0) {        // This is a new key.        loc.with(pos, hash, false);        return;      } else {        long stored = longArray.get(pos * 2 + 1);        if ((int) (stored) == hash) {          // Full hash code matches.  Let's compare the keys for equality.          loc.with(pos, hash, true);          if (loc.getKeyLength() == keyLength) {            final boolean areEqual = ByteArrayMethods.arrayEquals(              keyBase,              keyOffset,              loc.getKeyBase(),              loc.getKeyOffset(),              keyLength            );            if (areEqual) {              return;            }          }        }      }      pos = (pos + step) & mask;      step++;    }  }

• pos为使用mask对hashcode的截断。在LongArray中fullAddress为2*pos，hashcode位置为2*pos+1,
• 可以看到解决hash冲突的方法为开放地址法，pos冲突，就pos=pos+1,重试。

• 得到Location的实例后可以调用Location的append方法进行插入。

  public boolean append(Object kbase, long koff, int klen, Object vbase, long voff, int vlen) {  assert (klen % 8 == 0);  assert (vlen % 8 == 0);  assert (longArray != null);  if (numKeys == MAX_CAPACITY    // The map could be reused from last spill (because of no enough memory to grow),    // then we don't try to grow again if hit the `growthThreshold`.    || !canGrowArray && numKeys >= growthThreshold) {    return false;  }  // Here, we'll copy the data into our data pages. Because we only store a relative offset from  // the key address instead of storing the absolute address of the value, the key and value  // must be stored in the same memory page.  // (8 byte key length) (key) (value) (8 byte pointer to next value)  int uaoSize = UnsafeAlignedOffset.getUaoSize();  final long recordLength = (2 * uaoSize) + klen + vlen + 8;  if (currentPage == null || currentPage.size() - pageCursor < recordLength) {    if (!acquireNewPage(recordLength + uaoSize)) {      return false;    }  }  // --- Append the key and value data to the current data page --------------------------------  final Object base = currentPage.getBaseObject();  long offset = currentPage.getBaseOffset() + pageCursor;  final long recordOffset = offset;  UnsafeAlignedOffset.putSize(base, offset, klen + vlen + uaoSize);  UnsafeAlignedOffset.putSize(base, offset + uaoSize, klen);  offset += (2 * uaoSize);  Platform.copyMemory(kbase, koff, base, offset, klen);  offset += klen;  Platform.copyMemory(vbase, voff, base, offset, vlen);  offset += vlen;  // put this value at the beginning of the list  Platform.putLong(base, offset, isDefined ? longArray.get(pos * 2) : 0);  // --- Update bookkeeping data structures ----------------------------------------------------  offset = currentPage.getBaseOffset();  UnsafeAlignedOffset.putSize(base, offset, UnsafeAlignedOffset.getSize(base, offset) + 1);  pageCursor += recordLength;  final long storedKeyAddress = taskMemoryManager.encodePageNumberAndOffset(    currentPage, recordOffset);  longArray.set(pos * 2, storedKeyAddress);  updateAddressesAndSizes(storedKeyAddress);  numValues++;  if (!isDefined) {    numKeys++;    longArray.set(pos * 2 + 1, keyHashcode);    isDefined = true;    if (numKeys >= growthThreshold && longArray.size() < MAX_CAPACITY) {      try {        growAndRehash();      } catch (OutOfMemoryError oom) {        canGrowArray = false;      }    }  }  return true;}

  • 使用内存复制的方式将record添加到当前的page。然后更新LongArray。
  • 示意图
    

参考

• csdn:Spark 内存管理之BytesToBytesMap