MemoryConsumer 是内存的使用者,如 sort, shuffle。MemoryConsumer 的 spill 方法是抽象方法,由子类提供。
MemoryConsumer 的字段和构造方法如下:
public abstract class MemoryConsumer {protected final TaskMemoryManager taskMemoryManager;private final long pageSize;private final MemoryMode mode;protected long used;protected MemoryConsumer(TaskMemoryManager taskMemoryManager, long pageSize, MemoryMode mode) {this.taskMemoryManager = taskMemoryManager;this.pageSize = pageSize;this.mode = mode;}
spill
释放内存
/** @param size the amount of memory should be released* @param trigger the MemoryConsumer that trigger this spilling* @return the amount of released memory in bytes*/
public abstract long spill(long size, MemoryConsumer trigger) throws IOException;
分配内存
分配内存有3种,allocateArray 和 allocatePage 包括具体的占用这些内存的对象。acquireMemory 仅分配指定大小的内存。
public LongArray allocateArray(long size) {long required = size * 8L;MemoryBlock page = taskMemoryManager.allocatePage(required, this);if (page == null || page.size() < required) {throwOom(page, required);}used += required;return new LongArray(page);}
释放内存
同理,释放内存有 freeArray, freePage 和 freeMemory。
MemoryConsumer 的子类
BytesToBytesMap
HybridRowQueue
LongToUnsafeRowMap