使用内存映射文件获取巨大的矩阵

总览

矩阵可能真的很大，有时甚至比一个数组中可以容纳的更大。您可以通过具有多个数组来扩展最大大小，但这会使堆大小确实很大且效率低下。一种替代方法是在内存映射文件上使用包装器。内存映射文件的优点是它们对堆的影响很小，并且可以由操作系统相当透明地交换出来。

巨大的矩阵

此代码支持double的大型矩阵。它将文件分区为1 GB映射。（由于Java一次不支持2 GB或更大的映射，这是我的宠儿；）

import sun.misc.Cleaner;
import sun.nio.ch.DirectBuffer;import java.io.Closeable;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.util.ArrayList;
import java.util.List;public class LargeDoubleMatrix implements Closeable {private static final int MAPPING_SIZE = 1 << 30;private final RandomAccessFile raf;private final int width;private final int height;private final List mappings = new ArrayList();public LargeDoubleMatrix(String filename, int width, int height) throws IOException {this.raf = new RandomAccessFile(filename, "rw");try {this.width = width;this.height = height;long size = 8L * width * height;for (long offset = 0; offset < size; offset += MAPPING_SIZE) {long size2 = Math.min(size - offset, MAPPING_SIZE);mappings.add(raf.getChannel().map(FileChannel.MapMode.READ_WRITE, offset, size2));}} catch (IOException e) {raf.close();throw e;}}protected long position(int x, int y) {return (long) y * width + x;}public int width() {return width;}public int height() {return height;}public double get(int x, int y) {assert x >= 0 && x < width;assert y >= 0 && y < height;long p = position(x, y) * 8;int mapN = (int) (p / MAPPING_SIZE);int offN = (int) (p % MAPPING_SIZE);return mappings.get(mapN).getDouble(offN);}public void set(int x, int y, double d) {assert x >= 0 && x < width;assert y >= 0 && y < height;long p = position(x, y) * 8;int mapN = (int) (p / MAPPING_SIZE);int offN = (int) (p % MAPPING_SIZE);mappings.get(mapN).putDouble(offN, d);}public void close() throws IOException {for (MappedByteBuffer mapping : mappings)clean(mapping);raf.close();}private void clean(MappedByteBuffer mapping) {if (mapping == null) return;Cleaner cleaner = ((DirectBuffer) mapping).cleaner();if (cleaner != null) cleaner.clean();}
}public class LargeDoubleMatrixTest {@Testpublic void getSetMatrix() throws IOException {long start = System.nanoTime();final long used0 = usedMemory();LargeDoubleMatrix matrix = new LargeDoubleMatrix("ldm.test", 1000 * 1000, 1000 * 1000);for (int i = 0; i < matrix.width(); i++)matrix.set(i, i, i);for (int i = 0; i < matrix.width(); i++)assertEquals(i, matrix.get(i, i), 0.0);long time = System.nanoTime() - start;final long used = usedMemory() - used0;if (used == 0)System.err.println("You need to use -XX:-UseTLAB to see small changes in memory usage.");System.out.printf("Setting the diagonal took %,d ms, Heap used is %,d KB%n", time / 1000 / 1000, used / 1024);matrix.close();}private long usedMemory() {return Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();}
}

通过以下测试，该测试将一百万*一百万矩阵的每个对角线值写入。这太大了，无法希望在堆上创建。

Setting the diagonal took 314,819 ms, Heap used is 2,025 KB$ ls -l ldm.test
-rw-rw-r-- 1 peter peter 8000000000000 2011-12-30 12:42 ldm.test
$ du -s ldm.test 
4010600 ldm.test

在Java进程中，虚拟内存为8,000,000,000,000字节或〜7.3 TB！这行得通，因为它仅在您使用的页面中分配或分配页面。因此，尽管文件大小几乎为8 TB，但实际使用的磁盘空间和内存为4 GB。
使用100K * 100K矩阵的较小文件大小，您将看到类似以下的内容。它仍然是一个80 GB的矩阵，使用了很小的堆空间。 ;）

Setting the diagonal took 110 ms, Heap used is 71 KB$ ls -l ldm.test
-rw-rw-r-- 1 peter peter 80000000000 2011-12-30 12:49 ldm.test
$ du -s ldm.test 
400000 ldm.test

参考：在Vanilla Java博客上，使用我们的JCG合作伙伴 Peter Lawrey 的巨大内存矩阵的内存映射文件

相关文章：