Sparse Memory Mapped Files

Memory mapped files are a powerful addition to the .NET framework in version 4.0.  They’ve been a key part of the windows Operating System since Windows 95, but using them required a working knowledge of COM Interop, as they weren’t exposed directly through the framework.  It’s well worth reading up on them, especially if you’re considering working with files across multiple processes, etc., but one of their biggest features is how efficiently they let you interact with extremely large files.

NTFS has a feature called sparse files, which is particularly useful with large files that mostly contain zero-data (that is where there are large blocks of zeroes in the file).  Combining memory mapped files with sparse files can be extremely powerful.  Unfortunately, .NET doesn’t expose this ability yet, so it’s back to good old COM Interop.  The following snippet will give you an extension method that can be applied to any FileStream to turn the underlying file into a sparse file.

public static class Extensions
{
	[DllImport("Kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
	private static extern bool DeviceIoControl(
		SafeFileHandle hDevice,
		int dwIoControlCode,
		IntPtr InBuffer,
		int nInBufferSize,
		IntPtr OutBuffer,
		int nOutBufferSize,
		ref int pBytesReturned,
		[In] ref NativeOverlapped lpOverlapped
	);

	public static bool MarkAsSparseFile(FileStream fileStream)
	{
		int bytesReturned = 0;
		NativeOverlapped lpOverlapped = new NativeOverlapped();
		return DeviceIoControl(
			fileStream.SafeFileHandle,
			590020, //FSCTL_SET_SPARSE,
			IntPtr.Zero,
			0,
			IntPtr.Zero,
			0,
			ref bytesReturned,
			ref lpOverlapped);
	}
}

There’s a great article about sparse files at http://www.flexhex.com/docs/articles/sparse-files.phtml, which is well worth a read.  One particular caveat of note is that once a file is marked as sparse, the operation cannot be undone, the only way to make it ‘not sparse’ is to rebuild the file from scratch (i.e copy the data into a new file).  As such the best way to use the above is when you create the file initially:

using (FileStream f = File.Create(@"C:\MySparseFile.tmp"))
{
	// Try to mark file as sparse, if this fails then the SetLength will allocate the full file size.
	if (!MarkAsSparseFile(f))
		Log("Failed to create world as sparse file.");

	// Set file length
	f.SetLength(size);
	f.Close();
}

Now we can create our memory mapped file from our newly created sparse file:

using (MemoryMappedFile m = MemoryMappedFile.CreateFromFile(@"C:\MySparseFile.tmp"))
{
	// TODO Use our file!
}