Sparse Memory Mapped Files

Memory mapped files are a powerful addition to the .NET framework in version 4.0.  They’ve been a key part of the windows Operating System since Windows 95, but using them required a working knowledge of COM Interop, as they weren’t exposed directly through the framework.  It’s well worth reading up on them, especially if you’re considering working with files across multiple processes, etc., but one of their biggest features is how efficiently they let you interact with extremely large files.

NTFS has a feature called sparse files, which is particularly useful with large files that mostly contain zero-data (that is where there are large blocks of zeroes in the file).  Combining memory mapped files with sparse files can be extremely powerful.  Unfortunately, .NET doesn’t expose this ability yet, so it’s back to good old COM Interop.  The following snippet will give you an extension method that can be applied to any FileStream to turn the underlying file into a sparse file.

There’s a great article about sparse files at http://www.flexhex.com/docs/articles/sparse-files.phtml, which is well worth a read.  One particular caveat of note is that once a file is marked as sparse, the operation cannot be undone, the only way to make it ‘not sparse’ is to rebuild the file from scratch (i.e copy the data into a new file).  As such the best way to use the above is when you create the file initially:

Now we can create our memory mapped file from our newly created sparse file:

 

Comments 3

Leave a Reply

%d bloggers like this: