C++ Memory Mapped File I/O

The usual way to read a file is to ask the operating system to copy some of it into a buffer you own, process the buffer, and ask again. Memory mapping drops the copying entirely: you ask the OS to make the file appear in your address space, and you get back a pointer. From then on, the file is just an array of bytes. p[0] is the first byte of the file, p[size - 1] is the last, and the kernel quietly loads whichever pages you touch, when you touch them.

That's a powerful simplification, and for the right workloads it's a real performance win. It also moves file I/O errors out of return codes and into the memory system, which is where most of the surprises come from. This page covers both halves.

Everything here assumes a 64-bit build (x64 or ARM64), where a process has tens of terabytes of address space and a file of any practical size can be mapped as a single view. The examples enforce that with a static_assert. If you have to target 32-bit, the rules change in a few important ways; they're collected in A Note on 32-bit Builds.

What It Actually Is

Every modern OS already keeps recently used file data in RAM, in the page cache (Linux) or the cache manager's working set (Windows). When you call read() or ReadFile(), the kernel makes sure the requested pages are in that cache, then copies them into your buffer. Your buffer is a second copy of data that's already sitting in memory.

A memory mapping skips the second copy. The kernel edits your process's page tables so that a range of virtual addresses points straight at the page-cache pages for that file. No data moves when you create the mapping; you've only reserved address space and recorded which file backs it. The data arrives later, a page (usually 4 KB) at a time, through demand paging.

The first time your code reads an address in the mapping, the CPU finds no valid page-table entry and raises a page fault. The kernel handles it: if the file page is already cached, it just wires up the entry (a minor fault, around a microsecond); if not, it issues a disk read and blocks your thread until it completes (a major fault). Then it resumes your instruction as if nothing happened. Your code never sees any of this, except as latency, and except when the read can't be satisfied at all.

Two flavours matter for files:

• Shared mappings (MAP_SHARED, or PAGE_READWRITE + FILE_MAP_WRITE) write through to the file. Stores into the mapping eventually land on disk, and other processes mapping the same file see them.
• Private / copy-on-write mappings (MAP_PRIVATE, or PAGE_WRITECOPY + FILE_MAP_COPY) give you a view that starts out identical to the file, but any page you write is silently duplicated into anonymous memory. The file is never modified.

For read-only parsing, either works. The examples below use MAP_PRIVATE on Linux (the conventional choice for read-only) and plain PAGE_READONLY on Windows.

Why You'd Use It

• Random access without seek bookkeeping. Index files, B-trees, lookup tables, binary formats with offset tables: jumping to p + offset is much simpler than lseek + read + buffer management, and only the pages you touch are ever loaded.
• Parsers that want contiguous input. A tokenizer that can look ahead arbitrarily, a JSON/CSV parser that takes std::string_view, a regex over the whole file: no chunk-boundary handling, no "token split across two buffers" bugs.
• No second copy. For a 4 GB file, read()-ing it all means 4 GB in the page cache plus 4 GB in your heap. A mapping uses the cache pages directly, and the OS can evict them under pressure because it can always reload them from the file.
• Repeated access is nearly free. Once pages are mapped, re-reading them is an ordinary memory load with no system call.
• Sharing between processes. Ten processes mapping the same read-only file share one set of physical pages. This is exactly how the OS loads executables and shared libraries.
• Persistence for in-memory structures. Write a data structure into a shared mapping with offset-based (not pointer-based) links, and it is the file format. LMDB, many search indexes and game asset packs work this way.
• Inter-process communication. A shared mapping, file-backed or anonymous, is the fastest IPC channel the OS offers, since there's no copying at all.

The OS Facilities

C++ has no standard memory-mapping facility (nothing in <fstream> or <filesystem> touches it), so you either call the OS directly or use a library that does. The two native APIs follow the same lifecycle with different granularity.

Win32

Linux / POSIX

macOS and the BSDs share the POSIX core (mmap, munmap, msync, madvise) but not the Linux extensions (MAP_POPULATE, mremap, memfd_create), so keep those behind #ifdef __linux__ in portable code.

Popular Libraries

The native APIs are small, but the error handling, the empty-file case and the Windows/POSIX split are tedious to get right more than once. These are the common wrappers:

For comparison with the hand-written versions below, here are the two most common:

// mio
#include <mio/mmap.hpp>
std::error_code ec;
mio::mmap_source src = mio::make_mmap_source("readings.csv", ec);
if (!ec) parse_readings({src.data(), src.size()});

// Boost.Interprocess
#include <boost/interprocess/file_mapping.hpp>
#include <boost/interprocess/mapped_region.hpp>
namespace bip = boost::interprocess;
bip::file_mapping  fm("readings.csv", bip::read_only);
bip::mapped_region rg(fm, bip::read_only);
parse_readings({static_cast<const char*>(rg.get_address()), rg.get_size()});

Check how each library treats a zero-length file before relying on it; some throw, some return an empty view, and the native calls themselves fail.

Example: Parsing a File on Linux and Windows

The task: read a CSV of sensor readings (sensor-17,23.75 per line), count the valid rows, sum and take the max of the value column, and count malformed lines. The design principle is to keep the parser ignorant of how the bytes arrived. It takes a std::string_view, so it works identically on a mapping, a std::string read with ifstream, or a test fixture. Only a small RAII class differs per platform.

// parse_readings.hpp: platform-neutral; knows nothing about mapping
#pragma once
#include <algorithm>
#include <charconv>
#include <cstddef>
#include <cstring>
#include <limits>
#include <string_view>

struct Stats {
    std::size_t rows = 0;
    std::size_t bad  = 0;
    double      sum  = 0.0;
    double      max  = -std::numeric_limits<double>::infinity();
};

// Input lines look like "sensor-17,23.75". Blank lines and '#' comments are skipped.
// The buffer is NOT null-terminated: every scan is bounded by `end`.
inline Stats parse_readings(std::string_view text) {
    Stats s;
    const char* p   = text.data();
    const char* end = p + text.size();

    while (p < end) {
        const char* eol = static_cast<const char*>(std::memchr(p, '\n', end - p));
        if (!eol) eol = end;                        // last line has no newline
        std::string_view line(p, eol - p);
        p = (eol == end) ? end : eol + 1;

        if (!line.empty() && line.back() == '\r') line.remove_suffix(1);  // CRLF files
        if (line.empty() || line.front() == '#') continue;

        std::size_t comma = line.find(',');
        if (comma == std::string_view::npos) { ++s.bad; continue; }

        double v = 0.0;
        const char* first = line.data() + comma + 1;
        const char* last  = line.data() + line.size();
        auto [ptr, ec] = std::from_chars(first, last, v);
        if (ec != std::errc{} || ptr != last) { ++s.bad; continue; }

        ++s.rows;
        s.sum += v;
        s.max  = std::max(s.max, v);
    }
    return s;
}

Note what the parser doesn't do: it never calls strtod, atof, sscanf, strlen or strchr. Those all scan until a '\0', and a mapped file has no terminator. If the file's last byte sits at the very end of a page, the next byte is an unmapped address, and the scan crashes. std::from_chars and memchr take an explicit end, so they're safe.

// mmap_linux.cpp: g++ -std=c++17 -O2 mmap_linux.cpp -o mmap_linux
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>

#include <cerrno>
#include <cstdio>
#include <string_view>
#include <system_error>

#include "parse_readings.hpp"

static_assert(sizeof(void*) == 8, "64-bit build assumed; see the note on 32-bit builds");

class MappedFile {
public:
    explicit MappedFile(const char* path) {
        int fd = ::open(path, O_RDONLY | O_CLOEXEC);
        if (fd < 0) fail("open");

        struct stat st {};
        if (::fstat(fd, &st) != 0) { int e = errno; ::close(fd); fail("fstat", e); }
        if (!S_ISREG(st.st_mode))  { ::close(fd); fail("not a regular file", EINVAL); }

        size_ = static_cast<std::size_t>(st.st_size);
        if (size_ > 0) {                            // mmap of length 0 is EINVAL
            void* p = ::mmap(nullptr, size_, PROT_READ, MAP_PRIVATE, fd, 0);
            if (p == MAP_FAILED) { int e = errno; ::close(fd); fail("mmap", e); }
            data_ = static_cast<const char*>(p);
            ::madvise(p, size_, MADV_SEQUENTIAL);   // hint only; failure is harmless
        }
        ::close(fd);                                // the mapping keeps the file alive
    }

    ~MappedFile() { if (data_) ::munmap(const_cast<char*>(data_), size_); }

    MappedFile(const MappedFile&)            = delete;
    MappedFile& operator=(const MappedFile&) = delete;

    std::string_view view() const { return {data_, size_}; }

private:
    [[noreturn]] static void fail(const char* what, int err = errno) {
        throw std::system_error(err, std::generic_category(), what);
    }

    const char* data_ = nullptr;
    std::size_t size_ = 0;
};

int main(int argc, char** argv) {
    if (argc != 2) { std::fprintf(stderr, "usage: %s <file>\n", argv[0]); return 2; }
    try {
        MappedFile file(argv[1]);
        Stats s = parse_readings(file.view());
        std::printf("rows=%zu bad=%zu sum=%.3f max=%.3f\n", s.rows, s.bad, s.sum, s.max);
    } catch (const std::system_error& e) {
        std::fprintf(stderr, "error: %s\n", e.what());
        return 1;
    }
    return 0;
}

// mmap_win32.cpp: cl /std:c++17 /EHsc /O2 /W4 mmap_win32.cpp
#define WIN32_LEAN_AND_MEAN
#define NOMINMAX
#include <windows.h>

#include <cstdio>
#include <string_view>
#include <system_error>

#include "parse_readings.hpp"

static_assert(sizeof(void*) == 8, "64-bit build assumed; see the note on 32-bit builds");

class MappedFile {
public:
    explicit MappedFile(const wchar_t* path) {
        HANDLE file = ::CreateFileW(path, GENERIC_READ, FILE_SHARE_READ, nullptr,
                                    OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
        if (file == INVALID_HANDLE_VALUE) fail("CreateFileW");

        LARGE_INTEGER size{};
        if (!::GetFileSizeEx(file, &size)) { DWORD e = ::GetLastError(); ::CloseHandle(file); fail("GetFileSizeEx", e); }
        size_ = static_cast<std::size_t>(size.QuadPart);

        if (size_ > 0) {                            // mapping an empty file fails
            // 0, 0 = map the file at its current size
            HANDLE mapping = ::CreateFileMappingW(file, nullptr, PAGE_READONLY, 0, 0, nullptr);
            if (!mapping) { DWORD e = ::GetLastError(); ::CloseHandle(file); fail("CreateFileMappingW", e); }

            void* p = ::MapViewOfFile(mapping, FILE_MAP_READ, 0, 0, 0);   // 0 = whole file
            DWORD e = ::GetLastError();
            ::CloseHandle(mapping);                 // the view keeps the section alive
            if (!p) { ::CloseHandle(file); fail("MapViewOfFile", e); }
            data_ = static_cast<const char*>(p);

            WIN32_MEMORY_RANGE_ENTRY range{p, size_};
            ::PrefetchVirtualMemory(::GetCurrentProcess(), 1, &range, 0);  // hint only
        }
        ::CloseHandle(file);                        // ...and the file
    }

    ~MappedFile() { if (data_) ::UnmapViewOfFile(data_); }

    MappedFile(const MappedFile&)            = delete;
    MappedFile& operator=(const MappedFile&) = delete;

    std::string_view view() const { return {data_, size_}; }

private:
    [[noreturn]] static void fail(const char* what, DWORD err = ::GetLastError()) {
        throw std::system_error(static_cast<int>(err), std::system_category(), what);
    }

    const char* data_ = nullptr;
    std::size_t size_ = 0;
};

int wmain(int argc, wchar_t** argv) {
    if (argc != 2) { std::fwprintf(stderr, L"usage: %ls <file>\n", argv[0]); return 2; }
    try {
        MappedFile file(argv[1]);
        Stats s = parse_readings(file.view());
        std::printf("rows=%zu bad=%zu sum=%.3f max=%.3f\n", s.rows, s.bad, s.sum, s.max);
    } catch (const std::system_error& e) {
        std::fprintf(stderr, "error: %s\n", e.what());
        return 1;
    }
    return 0;
}

A few things worth pointing out in both versions:

• Handles are closed immediately after mapping. On Linux the mapping holds its own reference to the file; on Windows the view holds references to the section, which holds the file. The RAII class only has to remember the pointer and length.
• Empty files are special-cased. mmap with length 0 returns EINVAL, and CreateFileMappingW on an empty file fails with ERROR_FILE_INVALID. An empty file maps to an empty view without calling either.
• 64-bit is asserted, not assumed silently. With 64-bit size_t and pointers, a file of any size the file system allows fits in one view, so the size goes straight from st_size / QuadPart into size_t with no range check. The static_assert stops anyone compiling this as 32-bit, where that conversion could silently truncate (see the 32-bit note).
• Errors capture errno / GetLastError() first. Cleanup calls like close and CloseHandle can overwrite the error you're about to report.
• The hint calls ignore failure. madvise and PrefetchVirtualMemory only affect performance.

Build and run

# Linux (GCC 11+: floating-point from_chars needs libstdc++ 11 or newer)
$ g++ -std=c++17 -O2 -Wall -Wextra mmap_linux.cpp -o mmap_linux
$ ./mmap_linux readings.csv
rows=3 bad=2 sum=119.250 max=100.000

# Windows (x64 Native Tools prompt, VS 2019 16.4+)
> cl /std:c++17 /EHsc /O2 /W4 mmap_win32.cpp
> mmap_win32.exe readings.csv
rows=3 bad=2 sum=119.250 max=100.000

Given this input (CRLF on one line, a blank line, a comment, and two malformed rows), both builds produce the same result:

# sensor,value
sensor-1,23.75
sensor-2,-4.5

broken line                  ← no comma: bad
sensor-3,abc                 ← not a number: bad
sensor-4,100                 ← no trailing newline

To make the program portable from one source file, put the two MappedFile classes behind #ifdef _WIN32 with a common constructor taking a std::filesystem::path (use path.c_str(), which is wchar_t* on Windows and char* elsewhere), or use mio and skip both.

Access Hints and Performance

Mapping a file is not automatically faster than reading it. What it removes is the copy and the system call per buffer. What it adds is a page fault per page touched, page-table setup and teardown, and TLB pressure. Which side wins depends on the access pattern:

• One sequential pass over a cold file: roughly a tie. Both are bound by disk throughput, and read() into a reused 64 KB to 1 MB buffer has very predictable behaviour. Linux's fault-around maps up to 16 neighbouring cached pages per fault, which helps the mapped side.
• One sequential pass over a hot (cached) file: mapping usually wins by skipping the copy, especially with MAP_POPULATE or PrefetchVirtualMemory to batch the faults.
• Random access or repeated access: mapping wins clearly. No syscall per lookup, no buffer cache of your own to maintain.
• Many small files: read() wins. Setting up and tearing down a mapping costs more than copying a few KB.

Hints worth knowing: MADV_SEQUENTIAL makes readahead more aggressive and marks pages behind you as cheap to reclaim (it does not stop your resident set growing; see Very Large Files), MADV_RANDOM turns readahead off (good for index lookups, so you don't pull in 128 KB to read 8 bytes), MADV_WILLNEED or PrefetchVirtualMemory start I/O asynchronously before you need the data, and MAP_POPULATE faults everything in up front so the parsing loop never stalls. (That last one only makes sense for files that comfortably fit in RAM; on a file larger than memory it reads the whole thing and evicts the beginning before you get to it.) Always measure with the file both cold (drop caches: echo 3 | sudo tee /proc/sys/vm/drop_caches) and hot, because the answer is often different.

Very Large Files

On a 64-bit build, the question with a 5 GB, 50 GB or 500 GB file is not whether it can be mapped. It can, in one call, exactly like the examples above. The questions are how much memory the process will appear to use, how many page tables and faults the kernel has to manage, how fast you can get through it, and which sizes and offsets in your own code are quietly still 32-bit. This section works through each, with measurements from a real 4.8 GB file.

Address space is not the limit

A 64-bit process on x64 Linux or Windows has 128 TB of user address space (256 TB on 48-bit ARM64 Linux). Mapping a file reserves address space; it doesn't allocate RAM or read anything. Mapping a 1 TB file costs a few kernel bookkeeping structures and nothing else until you touch it. The practical ceilings are elsewhere:

• The file system. FAT32 caps a file at 4 GB, and ext4 at 16 TB with 4 KB blocks. NTFS, XFS and exFAT go far beyond anything you'll map.
• Physical RAM, but only for speed. A file bigger than RAM maps and parses correctly. Pages behind the parser are evicted to make room for pages ahead of it, and throughput becomes whatever the storage delivers.
• Per-process limits. ulimit -v (RLIMIT_AS) counts mapped address space, so a 100 GB mapping fails with ENOMEM under a 50 GB address-space limit even though no memory is used. Job objects on Windows limit committed memory, which read-only file views don't use.

Keep every size and offset 64-bit

The mapping APIs are 64-bit clean on a 64-bit build; the bugs come from the code around them. Past 2 GB and 4 GB, these all break:

• int or unsigned indexes and counters. for (int i = 0; i < size; ++i) overflows at 2 GB. Use std::size_t for positions inside a view and std::uint64_t for file offsets.
• long on Windows. Windows is LLP64: long is 32-bit even in a 64-bit build. fseek/ftell take long, so use _fseeki64/_ftelli64 there (and fseeko/ftello on POSIX), and never store a file size in long.
• Legacy Win32 calls. Use GetFileSizeEx, not GetFileSize. CreateFileMappingW and MapViewOfFile take 64-bit sizes and offsets as two DWORD halves, so split them with offset >> 32 and offset & 0xFFFFFFFF as the sliding-window example below does.
• printf formats. %zu for size_t, and %llu with a cast to unsigned long long for uint64_t. %d and %lu truncate.
• Size arithmetic. size * i / n overflows for huge files; divide first (size / n * i), as the parallel example does.

What "memory use" means for a mapped file

Every page you touch in a mapping is counted in your process's resident set (RSS on Linux, working set on Windows) for as long as it stays mapped and resident. Parse a 50 GB file through one big mapping on a machine with spare RAM, and top or Task Manager will show the process approaching 50 GB. Those pages are clean copies of file data, and the OS can drop them instantly under pressure. They aren't a leak. But monitoring alerts, container dashboards and colleagues don't know that.

• Commit charge. Read-only file views don't consume commit (Windows) or count against overcommit (Linux), because the file backs them. Writable private mappings do: Windows charges commit for the entire view up front when you map with FILE_MAP_COPY, and under strict overcommit (vm.overcommit_memory=2) Linux can refuse a large PROT_WRITE + MAP_PRIVATE mapping with ENOMEM. For huge read-only parsing, map read-only.
• Containers. Under cgroup v2, page cache is charged to the container that read it. A container with a 2 GB memory limit can still stream a 100 GB mapped file, because the kernel reclaims clean file pages at the limit instead of killing the process, but memory.current will sit at the limit, and reclaim inside the cgroup costs some throughput.
• Other processes' cache. One pass over a file bigger than RAM pushes everyone else's cached data out. On Linux, posix_fadvise(fd, off, len, POSIX_FADV_DONTNEED) on ranges you've finished (and unmapped or released) drops them from the page cache. Windows has no direct per-range equivalent; finished pages move to the standby list and get reused first.

Three strategies for a big sequential pass

You get the same results and roughly the same speed from all three. What differs is the memory footprint and the complexity:

1. Map the whole file and parse. This is the example above, unchanged. It's the simplest option, and the best one when RAM is plentiful or the file will be re-read (the second pass is all hits). Its resident set grows towards the file size.

2. Map the whole file, release pages behind the parser. Keep the single mapping and the simple string_view, but parse in chunks and, after each chunk, tell the OS you're done with the pages behind you. On Linux that's madvise(MADV_DONTNEED), which detaches them from your process (on a read-only mapping the data stays in the page cache, so touching it again just re-faults). On Windows, calling VirtualUnlock on a range that was never locked is the documented way to remove pages from the working set.

// drop_behind.hpp: one whole-file mapping, with processed pages handed back as you go
#pragma once
#ifdef _WIN32
#ifndef NOMINMAX
#define NOMINMAX
#endif
#include <windows.h>
#else
#include <sys/mman.h>
#endif
#include <cstring>
#include <string_view>
#include "parse_readings.hpp"

// Tell the OS we're done with [p, p + n). Clean file pages are simply dropped;
// touching them again later re-faults them from the page cache or the file.
inline void release_pages(const char* p, std::size_t n) {
#ifdef _WIN32
    // On a range that isn't locked, VirtualUnlock removes the pages from the
    // working set (and reports ERROR_NOT_LOCKED, which is expected here).
    ::VirtualUnlock(const_cast<char*>(p), n);
#else
    // Safe on a read-only mapping. On a MAP_PRIVATE page you've written to,
    // MADV_DONTNEED would throw your changes away.
    ::madvise(const_cast<char*>(p), n, MADV_DONTNEED);
#endif
}

// Parse in chunks of about `chunk` bytes, each extended to the end of its last line,
// releasing every whole page behind the parser. `page` is the OS page size.
inline Stats parse_release_behind(std::string_view whole, std::size_t chunk, std::size_t page) {
    Stats total;
    const char* base = whole.data();
    std::size_t pos = 0, released = 0;
    while (pos < whole.size()) {
        std::size_t end = std::min(pos + chunk, whole.size());
        const void* nl  = std::memchr(base + end, '\n', whole.size() - end);
        end = nl ? static_cast<const char*>(nl) - base + 1 : whole.size();

        Stats part = parse_readings(whole.substr(pos, end - pos));
        total.rows += part.rows;
        total.bad  += part.bad;
        total.sum  += part.sum;
        total.max   = std::max(total.max, part.max);
        pos = end;

        std::size_t done = pos - pos % page;            // whole pages fully behind us
        if (done > released) {
            release_pages(base + released, done - released);
            released = done;
        }
    }
    return total;
}

Use it with the whole-file MappedFile from either platform example: parse_release_behind(file.view(), 64 << 20, page_size), where page_size comes from sysconf(_SC_PAGESIZE) or GetSystemInfo().dwPageSize. Releasing every 1 MB chunk kept the peak at about 5 MB on both platforms in testing. Larger chunks mean fewer system calls, and a peak of roughly one chunk. On Linux 5.4+, MADV_COLD or MADV_PAGEOUT are gentler alternatives that keep pages mapped but make them first in line for reclaim.

3. Map a sliding window. Never map more than a fixed amount at once: map a window, parse it, unmap it, map the next. This bounds address space, resident set and page tables by construction, which matters if many threads or processes each work through their own huge files, and it's the only one of the three that also works on 32-bit. The cost is that records can straddle window boundaries, so each window has to be cut cleanly.

// mmap_windowed.cpp: parse a file of any size through a bounded sliding window
//   Linux:   g++ -std=c++17 -O2 mmap_windowed.cpp -o mmap_windowed
//   Windows: cl /std:c++17 /EHsc /O2 /W4 mmap_windowed.cpp
#ifdef _WIN32
#define WIN32_LEAN_AND_MEAN
#define NOMINMAX
#include <windows.h>
#else
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <cerrno>
#endif

#include <algorithm>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <stdexcept>
#include <string_view>
#include <system_error>

#include "parse_readings.hpp"

static_assert(sizeof(void*) == 8, "64-bit build assumed; see the note on 32-bit builds");

// An open file that can hand out read-only views of any byte range.
class MappableFile {
public:
#ifdef _WIN32
    explicit MappableFile(const wchar_t* path) try {
        file_ = ::CreateFileW(path, GENERIC_READ, FILE_SHARE_READ, nullptr,
                              OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
        if (file_ == INVALID_HANDLE_VALUE) fail("CreateFileW");
        LARGE_INTEGER size{};
        if (!::GetFileSizeEx(file_, &size)) fail("GetFileSizeEx");
        size_ = static_cast<std::uint64_t>(size.QuadPart);
        SYSTEM_INFO si{};
        ::GetSystemInfo(&si);
        granularity_ = si.dwAllocationGranularity;          // 64 KB
        if (size_ > 0) {
            mapping_ = ::CreateFileMappingW(file_, nullptr, PAGE_READONLY, 0, 0, nullptr);
            if (!mapping_) fail("CreateFileMappingW");
        }
    } catch (...) { close_handles(); throw; }     // the destructor won't run
    ~MappableFile() { close_handles(); }
#else
    explicit MappableFile(const char* path) try {
        fd_ = ::open(path, O_RDONLY | O_CLOEXEC);
        if (fd_ < 0) fail("open");
        struct stat st {};
        if (::fstat(fd_, &st) != 0) fail("fstat");
        size_        = static_cast<std::uint64_t>(st.st_size);
        granularity_ = static_cast<std::uint64_t>(::sysconf(_SC_PAGESIZE));
    } catch (...) { close_handles(); throw; }     // the destructor won't run
    ~MappableFile() { close_handles(); }
#endif
    MappableFile(const MappableFile&)            = delete;
    MappableFile& operator=(const MappableFile&) = delete;

    std::uint64_t size() const        { return size_; }
    std::uint64_t granularity() const { return granularity_; }

    // RAII view of [offset, offset + length). offset must be a multiple of granularity().
    class View {
    public:
        View(const MappableFile& f, std::uint64_t offset, std::size_t length) : size_(length) {
#ifdef _WIN32
            void* p = ::MapViewOfFile(f.mapping_, FILE_MAP_READ,
                                      static_cast<DWORD>(offset >> 32),
                                      static_cast<DWORD>(offset & 0xFFFFFFFFu), length);
            if (!p) fail("MapViewOfFile");
#else
            void* p = ::mmap(nullptr, length, PROT_READ, MAP_PRIVATE, f.fd_,
                             static_cast<off_t>(offset));
            if (p == MAP_FAILED) fail("mmap");
            ::madvise(p, length, MADV_SEQUENTIAL);
#endif
            data_ = static_cast<const char*>(p);
        }
        ~View() {
#ifdef _WIN32
            ::UnmapViewOfFile(data_);
#else
            ::munmap(const_cast<char*>(data_), size_);
#endif
        }
        View(const View&)            = delete;
        View& operator=(const View&) = delete;

        std::string_view bytes() const { return {data_, size_}; }

    private:
        const char* data_;
        std::size_t size_;
    };

private:
#ifdef _WIN32
    [[noreturn]] static void fail(const char* what) {
        throw std::system_error(static_cast<int>(::GetLastError()), std::system_category(), what);
    }
    void close_handles() {
        if (mapping_) ::CloseHandle(mapping_);
        if (file_ != INVALID_HANDLE_VALUE) ::CloseHandle(file_);
    }
    HANDLE file_    = INVALID_HANDLE_VALUE;
    HANDLE mapping_ = nullptr;
#else
    [[noreturn]] static void fail(const char* what) {
        throw std::system_error(errno, std::generic_category(), what);
    }
    void close_handles() { if (fd_ >= 0) ::close(fd_); }
    int fd_ = -1;
#endif
    std::uint64_t size_        = 0;
    std::uint64_t granularity_ = 0;
};

static void merge(Stats& into, const Stats& part) {
    into.rows += part.rows;
    into.bad  += part.bad;
    into.sum  += part.sum;
    into.max   = std::max(into.max, part.max);
}

// Each window is cut at its last '\n', so no line is ever split between two parses.
// The next window starts at the first unparsed byte, rounded down to the granularity.
Stats parse_windowed(const MappableFile& file, std::uint64_t window) {
    window = std::max(window, 2 * file.granularity());
    Stats total;
    std::uint64_t pos = 0;                                  // first unparsed byte
    while (pos < file.size()) {
        std::uint64_t base = pos - pos % file.granularity();
        std::uint64_t len  = std::min(window, file.size() - base);
        MappableFile::View view(file, base, static_cast<std::size_t>(len));

        std::string_view text = view.bytes().substr(static_cast<std::size_t>(pos - base));
        bool last = base + len == file.size();
        std::size_t cut = last ? text.size() : text.rfind('\n') + 1;   // npos + 1 == 0
        if (cut == 0) throw std::runtime_error("line longer than the window");

        merge(total, parse_readings(text.substr(0, cut)));
        pos += cut;
    }                                                       // view unmapped here
    return total;
}

#ifdef _WIN32
int wmain(int argc, wchar_t** argv) {
    const wchar_t* path = argc > 1 ? argv[1] : nullptr;
    std::uint64_t mb    = argc > 2 ? std::wcstoull(argv[2], nullptr, 10) : 256;
#else
int main(int argc, char** argv) {
    const char* path = argc > 1 ? argv[1] : nullptr;
    std::uint64_t mb = argc > 2 ? std::strtoull(argv[2], nullptr, 10) : 256;
#endif
    if (!path) { std::fprintf(stderr, "usage: mmap_windowed <file> [window MB]\n"); return 2; }
    try {
        MappableFile file(path);
        Stats s = parse_windowed(file, mb << 20);
        std::printf("rows=%zu bad=%zu sum=%.3f max=%.3f\n", s.rows, s.bad, s.sum, s.max);
    } catch (const std::exception& e) {
        std::fprintf(stderr, "error: %s\n", e.what());
        return 1;
    }
    return 0;
}

Points to note in the windowed version:

• One source file, both platforms. Unlike the whole-file examples, the file stays open for the duration (each window needs it), so MappableFile owns the descriptor or the file and section handles, and View is the per-window RAII object.
• The constructor uses a function-try-block to close whatever was opened before a failure, because a destructor doesn't run for an object whose constructor threw.
• Alignment is handled by rounding down, not by requiring the caller to align: base = pos - pos % granularity, then skip pos - base bytes into the view.
• A line longer than the window is an error, because there's nowhere to cut. Size the window well above your longest possible record. 64 MB to 1 GB windows are typical; even 1 MB windows ran at nearly full speed in testing (4.6 s versus 4.3 s on Windows).
• The last view isn't cut, so a final line without a trailing newline is still parsed.

Measured: one 4.8 GB file, three strategies

A 4.8 GB, 28-million-line CSV with malformed rows, CRLF lines and some 300 KB lines mixed in. Every run produced exactly the expected result (rows=27164550 bad=554988 sum=429081621.500). Measured on a 16-thread machine running Windows 11, natively and under WSL2 with the file on ext4, with the file already in the page cache:

Two things stand out. The time is identical across strategies: the parser, not the mapping, is the bottleneck. And Windows trimmed the whole-file working set to about 2 GB on its own, while Linux let it grow to the full file, despite MADV_SEQUENTIAL. Don't rely on either behaviour; if footprint matters, release pages explicitly.

Going faster: parse in parallel

With the data cached, a single-threaded parser ran at about 1.1 to 1.2 GB/s, far below what memory (or a modern NVMe drive) can deliver. A single mapping can be shared by any number of threads with no locking, since it's read-only. So the remaining speedup is parallelism: split the view into ranges that end on line boundaries, parse each on its own thread, and merge.

// parse_parallel.hpp: split one mapping at line boundaries and parse the pieces concurrently
#pragma once
#include <algorithm>
#include <string_view>
#include <thread>
#include <vector>
#include "parse_readings.hpp"

inline Stats parse_parallel(std::string_view whole, unsigned threads) {
    threads = std::max(threads, 1u);
    std::vector<std::string_view> ranges;
    std::size_t pos = 0;
    for (unsigned i = 1; i <= threads && pos < whole.size(); ++i) {
        std::size_t end = (i == threads) ? whole.size()
                                         : std::max(pos, whole.size() / threads * i);
        if (end < whole.size()) {                   // extend to the end of that line
            std::size_t nl = whole.find('\n', end);
            end = (nl == std::string_view::npos) ? whole.size() : nl + 1;
        }
        ranges.push_back(whole.substr(pos, end - pos));
        pos = end;
    }

    std::vector<Stats> parts(ranges.size());
    std::vector<std::thread> pool;
    for (std::size_t i = 0; i < ranges.size(); ++i)
        pool.emplace_back([&parts, &ranges, i] { parts[i] = parse_readings(ranges[i]); });
    for (auto& t : pool) t.join();

    Stats total;
    for (const Stats& p : parts) {                  // merge in a fixed order
        total.rows += p.rows;
        total.bad  += p.bad;
        total.sum  += p.sum;
        total.max   = std::max(total.max, p.max);
    }
    return total;
}

Same file, same correct result at every thread count. Those numbers are for a hot cache. From cold storage, the parse becomes I/O-bound and threads mostly help by keeping more page faults in flight; there, a background MADV_WILLNEED or PrefetchVirtualMemory on the range each thread will need next does more than extra threads. Two caveats: summing floating-point values in a different grouping can change the last bits of the result (merge in a fixed order, as above, to at least keep it deterministic for a given thread count), and parallel parsing combines naturally with the sliding window if each thread maps its own range.

Page tables, faults and TLB reach

• Page tables cost about 0.2% of what you touch. Each resident 4 KB page needs an 8-byte entry: 9.4 MB for the 4.8 GB file, about 2 GB if a process touches all of a 1 TB file. The release-behind and window strategies keep this small too.
• Faults are per page. 4.8 GB is 1.2 million pages. Linux's fault-around (16 pages per fault by default) and readahead cut the real number dramatically, and PrefetchVirtualMemory does the same job on Windows.
• Huge pages mostly don't apply. They would cut both page-table size and TLB misses, but Windows supports large pages only for pagefile-backed sections (SEC_LARGE_PAGES), and Linux support for huge pages in ordinary disk-file mappings is limited and depends on file system and kernel version. Plan for 4 KB pages.
• Readahead window. Linux's default readahead per device is often 128 KB (/sys/block/<dev>/queue/read_ahead_kb). For cold sequential scans of huge files on fast storage, raising it to a few MB can help more than any code change.

Files that grow, and files with holes

• Growing files (logs, captures). A view covers the file's size when it was mapped. On Linux, a MAP_SHARED mapping may be longer than the file: pages entirely past EOF raise SIGBUS until the file grows, then become valid without remapping. On Windows a file-backed section's size is fixed when it's created, so to see new data you create a new section and view. In both cases, re-check the file size before reading past the last known end.
• Sparse files. Unwritten regions (holes) read as zero bytes. They cost no disk I/O but do consume page cache when touched, and a parser looking for '\n' will see one enormous "line" of NULs: the sliding window rejects it as longer than the window, and the whole-file parser counts it as one bad row. Decide explicitly how your format should treat zero runs.

A Note on 32-bit Builds

Everything above assumes 64-bit. In a 32-bit process, memory-mapped I/O still works, but large files need different handling:

• Address space is tiny and fragmented. 2 GB of user space by default on 32-bit Windows (3 GB with /LARGEADDRESSAWARE and the boot option; 4 GB for a large-address-aware process under WOW64 on 64-bit Windows), about 3 GB on 32-bit Linux. DLLs, heaps and thread stacks fragment it, so the largest contiguous free block is often well under 1 GB. A 1.5 GB view can fail to map even though "2 GB is free".
• The sliding window becomes mandatory for anything beyond a few hundred MB. The windowed example above works unchanged once the static_assert is removed: its offsets are already std::uint64_t, and only the window length is a size_t. Keep windows in the 16 to 64 MB range.
• Sizes no longer fit in size_t. Converting a 64-bit file size to a 32-bit size_t silently truncates: a 4.5 GB file becomes a 512 MB view, and your parser quietly stops early. Check before converting:

if (static_cast<std::uint64_t>(size.QuadPart) > SIZE_MAX) {
    ::CloseHandle(file);
    fail("file larger than address space", ERROR_FILE_TOO_LARGE);   // map a window instead
}

• 32-bit Linux needs large-file support switched on. Without -D_FILE_OFFSET_BITS=64, off_t is 32-bit: open or fstat fails with EOVERFLOW on files over 2 GB, and mmap can't express offsets beyond 2 GB. With it defined, glibc routes to the 64-bit variants. On 64-bit Linux this is the default and the macro is harmless.
• Windows 32-bit offsets are fine in the API (high/low DWORD pairs), so the windowed example's offset handling carries over as-is. Only the view length is constrained.

Writing Through a Mapping

A mapping can't extend a file: writing past the end of the file is not an append, it's a fault. To produce output through a mapping, size the file first, then map it:

// Linux: create and size, then map shared
int fd = ::open(path, O_RDWR | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
::ftruncate(fd, total_size);                       // file is now total_size zero bytes
char* out = static_cast<char*>(::mmap(nullptr, total_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0));
// ... write into out[0 .. total_size) ...
::msync(out, total_size, MS_SYNC);                  // only if you need durability now
::munmap(out, total_size);
::fsync(fd);
::close(fd);

// Windows: CreateFileMappingW with an explicit size grows the file for you
HANDLE file = ::CreateFileW(path, GENERIC_READ | GENERIC_WRITE, 0, nullptr, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr);
HANDLE map  = ::CreateFileMappingW(file, nullptr, PAGE_READWRITE, size_hi, size_lo, nullptr);
char*  out  = static_cast<char*>(::MapViewOfFile(map, FILE_MAP_WRITE, 0, 0, 0));
// ... write ...
::FlushViewOfFile(out, 0);                         // start writeback of dirty pages
::UnmapViewOfFile(out);
::FlushFileBuffers(file);                          // wait for it to hit the disk
::CloseHandle(map);
::CloseHandle(file);

(Error checks omitted for brevity; the read example above shows the pattern.) If you don't know the final size up front, either over-allocate and truncate at the end (ftruncate / SetFileInformationByHandle after unmapping), or grow in chunks: extend the file, then mremap on Linux, or unmap and re-map on Windows, where every pointer into the old view becomes invalid.

Two durability facts that surprise people: without an explicit msync / FlushViewOfFile, dirty pages are written back whenever the OS decides (Linux defaults to around 30 seconds); and the write-back order is not the order you stored in. A crash can leave page 7 updated and page 3 not. Anything that needs crash consistency needs a journal or copy-on-write scheme on top, which is why databases are so careful here.

Drawbacks and Issues

I/O errors become signals and exceptions

With read(), a failing disk, a yanked USB drive or a dropped network share gives you -1 and an errno. With a mapping there's no call to return from, so the error is delivered as SIGBUS on Linux or a structured exception, EXCEPTION_IN_PAGE_ERROR, on Windows. Neither is a C++ exception; neither is caught by try/catch. By default both kill the process.

On Windows you can guard the access with SEH. It must live in a function with no C++ objects that need unwinding, so isolate it:

// MSVC only. Returns false if the mapped file couldn't be read.
static bool parse_guarded(const char* data, size_t size, Stats* out) {
    __try {
        *out = parse_readings({data, size});
        return true;
    } __except (GetExceptionCode() == EXCEPTION_IN_PAGE_ERROR
                  ? EXCEPTION_EXECUTE_HANDLER : EXCEPTION_CONTINUE_SEARCH) {
        return false;
    }
}

On Linux the equivalent is a SIGBUS handler plus sigsetjmp/siglongjmp, which is legal but fragile: it skips destructors, isn't thread-friendly without per-thread jump buffers, and is hard to make correct in a library. Most code accepts the crash for local files and avoids mapping files on unreliable storage.

Someone else truncates the file

This is the most common source of SIGBUS in practice. If another process shrinks the file while you have it mapped, any access to the pages past the new end-of-file faults, even though your mapping and your size_ still say they're valid. Log files rotated with truncate, editors that save by truncating and rewriting, and download tools that pre-allocate then shrink all do this.

Windows prevents the problem by refusing it: SetEndOfFile on a file with a mapped view fails with ERROR_USER_MAPPED_FILE, and the file generally can't be deleted while mapped. That protects your reader but surprises the other program, so Windows users see "file is in use" errors instead. On Linux, deleting (unlinking) a mapped file is harmless; the data lives until the last mapping is gone.

MAP_PRIVATE is not a snapshot

It's tempting to assume a private mapping freezes the file's contents. It doesn't: pages you haven't written are still shared with the page cache, and POSIX leaves it unspecified whether other processes' writes to the file become visible through them. On Linux they generally do. If you need a stable snapshot of a file that might change, copy it, or lock it with an advisory lock that every writer also respects.

Untrusted input changes under you

Related, and a security issue: if you validate a header in a mapped file and then trust it, an attacker who can write the file can change the bytes between your check and your use (a TOCTOU race), and your parser reads the new values. Treat every read from a mapping of an attacker-writable file as untrusted, every time, or read it into private memory first. The same applies to shared-memory IPC with a less-privileged process.

The buffer isn't null-terminated

Covered above, but it bites constantly: any C string function on a mapped buffer can run off the end. On both platforms, if the file size isn't a page multiple, the tail of the last page reads as zeros, which hides the bug in testing until a file of exactly 4096 bytes comes along. Use length-bounded functions everywhere.

Alignment and pointers

• Offsets must be aligned: page size on POSIX, 64 KB allocation granularity on Windows. To map from byte offset off, map from off & ~(gran - 1) and add the remainder to the returned pointer.
• Pointers are per-mapping. A mapping may land at a different address next time, so data structures stored in a file must use offsets, not raw pointers.

Performance cliffs

• Page-fault stalls are invisible in the code. A loop that looks like pure computation can block on disk at any memory access. That's a problem in latency-sensitive threads (UI threads, audio callbacks, game loops) and in async code, where a major fault blocks the whole event loop thread, not just one task.
• Unmapping is expensive with many threads. munmap and UnmapViewOfFile must invalidate TLB entries on every CPU that ran the process (a TLB shootdown). Mapping and unmapping lots of small regions from a heavily threaded process can bottleneck on this.
• Memory accounting looks alarming. Mapped file pages count toward the process's resident set (RSS / working set), so a process that maps a 10 GB file can appear to use 10 GB. Those pages are clean and reclaimable, but monitoring tools and container limits don't always see it that way. Very Large Files shows how to keep the footprint to a few MB.

Network and special file systems

NFS, SMB shares and FUSE file systems support mapping with weaker coherence guarantees (Windows explicitly does not guarantee coherence for remote files) and much higher odds of the I/O error path firing. Pipes, sockets, most character devices and many /proc files can't be mapped at all; check that the target is a regular file, as the Linux example does.

Databases are a special case

Using mmap as a database buffer pool looks attractive and has been tried many times. The 2022 CIDR paper "Are You Sure You Want to Use MMAP in Your Database Management System?" (Crotty, Leis, Pavlo) lays out why it usually goes wrong: no control over when dirty pages are written (breaking write-ahead logging), no way to handle I/O errors gracefully, blocking page faults, and TLB shootdowns limiting scalability. MongoDB's original MMAPv1 engine and early InfluxDB both moved away from it; LMDB is the well-known example built around it on purpose, with a design shaped by those constraints.

When to Reach for It

Good fits: read-only indexes and lookup tables, asset packs, binary formats with offset tables, large log or CSV files you'll scan with a contiguous parser, read-mostly data shared across many worker processes, and fast IPC. Poor fits: streaming input you'll touch once and discard (especially from network storage), files other programs actively rewrite, anything that needs ordered crash-safe writes, and large numbers of small files.

Checklist

What it is:
  [ ] Mapping makes the file's page-cache pages appear in your address space, with no copy
  [ ] Pages load on demand via page faults: minor if cached, major if read from disk
  [ ] MAP_SHARED writes reach the file; MAP_PRIVATE writes are copy-on-write

Why use it:
  [ ] Random or repeated access, contiguous parsing, multi-process sharing, IPC
  [ ] Not automatically faster for one sequential pass: measure cold and hot

OS facilities:
  [ ] Win32: CreateFileW, CreateFileMappingW, MapViewOfFile, UnmapViewOfFile
  [ ] Linux: open, fstat, mmap, munmap; madvise for hints, msync for durability
  [ ] Offsets aligned to page size (POSIX) or 64 KB allocation granularity (Windows)
  [ ] Close file handles right after mapping; the view keeps the file alive

Libraries:
  [ ] mio for a lightweight cross-platform map; Boost.Interprocess for persistence and IPC

Parsing example:
  [ ] Keep the parser platform-neutral: take std::string_view, not a mapping
  [ ] Special-case empty files; both native APIs fail on a zero-length mapping
  [ ] Never use strtod, sscanf, strlen or strchr on a mapped buffer: use from_chars and memchr
  [ ] Capture errno / GetLastError before cleanup calls overwrite it
  [ ] Build 64-bit and static_assert it; a whole file of any size then fits in one view

Very large files:
  [ ] Mapping reserves address space, not RAM; files bigger than RAM map and parse fine
  [ ] Keep sizes and offsets 64-bit: size_t / uint64_t, never int or Windows long
  [ ] Expect RSS / working set to grow toward the file size with a plain whole-file map
  [ ] Release pages behind the parser (MADV_DONTNEED / VirtualUnlock) to stay at a few MB
  [ ] Or use a sliding window: base rounded down to granularity, cut at the last newline
  [ ] Map read-only: writable private views charge commit for the whole view
  [ ] Parse in parallel over one shared read-only mapping, split on line boundaries
  [ ] Don't MAP_POPULATE a file larger than RAM; prefetch the next range instead
  [ ] Decide how the parser treats NUL runs from sparse-file holes

32-bit builds:
  [ ] Sliding window is mandatory; keep windows at 16 to 64 MB
  [ ] Check file size against SIZE_MAX before converting to size_t
  [ ] On 32-bit Linux, compile with -D_FILE_OFFSET_BITS=64

Writing through a mapping:
  [ ] Size the file before mapping it; writes past EOF fault, they don't append
  [ ] msync + fsync, or FlushViewOfFile + FlushFileBuffers, for durability
  [ ] Write-back order is not store order: add a journal if you need crash consistency

Drawbacks and issues:
  [ ] I/O errors arrive as SIGBUS or EXCEPTION_IN_PAGE_ERROR, not return codes
  [ ] Another process truncating the file causes SIGBUS on Linux
  [ ] MAP_PRIVATE is not a snapshot; re-validate untrusted mapped data on every read
  [ ] Page faults stall threads invisibly; avoid mapping on latency-critical threads
  [ ] Avoid network file systems and files you don't control