A PDF parser's first useful read is at the wrong end of the file. The format puts the startxref pointer in the final bytes, so processing a 1.8 GB archive begins with a seek to the tail, a one-kilobyte read, then a hop to wherever the cross-reference table says the document catalogue lives. From there the parse is a random walk across the whole byte range. Everything buffered IO is good at — sequential read-ahead behind the file pointer — is aimed at a workload PDF does not have
The first version of this article claimed a memory-mapped file solves the 32-bit out-of-memory failure that TMemoryStream hits on a 2 GB input. That claim is wrong, and the way it is wrong points at the real fix: a sliding mapping window. What follows is the access pattern, the corrected 32-bit story with a compilable windowed mapper, and the syscall arithmetic on a 1.8 GB, 300,000-object test file
Why PDF layout defeats buffered reads
Three structural facts shape the IO pattern. First, navigation is offset-driven: the cross-reference table maps every object number to an absolute byte position, and nothing requires those positions to be ordered. After years of incremental updates, object 4102 can sit at offset 1.6 GB while object 4103 sits at 30 KB. A TFileStream loop turns every fetch into a Seek plus a Read, two kernel transitions, with a buffer that contributes nothing because the next fetch is hundreds of megabytes away
Second, object streams (ISO 32000-1 §7.5.7) pack dozens or hundreds of small dictionaries into one deflated container. Fetching one 300-byte page dictionary can mean reading and inflating a 100 KB cluster. The flip side: objects written together tend to be read together, so a buffer sized to the cluster serves the next dozen fetches for free — the most exploitable regularity in the format
Third, linearisation. A linearised file front-loads the first page and a hint table so consumers can read it front to back. Gigabyte archives are almost never linearised: linearisation is destroyed by the same incremental updates and merges that made the file large. Plan for the hostile case: long hops, no ordering, tail-first entry
The 32-bit story, corrected
A 32-bit Windows process has 2 GB of user address space, and MapViewOfFile with a byte count of zero asks for one contiguous reservation the size of the file. For a 2 GB input that reservation cannot succeed: after the EXE, scattered DLLs, and thread stacks, the largest free contiguous block in a typical 32-bit Delphi process sits somewhere between 700 MB and 1.4 GB. The call fails with ERROR_NOT_ENOUGH_MEMORY, the same wall TMemoryStream.LoadFromFile hits, just moved from committed RAM to address-space reservation. A full-file mapping is no fix on 32-bit, just the same failure behind better-sounding API names
The fix is separating the two things a mapping does. CreateFileMapping creates the section object and costs no address space at all, whatever the file size. Only MapViewOfFile spends address space, and nothing forces it to map the whole section: it takes a 64-bit starting offset and a view length. Create the section once, map a 64 to 256 MB view over the region being parsed, unmap before sliding on: the address-space cost is one window, not one file. One constraint: view offsets must be multiples of SYSTEM_INFO.dwAllocationGranularity, 64 KB in practice, so a request for offset 1,000,000 gets rounded down to 983,040 and the caller's pointer adjusted forward by the difference
A sliding-window mapper in Delphi
The class below wraps the whole discipline: one section object, one live view, granularity realignment, and reads that cross a window boundary handled by growing that one view instead of stitching two
uses
Winapi.Windows, System.SysUtils;
type
TWindowedFileMapper = class
private
FFile: THandle;
FMapping: THandle;
FFileSize: Int64;
FGranularity: DWORD; // SYSTEM_INFO.dwAllocationGranularity
FWindowSize: NativeUInt; // default view size
FViewBase: PByte; // base of the current view (aligned)
FViewOffset: Int64; // file offset FViewBase corresponds to
FViewSize: NativeUInt; // bytes mapped in the current view
procedure Unmap;
public
constructor Create(const FileName: string;
WindowSize: NativeUInt = 64 * 1024 * 1024);
destructor Destroy; override;
function Map(Offset: Int64; Size: NativeUInt): PByte;
procedure ReadBytes(Offset: Int64; var Buffer; Count: NativeUInt);
property FileSize: Int64 read FFileSize;
end;
constructor TWindowedFileMapper.Create(const FileName: string;
WindowSize: NativeUInt);
var
Info: TSystemInfo;
begin
inherited Create;
FFile := CreateFile(PChar(FileName), GENERIC_READ, FILE_SHARE_READ, nil,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
if FFile = INVALID_HANDLE_VALUE then
RaiseLastOSError;
if not GetFileSizeEx(FFile, FFileSize) then
RaiseLastOSError;
// The section object reserves no address space, whatever the file size
FMapping := CreateFileMapping(FFile, nil, PAGE_READONLY, 0, 0, nil);
if FMapping = 0 then
RaiseLastOSError;
GetSystemInfo(Info);
FGranularity := Info.dwAllocationGranularity; // 64 KB in practice
FWindowSize := WindowSize;
end;
destructor TWindowedFileMapper.Destroy;
begin
Unmap;
if FMapping <> 0 then CloseHandle(FMapping);
if FFile <> INVALID_HANDLE_VALUE then CloseHandle(FFile);
inherited;
end;
procedure TWindowedFileMapper.Unmap;
begin
if FViewBase <> nil then
begin
UnmapViewOfFile(FViewBase);
FViewBase := nil;
FViewSize := 0;
end;
end;
function TWindowedFileMapper.Map(Offset: Int64; Size: NativeUInt): PByte;
var
AlignedOffset: Int64;
Delta, MapSize: NativeUInt;
begin
if (Offset < 0) or (Offset + Int64(Size) > FFileSize) then
raise ERangeError.CreateFmt(
'Map request at %d for %d bytes is outside the file',
[Offset, Int64(Size)]);
// Fast path: the requested range already sits inside the live view
if (FViewBase <> nil) and (Offset >= FViewOffset) and
(Offset + Int64(Size) <= FViewOffset + Int64(FViewSize)) then
Exit(FViewBase + NativeInt(Offset - FViewOffset));
Unmap; // slide: never hold two views at once
// Views must start on an allocation-granularity boundary
AlignedOffset := Offset - (Offset mod FGranularity);
Delta := NativeUInt(Offset - AlignedOffset);
MapSize := FWindowSize;
if MapSize < Size + Delta then // request straddles the window end:
MapSize := Size + Delta; // grow this one view to cover it
if AlignedOffset + Int64(MapSize) > FFileSize then
MapSize := NativeUInt(FFileSize - AlignedOffset); // clamp at EOF
FViewBase := MapViewOfFile(FMapping, FILE_MAP_READ,
DWORD(AlignedOffset shr 32), DWORD(AlignedOffset and $FFFFFFFF),
MapSize);
if FViewBase = nil then
RaiseLastOSError;
FViewOffset := AlignedOffset;
FViewSize := MapSize;
Result := FViewBase + NativeInt(Delta);
end;
procedure TWindowedFileMapper.ReadBytes(Offset: Int64; var Buffer;
Count: NativeUInt);
begin
Move(Map(Offset, Count)^, Buffer, Count);
end;Two details carry the weight. The fast path at the top of Map returns a pointer with no kernel transition when the requested range already sits inside the live view; thanks to object-stream clustering this is the common case and where the savings come from. And a request that straddles the end of the default window grows MapSize for that one view rather than stitching two, which keeps ReadBytes a one-liner and callers free of partial-read loops
Window size is a forgiving knob: at 64 MB a full sweep of a 1.8 GB file is 29 views, at 256 MB it is 8 but each reservation is harder to place in a fragmented 32-bit space, and below about 16 MB hop-heavy files remap often enough to notice. Anywhere in the 64 to 256 MB range, map traffic is statistical noise
Counting the syscalls
Now the arithmetic. Test file: 1.8 GB, 300,000 indirect objects averaging about 600 bytes of payload. A per-object parser fetches each one with SetFilePointerEx plus a 4 KB ReadFile: 600,000 kernel transitions. A cached read syscall round-trips in roughly 1.5 μs on current x64 hardware, so that is 600,000 × 1.5 μs ≈ 0.9 seconds of pure kernel overhead before parsing a single byte — the warm-cache best case. Cold, each hop is a device operation: at the ~20 μs effective latency of NVMe 4 KB random reads, 300,000 of them cost about 6 seconds of device time; on SATA-class storage, minutes
The reads also move the wrong data: 300,000 × 4 KB pushes 1.2 GB through user buffers to deliver roughly 180 MB of payload — six-fold amplification, every byte copied kernel to user
A read-ahead buffer sized to the object-stream clusters is the first honest improvement: one 256 KB read per cluster instead of one per object cuts the transition count by one to two orders of magnitude. It is also the right tool where mapping is awkward, usually network shares
The windowed mapper goes further. A full sweep is 29 MapViewOfFile and 29 UnmapViewOfFile calls, 58 explicit transitions against 600,000. A real xref-driven parse is not a clean sweep, but the fast path absorbs every fetch inside the live window; a metadata-indexing pass over the test archive settled at a few hundred remaps. Mapping does not remove kernel work: it converts explicit syscalls into page faults the memory manager resolves in multi-page clusters, straight from the file cache with no user-space copy, and regions never touched cost nothing. End to end, the indexing pass went from 23 s cold and 7.1 s warm with per-object reads to 6.5 s cold and 1.9 s warm with the mapper; what remains is zlib inflate, not IO
Where FILE_FLAG_NO_BUFFERING fits
FILE_FLAG_NO_BUFFERING bypasses the system cache in exchange for hard alignment rules: offsets, lengths, and buffer addresses all sector-aligned. It earns its keep on single-pass sequential jobs that would otherwise flood the cache with bytes nobody reads twice — a batch re-serialisation that rewrites the whole archive, or a linearisation pass over finished output. With 4 to 8 MB aligned buffers it approaches device sequential bandwidth without polluting the cache
It is exactly wrong for parsing. Random xref hops through an unbuffered handle turn every 300-byte dictionary fetch into a full physical read with no cache to absorb the second visit — and PDF parsing revisits regions constantly, because different pages resolve into the same object streams. Unbuffered IO for the sequential rewrite, mapped or cached IO for the random parse; the flag is per-handle, so one pipeline can hold both on the same file
64-bit, working sets, and the write side
On a 64-bit build the address-space objection disappears: pass the file size as the window and the class above degenerates into a single full mapping. The catch in long-running services: read-only file-backed pages charge no commit, so commit counters stay calm, but every page touched joins the working set; parse most of 1.8 GB and the working set grows to match, evicting everything else. Bounded windows put a ceiling on that, so the sliding pattern stays the right default even where address space is free
On the write side, the cheapest IO is the IO never issued. PDF's incremental update mechanism (ISO 32000-1 §7.5.6) appends the changed objects and a new cross-reference section after the original bytes, which never move. Stamping one page onto the 1.8 GB archive appends tens of kilobytes; a full rewrite moves all 1.8 GB, five orders of magnitude apart, and the append is pure sequential output at the tail
Where the losLab libraries fit
Both losLab PDF libraries ship this discipline as API surface. The HotPDF Direct File API reads page counts and structure through a file handle without building the object tree, copies and decrypts at file level, and writes deltas through BeginIncrementalUpdate — the append-only strategy above, packaged. PDFlibPas takes the same route with its Direct Access layer: a streaming reader that walks the cross-reference table in place, fetches objects lazily, extracts page ranges file to file, and persists edits as incremental revisions. If you are writing your own parser, the mapper class is yours to take; if you are running a document pipeline, let the library keep the window honest
Note: Optimised IO handling for gigabyte-scale documents is built directly into the HotPDF VCL Component for Delphi and C++Builder