You want one dictionary out of a 2 GB PDF and the tool first expands the whole cross-reference table into an array sized by the trailer /Size. PDFiumPas replaces that step with a sparse lazy object index: it keeps only the xref section descriptors, resolves a single object number on demand through bounded windows, and caches just the entries you actually touched
The old shape of this code in FPdfCompress was honest but expensive. ApplyDefaultOpenAction read the complete file into one TBytes, then allocated a dense TPdfActiveXrefEntries array with one slot per object number up to /Size. Two things went wrong at scale. The read cost grew linearly with document size even when the caller wanted four dictionaries, and the dense array collided with the parser budget: TPdfParserResourceBudget.Default sets MaxObjects to 4,000,000, so a perfectly valid file whose highest object number sits above that ceiling was rejected on a memory argument rather than a correctness one
Why does the PDFium public API not answer this question?
Because the information exists inside PDFium but never crosses the C boundary. CPDF_Parser maintains the cross-reference table, the object stream membership and the revision precedence internally, yet the published headers expose no entry point that takes an object number and returns its raw offset, its generation, which revision won, or which ObjStm it lives in. The save side is equally closed: FPDF_SaveAsCopy and FPDF_SaveWithVersion only hand you a sequential write callback. Any byte-level patch to a catalog after a native save therefore has to be built in the Pascal layer, which is why PDFiumPas parses these structures itself instead of reusing the DLL
What does the sparse index actually keep in memory?
Descriptors, not entries. For a classic table (ISO 32000-1 §7.5.4) a TPdfSparseXrefSubsection stores the first object number, the object count, the byte offset where the entry rows begin and the measured entry width. The entries themselves stay in the file. The width is measured from the first row rather than assumed to be 20 bytes, because producers disagree about line endings; PDFiumPas accepts 18 to 64 and rejects anything outside that band, along with any subsection whose declared count would run past the end of the stream. For a cross-reference stream (§7.5.8) the section holds the three /W field widths, each constrained to 0 through 8, the flattened /Index pairs, and the decoded entry bytes, whose expected length is computed from /W and /Index before a single byte is inflated
The whole index is built by Initialize from a tail window of at most 1 MiB, which is where startxref is found, and every subsequent object read uses a 1 MiB object window. The raw stream ceiling is 64 MiB and a single xref line may not exceed 1024 bytes. If you have read our note on validating object and cross-reference streams with PDFiumPas, the same field-width discipline applies here, only now it is used to address one entry instead of to audit a whole table
uses
FPdfCompress;
var
Source: TFileStream;
Revision: TPdfSparseRevisionInfo;
begin
Source := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
try
{ walks startxref, the /Prev chain and the catalog only }
if ReadPdfSparseRevisionInfo(Source, Revision) then
begin
Writeln('root ', Revision.RootObjectNumber, ' ',
Revision.RootGeneration);
Writeln('max obj ', Revision.MaximumObjectNumber);
Writeln('xref str ', Revision.UsesXrefStream);
Writeln('encrypted ', Revision.HasEncrypt);
Writeln(string(Revision.CatalogDictionary));
end;
finally
Source.Free;
end;
end;
How does one lookup reach one object?
By arithmetic, in both layouts. A classic subsection has fixed-width rows, so the address of an entry is the subsection start plus the object offset times the measured width; PDFiumPas then reads that one line, parses the ten-digit offset and five-digit generation, checks the generation against the 65535 ceiling from §7.5.4, and classifies the trailing keyword as axkDirect or axkFree. A cross-reference stream needs one more step because /Index subsections are concatenated in the decoded byte run, so the index accumulates the counts of preceding subsections before multiplying by the summed /W width. Type 1 yields an offset, type 2 yields an object stream number and a member index, and anything else becomes axkUnknown rather than a guess
{ classic table, ISO 32000-1 section 7.5.4 }
EntryOffset := Subsection.EntryOffset +
Int64(ObjectNumber - Subsection.FirstObject) * Subsection.EntryWidth;
{ cross-reference stream, ISO 32000-1 section 7.5.8 }
EntryWidth := Section.Widths[0] + Section.Widths[1] + Section.Widths[2];
EntryPosition := Integer((PriorCount + ObjectNumber -
Section.IndexValues[I]) * EntryWidth);
Nothing in either path is proportional to /Size. That is the whole point of the rewrite: the trailer size value is carried forward as metadata and used when writing the incremental revision, but it never drives an allocation. The regression suite pins this with a fixture whose page tree lives at object 1,000,000,000 and 1,000,000,001 under a trailer declaring /Size 1000000002. The old dense implementation refused that file; the sparse index resolves both references and preserves the declared size in the output trailer
Hybrid revisions, /Prev chains and the guards around them
Revision precedence is where a naive lazy index goes wrong. PDFiumPas walks the chain from startxref in newest-first order and stops a lookup at the first section that answers, which reproduces the precedence rule without materialising a merged table. Hybrid-reference files (§7.5.8.4) are handled inside the classic branch: when the trailer carries an /XRefStm, the supplemental stream section is registered before the classic section that referenced it, so compressed objects invisible to the plain table are still found while the classic entries keep their standing. Older revisions are then followed through /Prev
Two guards bound that walk, and both matter on damaged files. Every offset visited is recorded, so a /Prev pointing back into the chain terminates instead of spinning, and the traversal depth is capped by MaxRecursionDepth, which defaults to 1024. The encryption flag is accumulated across the entire chain rather than read from the newest trailer alone, because a document whose latest trailer omits /Encrypt can still be encrypted further back; callers that append revisions rely on that flag to refuse writing plaintext objects into an encrypted file
Type-2 entries: why the object stream waits
A type-2 entry names an object stream, and PDFiumPas does not touch that stream until a caller asks for a member of it. When it finally does, /Type /ObjStm is verified, /N is checked against the object budget and /First against the decoded-byte ceiling, and /N is sanity-checked against /First since each header pair needs at least four bytes. Only then is the stream inflated, and the header scan stops at the requested member and its successor rather than building a full member table. One decoded object stream is retained at a time, which is the right trade when a page tree branch clusters into a single ObjStm; our write-up on object stream and predictor decoding in Delphi covers what happens inside that inflate step (§7.5.7)
var
Reader: TPdfSparseDictionaryReader;
Generation: Integer;
Dict: AnsiString;
begin
{ one retained index, many generation-aware reads }
Reader := TPdfSparseDictionaryReader.Create(Source);
try
if Reader.Valid and
Reader.ReadLatestDictionary(PageObjectNumber, Generation, Dict) then
HandlePage(PageObjectNumber, Generation, Dict);
finally
Reader.Free; { Source stays yours }
end;
end;
Where the cache stops making promises
The index is a snapshot, and it is worth being blunt about that. Sections are parsed once in Initialize; if the underlying stream is modified afterwards, every cached entry is stale and the class will not notice. TPdfSparseDictionaryReader holds the index for the caller-owned lifetime of the source, which is exactly what a recursive walk over a page tree wants and exactly what you must not do across a rewrite. The entry cache is a flat array searched linearly and it stores negative results too, so a few hundred lookups are cheap and a few hundred thousand are not. ReadDictionary demands an exact generation match while ReadLatestDictionary resolves the active one, and the difference is deliberate: reference resolution needs the former, catalog inspection needs the latter. Where these bounds cannot be honoured, the surrounding units fall back to the legacy whole-file parser instead of narrowing the set of files that still work, a pattern we also use for on-demand streaming of large PDFs
Cross-compiler regressions cover the same behaviour on all three toolchains, including an assertion that a 2 MiB source never sees a single read larger than 1 MiB. If you maintain Delphi, C++Builder or Lazarus code that touches PDF structure directly and you are tired of paying whole-file parse costs for four dictionaries, the sparse index and the public seam around it ship in the PDFiumPas Delphi PDFium component