Lazy Object Stream Members and Full PDF Rewrites in Delphi

When the HotPDF Delphi Component loads a PDF 1.5 file with LoadFromFile, it does not parse the objects packed inside /Type /ObjStm containers. It records where each compressed member lives and parses it only when something asks for it. That lazy invariant is what keeps load time proportional to what you actually touch, and it is also the reason a full rewrite has to do one extra job before any bytes go out: expand every member that is still unparsed, because the rewrite is about to throw away the containers those members live in

The symptom that motivated this note is easy to describe and unpleasant to debug. Load a file whose fonts, color spaces, and structure tree sit in object streams, run it through the BeginDoc and EndDoc generation pair, and the output opens without complaint. Page count is right, text is visible on the pages you spot-check. Then a colleague opens page 40 and the body text renders in a substituted font, or the Extract Text command returns garbage where an ActualText replacement used to be. Nothing crashed. The writer simply serialized an object that was never loaded, and an unloaded object serializes as nothing

What does LoadFromFile actually keep for a compressed object?

For every type-2 cross-reference entry, LoadFromFile keeps a small record in FCompactObjects: the object number, the index of the containing stream in the container table, the member's position inside that stream, and a ParsedObject pointer that starts out nil. The container itself is located, decrypted if the document is encrypted, and inflated, but the member bodies are left as bytes. ISO 32000-1 §7.5.7 defines the container layout that makes this possible: a header of object-number and offset pairs, then the member bodies concatenated after /First, so any single member can be sliced out without touching its neighbors

EnsureCompressedObjectLoaded is the only path that turns a record into an object. It finds the record by object number, and if ParsedObject is already set it returns that cached object and counts a cache hit. Otherwise it reloads the container if it was evicted, computes the member's byte range from the offset table, hands the parser a zero-copy view of that slice, and stores the result back into the record. From then on the object is indirect, carries its real object number, and is registered in the document's object index like any object that was parsed from the file body. The catalog, the info dictionary, the page tree root, and the page objects go through this path at load time because navigation needs them. Fonts, color spaces, ExtGState dictionaries, and structure elements do not, and they stay as records until a page render or a rewrite touches them

You can watch this from outside. GetLoadedObjectStreamCacheInfo reports how many containers exist, how many members were indexed, and how many of those have been parsed so far:

var
  Pdf: THotPDF;
  Info: THPDFObjectStreamCacheInfo;
begin
  Pdf := THotPDF.Create(nil);
  try
    Pdf.LoadFromFile('tagged-report.pdf');
    if Pdf.GetLoadedObjectStreamCacheInfo(Info) then
      Writeln(Format('%d containers, %d members indexed, %d parsed so far',
        [Info.ContainerCount, Info.IndexedObjectCount,
         Info.MaterializedObjectCount]));
  finally
    Pdf.Free;
  end;
end;

On a structure-heavy file the third number is a small fraction of the second right after load. That gap is the whole point of lazy loading, and it is also exactly the set of objects a full rewrite has to go back for

Why does a full rewrite drop fonts that an incremental save keeps?

A full rewrite discards the source file's /ObjStm and /XRef containers and re-serializes the object graph from scratch, so any member whose ParsedObject is still nil has no representation left in the output. An incremental update never has this problem, because it appends new objects after the original bytes and leaves the old containers in place for the previous cross-reference section to address. The difference is not in how the two modes treat fonts. It is in whether the original containers survive to be read by the next viewer

The fix lives in SaveToStream, the serializer that EndDoc drives whether you set FileName or OutputStream. Before it dispatches to any writer branch, it walks FCompactObjects and calls EnsureCompressedObjectLoaded on every entry. If a member cannot be loaded, the save raises rather than continuing, because a rewrite that silently drops a font dictionary is worse than one that stops. The expansion has to sit at that level, above the classic, packed, and linearized branches, and above the linearized route's pruning of reloaded structural streams. An earlier version expanded members only inside SaveLoadedDocument, which covered the loaded-document vocabulary and missed the generation vocabulary entirely. LoadFromFile followed by BeginDoc, page edits, and EndDoc went straight to the writer with every untouched member still unparsed

// Both rewrite vocabularies now expand compact members before any writer runs.
// Loaded-document path:
Pdf.LoadFromFile('quarterly.pdf');
Pdf.SaveLoadedDocument('quarterly-rewritten.pdf');

// Generation path over a loaded file:
Pdf.LoadFromFile('quarterly.pdf');
Pdf.FileName := 'quarterly-stamped.pdf';
Pdf.BeginDoc;
Pdf.CurrentPage.SetFont('Arial', [], 9);
Pdf.CurrentPage.TextOut(40, 20, 0, 'Reviewed 2026-09-11');
Pdf.EndDoc;   // SaveToStream materializes every FCompactObjects entry first

Cached members keep whatever you did to them. An object that was parsed, edited, and marked dirty before the save is returned from the cache with its edits, and a member you deleted keeps its deletion state across repeated saves. The expansion pass is idempotent by construction: it only ever fills nil slots

Why pixel checks on three pages miss the ActualText case

Structure elements are where this bug hides longest. An ActualText entry on a marked-content sequence, defined in ISO 32000-1 §14.9.4, replaces the glyphs for extraction and accessibility but does not affect rendering. If the structure element lives in an object stream and the rewrite loses it, the page still draws correctly, the first, middle, and last pages compare pixel for pixel against the source, and the regression only shows when someone runs text extraction or a screen reader. A rewrite test that only renders pages is not a rewrite test for tagged PDF. Diff the extracted text and the structure tree as well

How does an empty user password change the load?

An empty user password still means the file is encrypted, and object streams in such a file are ciphertext until the file key is recovered. ISO 32000-1 §7.6.3.4 Algorithm 2 derives that key from the password, the /O entry, /P, and the first document identifier, and HotPDF has to run it against the empty string before the type-2 pass can inflate a single container. That is why BeginDoc on a loaded encrypted document calls DecryptLoadedDocument with an empty password before anything else: the object graph must be authenticated and decrypted before a rewrite can begin, regardless of whether the caller intends to protect the output. Output encryption is a separate decision, driven by the caller's protection settings, and BeginDoc restores those settings after the decrypt pass so an encrypted input does not silently turn into an encrypted output

The container policy is read from the /Encrypt dictionary before any password is tried. For /V 1 and 2 every stream is encrypted with the file key. For crypt filters, HotPDF resolves /StmF through /CF: an Identity filter or a /CFM of None means plaintext containers, while V2 and AESV2 mean encrypted ones. The answer lands in FReloadObjectStreamsEncrypted, and it matters for one specific case. When containers are plaintext but strings are not, the members carry encrypted strings that must be decrypted individually, so MaterializeMembersOfPlaintextObjectStreams expands every compact member ahead of the per-object decryption pass. It does nothing when the policy is not yet known and nothing when the containers themselves were encrypted, because members of an encrypted container were already decrypted with it and must never be decrypted twice

What happens when a container cannot be decrypted?

A container that fails decryption is quarantined, not fatal. The type-2 pass records a THPDFObjStmQuarantineInfo entry in FObjStmQuarantine with the container's object number, a THPDFObjStmQuarantineReason, a diagnostic string, and the list of member object numbers the cross-reference had routed into it. osqrDecryptFailed is raised for four distinct situations: no crypt filter could be resolved, the AES-256 or AES-GCM decrypt threw, the legacy RC4 or AES-128 decrypt threw, or no usable file key exists at all. Independent containers keep loading, so a document with one damaged container still opens and still renders every page that does not depend on it

The quarantine list survives parser fallback. If the primary cross-reference load fails and HotPDF reconstructs the object table by scanning the file, the encrypted flag from the first attempt may not survive that reconstruction, but the quarantine records do. That is why BeginDoc checks the quarantine list rather than the encrypted flag: on a loaded document it walks FObjStmQuarantine and raises on the first osqrDecryptFailed entry, naming the container and asking for a reload with a valid password. A rewrite that proceeded past that point would write the members the container was supposed to hold as empty objects and report success. You can run the same check yourself, earlier and with your own policy, through the public accessors:

var
  Info: THPDFObjStmQuarantineInfo;
  I: Integer;
begin
  Pdf.LoadFromFile('vendor-form.pdf');   // empty user password
  for I := 0 to Pdf.GetLoadedQuarantinedObjStmCount - 1 do
    if Pdf.GetLoadedQuarantinedObjStmInfo(I, Info) and
       (Info.Reason = osqrDecryptFailed) then
      raise Exception.CreateFmt(
        'Object stream %d is unreadable (%s); %d members unresolved',
        [Info.ContainerObjNum, String(Info.Diagnostic),
         Length(Info.MemberObjNums)]);
  // safe to rewrite from here
end;

Other quarantine reasons cover the non-cryptographic failures: a container that is not a stream, a missing dictionary, an invalid /N or /First, a stream size outside the accepted range, a decompression failure, a /First pointing past the data, or a member body that decoded but did not parse. Those are worth logging on ingest, since each one names the exact members you will be missing downstream

Why does a rewrite need the original numeric token?

HotPDF stores every numeric object as a Single, and a Single cannot reproduce the source text of a real number. ISO 32000-1 §7.3.3 lets a writer emit 0.750000, .75, or 0.75 for the same value, and none of those survive a round trip through 24-bit binary and a generic formatter unchanged. Worse, a value like 0.7 is not representable in a Single at all; it parses to the nearest float, and reformatting that float can produce 0.69999999 or a rounded neighbor depending on the digit loop. On a fill color or a /CA transparency constant, that is a one-count difference in an 8-bit channel, which is enough to fail a pixel comparison against the source and, on gradient boundaries, enough to see

THPDFNumericObject.RememberSourceToken solves this for the unmodified case. The parser calls it with the raw token right after assigning Value; the method accepts only tokens made of digits, at most one decimal point, and an optional leading sign, and stores the token together with the value it corresponded to in FSourceValue. The SourceToken property returns the stored text only while Value still equals FSourceValue. Change the number and the token evaporates, so a modified value always goes through the existing formatting path and never emits stale text. SaveNumericObject checks SourceToken first and writes it verbatim when present, then falls through to the integer, color-space reference, and fractional branches only for numbers that were created or edited in memory

The invariant is small and worth stating plainly: a number you did not touch is written with the bytes it was read with, and a number you did touch is written by HotPDF's own formatter. Compact members benefit from this the same way body objects do, since EnsureCompressedObjectLoaded runs the same parser over the member slice. The number formatting itself, and its independence from the process locale, is covered in the article on locale-invariant PDF number formatting in HotPDF

Testing a rewrite path against object streams

Three checks catch every failure described above, and none of them require Acrobat. First, compare IndexedObjectCount against MaterializedObjectCount after the save; on a full rewrite they must be equal, and any gap is a member that was dropped. Second, extract text and enumerate the structure tree on both files, not just render them, so a lost ActualText or a lost structure element shows up as a diff. Third, load the output with a fresh instance and assert GetLoadedQuarantinedObjStmCount is zero, which also proves the writer did not produce a container the reader cannot open. The crypt filter combinations that decide FReloadObjectStreamsEncrypted are laid out in the StmF, StrF, and EFF policy article. The writer side of this story, how to emit object streams and when to prefer an incremental update over a rewrite, is in the object streams and incremental updates guide

Lazy member loading, the pre-writer expansion pass, decrypt quarantine, and source-token preservation all ship in the HotPDF Delphi Component for Delphi and C++Builder. The product page links the API reference if you want to trace GetLoadedObjectStreamCacheInfo and the quarantine accessors against your own ingest pipeline