Loading Hybrid-Reference PDFs From Word and Excel in Delphi

Open a PDF that Microsoft Word or Excel produced, page through it, and nothing looks unusual. Load it into a Delphi program, read the page count back, and the number is right. Then re-save it with encryption turned on and the save step throws, or the output opens to a damaged cross-reference warning. The file was never corrupt. It is a hybrid-reference file, and the very structure that lets a fifteen-year-old viewer open it is the structure that defeats a loader which stops reading too early

This is one of the most common ways a PDF pipeline that passed every internal test meets a file it cannot round-trip. The inputs were all generated in-house, so they were never hybrid. The first hybrid file arrives the day a customer forwards an invoice exported from a spreadsheet

What Word and Excel actually write

ISO 32000-1 describes the hybrid-reference layout in §7.5.8.4. An application that wants PDF 1.5 features such as object streams, while still letting a PDF 1.4 reader open the file, writes the cross-reference information twice. There is a classic cross-reference table, the fixed-width ASCII rows that ended every PDF up to version 1.4, and there is a cross-reference stream that indexes the rest. The trailer of the classic section carries a /XRefStm entry whose value is the byte offset of that stream

The division of labor is deliberate. Objects an old reader has to reach, the catalog and the page tree among them, are addressable from the classic table. Objects that were folded into compressed object streams are marked free in the classic table, with a type f entry, so a 1.4 reader skips straight past them and never trips over a structure it cannot parse. Their real locations live only in the cross-reference stream. The signature of such a file is its tail: a short classic section that covers only the objects a legacy reader needs, whose trailer points at the /XRefStm where the actual recovery data sits

Why a correct page count proves nothing

Because the catalog and page tree are reachable from the classic table on purpose, a loader that reads only that table finds /Root, walks the page tree, and reports the right number of pages. Everything an old reader needs is present, so the file appears healthy. The objects that went missing are the ones packed into object streams: AcroForm field dictionaries, tagged-PDF structure elements, the long tail of small dictionaries that never had to be visible to a legacy viewer

You do not notice the gap until something touches those objects, and a full resave touches all of them. Walking the document to re-encrypt or rewrite it is precisely the operation that asks for every object number in turn, which is why the symptom surfaces at save time rather than load time, far from its cause

The trap is a detector that sees xref and stops

The cheap way to decide how a file is indexed is to follow startxref and inspect the first bytes it points at. The keyword xref means a classic table; a stream object means a cross-reference stream. That test is correct for any file that commits to one scheme. It is wrong for a hybrid file, whose startxref aims at a classic section for the sole purpose of satisfying old readers, while the /XRefStm in that section's trailer is where most of the document is actually indexed. A detector that returns "classic" on the first xref it meets never reads /XRefStm, and every object that lives only in the stream becomes invisible

var
  Pdf: THotPDF;
  PageCount: Integer;
begin
  Pdf := THotPDF.Create(nil);
  try
    PageCount := Pdf.LoadFromFile('Invoice_XLS.pdf');  // pages are reachable from the classic table
    // inspect or edit the loaded document here
    Pdf.SaveLoadedDocument('Invoice_secured.pdf');     // needs every object, stream-only ones too
  finally
    Pdf.Free;
  end;
end;

Run that sequence on a loader built around the early-exit detector and the load looks fine; the resave is where the absent objects announce themselves. The fix is not to read more bytes at the start; it is to recognize the hybrid trailer and follow /XRefStm before deciding the file is done

Merge order is not negotiable

Once both indexes have been read, they can be combined in one direction only. The cross-reference stream has to be merged first, with the classic entries filled in around it. The reason is the small deception at the heart of the format. A hybrid file marks its compressed objects as free in the classic table so that old readers ignore them. A loader that honors a first-seen-wins policy and reads the classic table first will record those object numbers as free, then discard the stream entries that actually locate them, because the slots are already taken. Reverse the order and the type 2 entries from the stream, each one an object-stream number plus an index, win the slots they are meant to own, and the classic entries settle around them

The same discipline guards against an older revision resurrecting a deleted object. Incremental updates chain backward through /Prev, and a type 0 free entry is a sentinel that a more recent section has retired an object number. A later, older section in the chain must not be allowed to overwrite that sentinel with a stale location. Treat first-seen as authoritative for free markers and the deleted object stays deleted; treat it carelessly and a file's own history reanimates content the latest revision removed

How the two views drift apart

All of that assumes the two indexes agree. Straight out of Word, they do: each view describes the same document within the scope it declares. Trouble starts when the file passes through a tool that understands only one of them. Stamping utilities are the usual suspects. One appends a classic-style incremental update with a few new objects, a new xref section, a /Prev link back to the previous section, and a fresh trailer. If that trailer drops /XRefStm, the stream view is orphaned. If it copies the old value forward, the stream view still describes the document as it stood before the stamp went on. Either way, the indexes now disagree

The signature is easy to recognize once you know it. Objects visible through one view are missing or stale in the other. A reader that resolves through the stream finds the pre-edit version of an updated object, or no entry at all for an appended one. A reader that trusts only the table sees the edit but loses every compressed object the stream alone could locate. Users describe it as form fields that show up in one viewer and vanish in another, annotations the stamping pass seems to have deleted, or lookups that land on the wrong object entirely

Adobe Acrobat is what makes these files so expensive to chase. When the declared index disagrees with the bytes, Acrobat usually rebuilds the cross-reference data quietly by scanning for object headers, so whoever produced the broken file opens it, sees nothing wrong, and sends it on. The failure surfaces later and somewhere else — a preflight validator, a signing service, an archival ingest job — any consumer that trusts the declared structure and reports missing objects or a cross-reference mismatch. "It opens fine in Acrobat" is how nearly every hybrid desynchronization ticket begins

What this means in HotPDF

The engine resolves hybrid-reference files for you, and it does so on every path that has to parse the cross-reference data. Load a document with LoadFromFile or LoadFromStream, make your changes, and call SaveLoadedDocument; or run a one-shot operation such as EncryptFile that reads an input and writes an output. Either way the recovery reads /XRefStm, merges the stream section ahead of the classic entries, and resolves the objects that live in streams before the write enumerates them. The AES-256 encryption path is where this matters most visibly, because encrypting a document rewrites every object and so demands that every object has already been located

// One-shot: read the hybrid input, write an AES-256 encrypted copy
Pdf.EncryptFile('Letter_DOC.pdf', 'Letter_secured.pdf',
  'owner-secret', '', aes256, [prPrint, prFillAnnotations]);

The detail worth carrying away sits upstream of the API. Files that arrive from Word, Excel, PowerPoint, and a long list of "Save as PDF" pipelines are routinely hybrid, so a loader you exercise only against your own generator output may never meet one in testing. Seed your fixtures with documents exported from real Office applications, not only with files your own code produced

Checking a file you suspect

Two inspections settle the question quickly. Open the file in a hex view and read the bytes after the final startxref; a hybrid file shows a short classic section whose trailer dictionary contains /XRefStm. Or compare the object count a full parse reports against the highest object number that /Size declares in the trailer. A large gap means objects are hiding in streams the loader has not opened, which is the same shortfall that turns into a save-time failure later

The tail of a typical Excel export makes the first check concrete. Everything after the final xref keyword is plain ASCII, so the signature is readable straight out of a hex view (offsets illustrative, annotations added)

xref
0 16                         % objects 0-15: what a legacy reader needs
0000000000 65535 f
0000000017 00000 n
...                          % 14 more rows; the rest live in object streams
trailer
<< /Size 216                 % one past the highest object number in use
   /Root 1 0 R
   /Info 15 0 R
   /ID [<5C9A...> <5C9A...>]
   /XRefStm 87325            % byte offset of the cross-reference stream
>>
startxref
88710                        % points at the classic section above
%%EOF

The /XRefStm key in the trailer is the tell. The classic section above it lists 16 of the 216 object numbers the file uses; everything packed into object streams is either absent from it or marked free. A detector that stops at the xref keyword has, at this point, seen only that partial index. When you would rather script the check than eyeball it, the marker sits within the last couple of kilobytes of the file, so a bounded backward read is enough. If the file was later updated incrementally, the tail holds more than one trailer, and the one that counts is the last

// Returns the /XRefStm offset from the newest trailer in the file's tail,
// or -1 if the marker is absent (the file is not hybrid, or not a PDF at all)
function FindXRefStm(const FileName: string): Int64;
var
  FS: TFileStream;
  Tail: AnsiString;
  Len, P, Q: Integer;
begin
  Result := -1;
  FS := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  try
    Len := 2048;                        // the trailer lives in the tail
    if FS.Size < Len then
      Len := Integer(FS.Size);
    FS.Position := FS.Size - Len;       // bounded backward read: 2 KB max
    SetLength(Tail, Len);
    FS.ReadBuffer(Tail[1], Len);
  finally
    FS.Free;
  end;
  P := 0;                               // keep the last marker: incremental
  Q := Pos(AnsiString('/XRefStm'), Tail); // updates append newer trailers
  while Q > 0 do
  begin
    P := Q;
    Q := Pos(AnsiString('/XRefStm'), Tail, P + 1);
  end;
  if P = 0 then
    Exit;                               // no hybrid marker in the tail
  Inc(P, Length('/XRefStm'));
  while (P <= Len) and (Tail[P] in [' ', #9, #13, #10]) do
    Inc(P);                             // skip whitespace after the key
  Result := 0;
  while (P <= Len) and (Tail[P] in ['0'..'9']) do
  begin
    Result := Result * 10 + Ord(Tail[P]) - Ord('0');
    Inc(P);
  end;
end;

// Usage: a non-negative result names the byte where the stream starts
if FindXRefStm('Invoice_XLS.pdf') >= 0 then
  Writeln('hybrid-reference file: resave will need the /XRefStm section');

Treat the probe as triage, not as a parser: it tells you which files in a batch deserve attention before a resave job runs, and nothing more

Point it at a folder of real inputs and the files sort into three groups. A classic-only file has a trailer keyword in its tail but no /XRefStm, so the probe returns -1. A file that commits fully to cross-reference streams also returns -1, correctly. It has no trailer keyword at all, because its trailer keys live in the stream dictionary. Only the double-indexed layout yields an offset. If your triage report needs to tell the first two groups apart, search the same tail for trailer before discarding it; one more Pos call does it

Before the probe runs unattended, harden it in two places. An offset is only a claim. Seek to it and confirm that an object whose dictionary says /Type /XRef really sits there, because a damaged or truncated file can still carry the key while the stream it names is gone. That file belongs in a different bucket from a healthy hybrid. The check is a few lines on top of the same file handling

// Offset is the value FindXRefStm returned; FS is a TFileStream opened
// the same way, Head an AnsiString (Min comes from System.Math)
if Offset >= FS.Size then
  Exit(False);                          // the key survived, the stream did not
FS.Position := Offset;
SetLength(Head, Integer(Min(Int64(512), FS.Size - Offset)));
FS.ReadBuffer(Head[1], Length(Head));
Result := (Pos(AnsiString('/Type /XRef'), Head) > 0) or
  (Pos(AnsiString('/Type/XRef'), Head) > 0);  // PDF syntax allows both spellings

Then make the 2 KB window a parameter. Two kilobytes covers ordinary Office output, but an unusually large trailer dictionary can push the marker out of range, and widening the window beats declaring the file classic by accident

Routing hybrid files through a pipeline

Detection pays off when it changes where a file goes next. Where that is depends on what the pipeline is about to do with it

Files that are only read, rendered, or validated should go to a loader that resolves both views. The PDFium Component parses the /XRefStm chain during load, so the object table your code sees is already the merged one, and the checks in our article on validating object and cross-reference streams apply unchanged. When a desynchronized hybrid is damaged badly enough to refuse loading, the engine reports it through its load error code, and FPDF_ERR_FORMAT is the one structural damage produces. A clean load is weaker evidence. PDFium is lenient by design and rebuilds most inconsistent files without comment, so success proves the file was recoverable and says nothing about whether its two views agree. For that question, go back to comparing what a full object walk finds against the trailer's /Size

Files the pipeline modifies are another story. Our preference here is blunt: stop them being hybrid at all. Load the document with HotPDF and save it in full, and the output carries a single, self-consistent cross-reference in one form. No /XRefStm, no second view left to drift, every object owned by exactly one index entry. That normalization is what you want before archival ingest, before a strict downstream RIP or signing service, and after any edit applied to a hybrid input. It leans entirely on the stream-first merge described earlier, since a full save can only write out the objects the load managed to locate

Signed documents are the exception. A full rewrite moves every byte, which invalidates any signature computed over the original byte ranges. A change to a signed hybrid has to go in as a proper incremental update that keeps both views in step, and a signed file that only needs reading should pass through the pipeline untouched

The writer's side of this story, how object streams and compressed cross-references are produced in the first place, is covered in our article on object streams and incremental updates. When the hybrid file in question is also very large, the loading techniques in the Direct File API walkthrough for large PDF workflows let you inspect it without reading the whole thing into memory. Both pair naturally with the recovery described here, which ships as part of the HotPDF Delphi Component for Delphi and C++Builder alongside the loading, editing, encryption, and signing APIs covered elsewhere on this blog