When a PDF cross-reference table is unusable, the fix is to ignore it entirely and rebuild it from the file body. The PDFlibPas Delphi PDF Library does this with a single pass token scanner that records every genuine indirect object header it sees, then recovers the trailer dictionary and hands the reconstructed table to the normal loader
What breaks first when a PDF is damaged
The cross-reference table is the most fragile part of a PDF, because it is the only part that stores absolute byte offsets. ISO 32000-1 §7.5.4 defines those entries as ten-digit offsets from the start of the file, and §7.5.5 puts the startxref keyword near the end pointing at the table itself. Every one of those numbers is invalidated by any edit that shifts bytes. An FTP session that ran in text mode and translated CRLF, a truncated download, a sector that went bad on a shared drive, a batch tool that appended without writing an incremental update correctly: all of them leave the object data perfectly readable and the index pointing at rubbish
That is why "the file is damaged and is being repaired" is such a common dialog. The bytes are almost always still there. What is gone is the map. Reconstruction is therefore not forensic recovery of lost data, it is a rebuild of an index that can be derived from the body, and it succeeds far more often than users expect because the expensive content, page trees and fonts and images, is untouched
Why does scanning for N 0 obj find false matches?
A naive rebuild searches the raw bytes for the pattern "integer, integer, obj" and records every hit. It finds too much. PDF is a container format, and three regions of a file are opaque to the object grammar: comments (§7.2), strings (§7.3.4), and stream data (§7.3.8). Any of them can contain bytes that read exactly like an object header, and none of them is an object header. A caption in a literal string, a leftover debug comment, or two megabytes of Flate or DCT output will all happily produce something that looks like 99 0 obj
const
Trap: AnsiString =
'4 0 obj'#10 +
'(a caption that mentions 88 0 obj)'#10 + // literal string, not an object
'endobj'#10 +
'% 77 0 obj left over from a debug dump'#10 + // comment, not an object
'5 0 obj'#10 +
'<< /Length 2097152 >>'#10 +
'stream'#10 +
{ two MiB of compressed bytes that contain the byte sequence
99 0 obj and, further along, a complete endstream }
'endstream'#10 +
'endobj'#10;Every false entry costs twice. It pollutes the rebuilt table with an object number that does not exist, and it can shadow a real object with the same number that appears later in the file. PDFlibPas therefore does not pattern-match at all. It tokenises, which means it always knows whether the bytes under the cursor are code or payload, and payload is skipped without ever being interpreted
A single pass state machine over 64 KiB blocks
PDFlibPas scans the whole file exactly once, in 64 KiB blocks, with a state machine built on the token rules of ISO 32000-1 §7.2 and the indirect object syntax of §7.3.10. A token ends at white space or at one of the delimiter characters, and an object header is recorded only when a complete sequence of a positive object number, a non-negative generation number, and a bare obj keyword has been seen. The recorded offset is the start of the object number token, which is what a cross-reference entry has to point at, not the position of the obj keyword
function RebuildIsWhiteSpace(Value: Byte): Boolean;
begin
Result := (Value = 0) or (Value = 9) or (Value = 10) or
(Value = 12) or (Value = 13) or (Value = 32);
end;
function RebuildIsDelimiter(Value: Byte): Boolean;
begin
Result := (Value = Ord('(')) or (Value = Ord(')')) or
(Value = Ord('<')) or (Value = Ord('>')) or
(Value = Ord('[')) or (Value = Ord(']')) or
(Value = Ord('{')) or (Value = Ord('}')) or
(Value = Ord('/')) or (Value = Ord('%'));
end;The important detail is that token state and string state survive a block boundary. A header that straddles the 65536-byte line is still recognised, because the partial token, the pending integer pair, and the in-string flags all carry into the next block. The buffers are fixed: 64 KiB for the scan, 32 bytes for the longest token that can possibly matter, and the only arrays that grow with the file are the object number, generation number and 64-bit offset lists, which are proportional to the real object count rather than to the file size. In practice the scan issues sequential reads and at most two explicit seeks over the entire document, which is what makes it viable on the multi-hundred-megabyte inputs discussed in the article on direct access merge and split
Why can a stream not be trusted to end at endstream?
Because stream data is arbitrary bytes, and arbitrary bytes can spell endstream by accident. A stream that begins after the stream keyword must be skipped as opaque data until it genuinely ends, but the first occurrence of the closing keyword is only a candidate. PDFlibPas resolves this by requiring corroboration: an endstream token is accepted as the real end of the stream only when the next non-white-space token is a standalone endobj, the sequence §7.3.8 requires around a stream object. A chance hit inside compressed data almost never has that follow-up, so the scanner stays inside the stream and keeps going. Two smaller rules matter as much. The stream keyword only enters stream state when it is a bare keyword, so a name object such as /stream in a dictionary never triggers it. And an obj or trailer token is only honoured when the token did not overflow the 32-byte cap and did not begin with a solidus. Without those two guards a resource dictionary with the wrong key names would be enough to derail the scan, which is exactly the class of adversarial input covered in the notes on parsing untrusted PDF safely
Finding the real end of the trailer dictionary
Recovering the objects is only half the job, because the loader still needs a trailer to find /Root. PDFlibPas remembers the last 64 trailer keyword positions found during the scan and validates them backwards, most recent first, so the newest usable trailer wins and a stray keyword that is not followed by a dictionary simply fails validation and falls through to the previous candidate. Each candidate is read with a cap of 1 MiB, and the dictionary end is located by tracking nested << and >> depth along with literal string escapes, hexadecimal strings and comments
// A naive reader that stops at the first '>>' truncates this trailer,
// and a fixed 2048-byte window can cut it in half on a large one
'trailer'#10 +
'<< /Size 5 /Root 1 0 R' +
' /Custom << /Text (value >> preserved) >> >>'#10Depth tracking is not academic. A truncated trailer that loses /Encrypt turns a recoverable encrypted document into an unopenable one, and losing /Info or a custom sub-dictionary silently discards metadata a downstream system may depend on. If the file is encrypted, the recovered trailer is what lets the normal credential path run, and the retry semantics are the same ones described in the article on encrypted document loading
What reconstruction cannot give you back
Reconstruction is a best effort, and being honest about its limits is part of shipping it. Three cases fail outright. Objects packed inside object streams (§7.5.7) are not individually visible to a byte scan, so if a container survives but its cross-reference stream (§7.5.8) does not, the objects it holds are not indexed by the rebuild. A file whose body was actually corrupted, rather than merely misindexed, will produce headers whose contents no longer parse. And a file with no recoverable trailer keyword and no readable catalogue has nothing to anchor a document tree to, regardless of how many object headers were found
Duplicate object numbers are the interesting middle case. An incrementally updated file legitimately contains several generations of the same object number, and the surviving cross-reference chain is the only record of which one was current. A rebuild does not have that chain, so it records every header it sees in file order and resolves by object number afterwards. Usually the later revision wins, which is usually right, but a document that was updated and then partially rolled back can come back subtly different from what the original xref described. Linearised files carry the same caveat from the other direction: the first-page layout and hint tables are meaningless once the index is regenerated, so a repaired file should be treated as a plain, non-linearised document
var
Pdf: TPDFlib;
begin
Pdf := TPDFlib.Create;
try
if Pdf.LoadFromFile('truncated-invoice.pdf', '') = 1 then
begin
if Pdf.GetDocumentRepaired = 1 then
LogWarning('xref was unusable; the table was reconstructed');
if Pdf.PageCount > 0 then
Pdf.SaveToFile('recovered-invoice.pdf'); // writes a clean xref
end;
finally
Pdf.Free;
end;
end;The fallback is automatic: PDFlibPas runs the raw scan whenever the cross-reference chain cannot be read, and also when every in-use entry claims offset zero, which is the signature of a table that was written but never filled in. GetDocumentRepaired returns 1 when that path ran, and it is worth logging rather than ignoring, because a document that loaded through reconstruction should be re-saved to a clean file rather than left in a pipeline as though nothing happened. Saving it writes a fresh, consistent cross-reference table, which is the cheapest possible fix for every downstream consumer
The reconstruction path, the GetDocumentRepaired flag and the streaming loader shown here are part of the PDFlibPas Delphi PDF Library, alongside the parsing, rendering and signing APIs covered elsewhere on this blog