Substring searches such as Pos('/Length', Dict) are the wrong tool for reading a PDF dictionary, because PDF name keys share prefixes: /Length is a prefix of /Length1, and /Encrypt is a prefix of /EncryptMetadata. ISO 32000-1 §7.3.5 defines a name as a token that ends only at a delimiter or whitespace, so a key counts as found only when the byte after it is one of those characters. A dictionary reader that skips that one check will eventually read the wrong value from a perfectly valid file
The bug that taught us this looked nothing like a lexer problem. A conformance scan started reporting a FontFile stream as damaged: the decompressed font program was a fragment, cut off mid-table. The file opened fine in every viewer. The stream data on disk was intact. The root cause sat in one line of our shared dictionary reader: Pos('/Length', ...) had matched /Length1, a standard key that FontFile stream dictionaries carry per ISO 32000-1 Table 127, and the reader took the integer after /Length1 as the stream length. The writer of that particular file happened to serialise /Length1 before /Length, which it is entirely free to do, since dictionary entries are unordered per §7.3.7. The stream was truncated to a garbage byte count, and every downstream check that consumed it went silently blind
Why does substring matching break PDF dictionary parsing?
Substring matching breaks because the PDF name space is full of deliberate prefix families, and because dictionary entry order is unspecified. Table 127 of ISO 32000-1 defines /Length1, /Length2 and /Length3 for embedded font streams, all sitting next to a /Length in the same dictionary. The encryption dictionary pairs /Encrypt in the trailer with /EncryptMetadata inside it. Short keys are worse: a lookup built as Pos('/' + Key, ...) with Key = 'N' happily lands on /Name or /Nums. None of these collisions require a malformed file. A writer that orders /Length1 before /Length is fully conforming, which means the substring bug is not a robustness gap against broken input — it is a correctness gap against valid input
The failure mode is also the quiet kind. A wrong /Length does not raise an exception; it hands you a shorter or longer byte slice than the stream actually occupies. If that slice feeds a font-subset check, a CMap parse or a metadata scan, the consumer sees garbage and typically reports nothing at all, because half a zlib stream simply fails to inflate and the code moves on. We shipped exactly this class of defect and fixed it in v2.14.3 of our shared reader, after a clause-by-clause audit of ISO 32000-1 §7.2–§7.3 flagged every Pos-style key lookup as suspect
What ISO 32000-1 §7.3.5 actually defines a name to be
Section 7.3.5 is short and precise: a name object is a solidus followed by a sequence of regular characters, and the token is terminated by the first delimiter or whitespace character. The delimiters are the eight bracket characters plus the solidus and the percent sign — ( ) < > [ ] { } / % — and whitespace is null, tab, line feed, form feed, carriage return and space (§7.2.2–§7.2.3). That termination rule is the whole story. /Length1 is not "/Length followed by a 1"; it is a single, indivisible token, exactly as LengthOne and Length are different identifiers in Pascal. Any reader that finds keys by raw byte search is reimplementing the lexer with the termination rule deleted
Here is the shape of the defect, reduced to its essentials. This version compiles, passes tests against files whose writers order /Length first, and corrupts streams for writers that do not
// WRONG: matches /Length1, /Length2, /Length3 as well
function ReadStreamLength(const Dict: AnsiString): Integer;
var
P: Integer;
begin
Result := -1;
P := Pos('/Length', Dict);
if P > 0 then
Result := ReadIntAt(Dict, P + Length('/Length'));
end;Whole-token matching: check the byte after the key
The correct predicate follows directly from §7.3.5: a candidate match is a real key only if the character immediately after it is a delimiter, whitespace, or the end of the buffer. Everything else is a longer name that merely shares a prefix, so the search must continue past it rather than give up. The fix in our reader replaced every raw Pos lookup with a single shared routine built on this rule
function IsPdfDelimOrWs(C: AnsiChar): Boolean;
begin
Result := C in [#0, #9, #10, #12, #13, ' ',
'(', ')', '<', '>', '[', ']', '{', '}', '/', '%'];
end;
// Correct: whole-token match per ISO 32000-1 §7.3.5
function FindDictKey(const Dict, Key: AnsiString): Integer;
var
P, After: Integer;
begin
Result := 0;
P := Pos(Key, Dict);
while P > 0 do
begin
After := P + Length(Key);
if (After > Length(Dict)) or IsPdfDelimOrWs(Dict[After]) then
Exit(P); // token ends here: genuine key
P := PosEx(Key, Dict, P + 1); // prefix of a longer name: keep looking
end;
end;Two details in that loop carry weight. First, it keeps searching instead of returning failure on the first prefix collision, because /Length1 120 /Length 4076 is a legal ordering and the real key is still ahead. Second, the end-of-buffer case counts as a terminator, since a dictionary fragment can legitimately end right after a name. A subtler point worth auditing in your own code: the same rule applies on the left side of the match if your search string does not include the solidus, otherwise Pos('Length', ...) can land inside /PieceLength. Anchoring the search string with the leading /, as above, handles the left edge because / is itself a delimiter that ends the preceding token
How can a hostile PDF turn a parser bug into a gigabyte allocation?
A malformed or malicious file escalates these lexical mistakes into resource exhaustion, because dictionary integers frequently feed allocation sizes. Our audit found a chain of exactly this shape in object-stream expansion. The /N entry of an ObjStm dictionary says how many compressed objects the stream holds, and the expansion code called SetLength on an array sized by it. The integer parser, however, left its out-parameter untouched on failure while still returning it — so a non-numeric /N handed SetLength an uninitialised stack value. A garbage positive integer there means an allocation request in the gigabytes, triggered by a few bytes of corrupt input, while merely scanning a document you have not even agreed to trust yet
The repair had two independent parts, and both generalise. The parser now returns an explicit 0 on failure, never uninitialised memory. And the consumer no longer trusts /N arithmetic-free: the ObjStm header region before /First stores a pair of integers — object number and offset — for every compressed object, and each pair occupies at least four bytes including separators. Any /N above FirstVal div 4 + 1 is therefore physically impossible for the declared header size and is rejected before any allocation happens. The bound costs one comparison and is derived from data already in hand, which is the pattern to look for: a ceiling the file itself proves, not an arbitrary constant
// /N is attacker-controlled; bound it by what /First can hold
if not TryReadDictInt(Dict, '/N', NVal) then
NVal := 0; // explicit zero, never stack garbage
if (NVal <= 0) or (NVal > FirstVal div 4 + 1) then
Exit; // header cannot contain that many pairs
// /Length can never exceed the file that contains the stream
if (LenVal < 0) or (LenVal > SourceSize) then
Exit; // refuse before allocating the bufferTwo more ceilings complete the defensive perimeter in our reader, both shipped in v2.12.0. The stream reader refuses any /Length larger than the whole file before allocating the result buffer — a stream cannot be bigger than the container it lives in, so the check is free of false positives. And the inflate path caps decompressed output at 256 MiB, which shuts down the classic zlib bomb where a few kilobytes of input expand without bound; the cap is generous for any real PDF stream while keeping the worst case survivable. The theme across all three is the same: every size a file declares is a claim, and the parser verifies each claim against something it can measure before committing memory to it. The same audit posture applies one layer down at the binding boundary, which is covered in hardening the PDFium ABI and memory safety in Delphi
Where the whole-token rule is not enough
Honest limits, so you do not overtrust the routine above. Whole-token matching fixes key identification, but a flat byte search over a dictionary span still cannot tell whether a match sits inside a nested dictionary, a literal string or a comment — FindDictKey on a page object can land on a key inside its /Resources subdictionary if you hand it too wide a span. Our reader constrains the span to a single object body first and treats string and comment contexts as a separate, still-open audit item. Substring safety is one rung of a ladder, not the whole ladder: cross-reference consistency is its own discipline, covered in validating object and xref streams, and the wider threat catalogue for documents you did not author is in auditing PDF security risks
If you maintain a hand-written dictionary reader in Delphi or Lazarus, the checklist from this incident is short. Grep for every Pos('/ in the codebase and route the hits through one whole-token helper. List the prefix families your keys participate in — /Length, /Encrypt, /N, /Type against /Type1 all appear in real files. Then walk every integer that reaches SetLength, GetMem or a copy loop and ask what bounds it: the file size, a header-derived ceiling, or nothing. The parsing layer described here is the foundation under our PDFium Component, where the byte-level reader and the rendering engine cross-check each other on every document they touch