Import an FDF into a form and the field values come back subtly wrong: a bullet arrives as a dagger, an ellipsis as an em dash. PDFium Component traced that whole family of symptoms to a single row in its PDFDocEncoding table — the special-character block in ISO 32000-1 Annex D.2 starts at $80, not $7F
Why does an FDF import return the neighboring character?
Because an encoding table shifted by one position is still a perfectly valid-looking table, and every lookup in it succeeds. The block that Annex D.2 assigns to $80 through $9E had been transcribed one row early, so byte $80 resolved to the entry that belongs to $81, byte $81 to the entry for $82, and so on across nineteen consecutive code positions. Nothing raised, nothing logged, and no field was ever rejected — TPdf.ImportFDF reported the same field count it always had
The damage was uniform and quiet. A byte carrying $80, which Annex D.2 defines as BULLET (U+2022), came back as DAGGER (U+2020). Byte $83, HORIZONTAL ELLIPSIS (U+2026), came back as EM DASH (U+2014). Byte $8D, LEFT DOUBLE QUOTATION MARK (U+201C), came back as RIGHT DOUBLE QUOTATION MARK (U+201D) — the case nobody notices, because a wrong curly quote still looks like a curly quote in a form field. Field names suffered the same fate as field values, which is worse: a name that no longer matches the widget in the document silently fails to apply
PDF text strings carry two different encodings
ISO 32000-1 §7.9.2.2 defines the text string type as one of exactly two encodings, chosen by inspecting the first two bytes rather than by any dictionary entry. A string that opens with the byte order mark FE FF is UTF-16BE; anything else is PDFDocEncoding. That single branch decides which of two completely different decoders runs, and PDFium Component applies it inside the FDF parser before touching a single character
// FDF text string: UTF-16 when the BOM is present, PDFDocEncoding otherwise
if (Length(Raw) >= 2) and
(((Byte(Raw[1]) = $FE) and (Byte(Raw[2]) = $FF)) or
((Byte(Raw[1]) = $FF) and (Byte(Raw[2]) = $FE))) then
begin
BigEndian := Byte(Raw[1]) = $FE;
// ... decode UTF-16 code units, validating surrogate pairs
end
else
begin
SetLength(Result, Length(Raw));
for I := 1 to Length(Raw) do
Result[I] := FdfPdfDocEncodingChar(Byte(Raw[I]));
end;This is why the bug hid so long in ordinary testing. Any FDF produced by a tool that writes UTF-16BE by default never reaches FdfPdfDocEncodingChar at all, so the shifted table stays dormant. Only single-byte strings — which is what a lot of FDF generators emit for plain Latin field names, because they are shorter — go down the branch where the table matters. Pure ASCII field names then decode correctly anyway, since below $80 PDFDocEncoding and ASCII agree. You need a single-byte string containing one typographic symbol before anything goes visibly wrong
The fixed table, and the two anchors that pin it
PDFium Component now spells out the boundary in FdfPdfDocEncodingChar itself, because the two ends of the block are exactly what an implementer gets wrong. $7F is DEL and has no PDFDocEncoding meaning at all; the block opens at $80 with BULLET and runs to $9E with zcaron; $9F is unassigned; and the Euro sign sits alone at $A0
function FdfPdfDocEncodingChar(Value: Byte): WideChar;
begin
case Value of
// ISO 32000-1 Annex D.2. The block runs from $80, not $7F: $7F is DEL and
// has no PDFDocEncoding meaning, and the Euro sits at $A0, which the
// default branch would otherwise hand back as no-break space
$7F: Result := WideChar($FFFD);
$80: Result := WideChar($2022); // BULLET, first row of the block
$81: Result := WideChar($2020); // DAGGER
// ... $82 through $9D
$9E: Result := WideChar($017E); // zcaron, last defined row
$9F: Result := WideChar($FFFD); // unassigned in Annex D.2
$A0: Result := WideChar($20AC); // EURO SIGN
else
Result := WideChar(Value);
end;
end;Look at what the shift did to those two ends, because that is where an off-by-one stops being cosmetic. zcaron sat at $9E in the specification but at $9D in the shifted table, so a document byte of $9E fell past the last row anyone had transcribed and dropped straight through to the identity fallback, returning a C1 control character instead of a letter. And $A0, the Euro, was never written into the table in the first place — the identity branch handed it back as U+00A0 NO-BREAK SPACE, a character that survives every subsequent validation, prints as nothing, and never matches the field name you are looking for
How do you test a transcribed encoding table?
Test the first row, the last row, and the gaps — not a sample from the middle. This is the part of the fix worth carrying to any other table you copy out of a specification, because a middle-of-the-block spot check is precisely the test that a whole-block shift survives. Verifying that $8D gives a left double quote proves nothing when $8C also gives something plausible; verifying that $80 gives BULLET and $9E gives zcaron pins both ends, and a shifted block cannot satisfy both at once
procedure CheckPdfDocEncodingAnchors;
begin
// Both ends of the Annex D.2 special block: a whole-block shift breaks these
Assert(FdfPdfDocEncodingChar($80) = WideChar($2022), 'first row: BULLET');
Assert(FdfPdfDocEncodingChar($9E) = WideChar($017E), 'last row: zcaron');
// Code positions the specification leaves undefined must not decode to a letter
Assert(FdfPdfDocEncodingChar($7F) = WideChar($FFFD), '$7F is DEL, unmapped');
Assert(FdfPdfDocEncodingChar($9F) = WideChar($FFFD), '$9F is unassigned');
// The isolated entry above the block, easy to omit entirely
Assert(FdfPdfDocEncodingChar($A0) = WideChar($20AC), 'EURO SIGN');
end;The undefined positions deserve their own assertions rather than being skipped as uninteresting. An off-by-one has to put something somewhere, and where it puts it is almost always a code position the specification left empty — $7F at the front, $9F at the back. A test that only checks defined positions can pass while the table is wrong in both directions; a test that pins the holes catches the shift on the first run
What this costs in a round-trip form workflow
An FDF import that quietly rewrites one character per symbol is worse than one that fails, because the failure surfaces downstream and looks like something else. Field names are the sharp edge here. TPdf.ImportFDF matches incoming names against the widgets in the open document, and a name that decoded one character wrong simply does not match — the field stays empty, the return value still counts the field as parsed, and the report that comes out of the pipeline is short one value with nothing in the log explaining why. Anyone building an export-and-reimport loop of the kind described in exporting and importing XFDF form data and annotations with PDFium Component should round-trip a field name containing a typographic symbol, not just plain ASCII, before trusting the loop
It is also worth being honest about the limits of the fix. The table now matches Annex D.2 for all 256 byte values, which makes single-byte FDF strings decode correctly, but PDFDocEncoding itself covers a Latin repertoire and nothing more — anything outside it has to arrive as UTF-16BE with a BOM, which is exactly what §7.9.2.2 intends. And a UTF-16 string still has its own separate hazards on the Delphi side, since a code point beyond U+FFFF needs a surrogate pair rather than one WideChar, the problem covered in why emoji and CJK characters break WideChar in Delphi. The broader point generalizes past encoding tables: a PDF parser can reject something it should accept, which you find immediately, or accept something and quietly get it wrong, which you find in production. The name-token traps described in parsing PDF dictionaries safely sit in that second category, and so does a table shifted by one row — the code path succeeds, the return value looks reasonable, and only a test written against the specification rather than against the implementation says otherwise
ImportFDF, ExportFDF, and ImportXFDF on TPdf, together with the FDF and XFDF parsers behind them, are part of the standard PDFium Component for Delphi and C++Builder