You fill a field with Café — ‘quoted’, /V reads back fine, yet the saved PDF shows Caf?, a stray curly quote, or auto-sized text spilling past the box. HotPDF keeps two conversions apart: it decodes /V to Unicode, then re-encodes each character for the font that /DA names and sizes the text from the glyphs it emits
What happens to Café between /V and the glyphs?
The value and the picture of the value live in different worlds. /V holds a PDF text string (ISO 32000-1 §7.9.2.2): bytes in PDFDocEncoding, or UTF-16BE behind an FE FF byte order mark, and ISO 32000-2 adds UTF-8 behind EF BB BF. None of that concerns the font. The widget's appearance is a content stream, and the operand of its Tj operator is a run of character codes that mean something only through the encoding of the current font (§9.6.6), so a generator owes you two lookups that must never be fused into one: text string to Unicode, and Unicode to whichever code selects the right glyph in that particular font. Copying the /V bytes into the operand, or keeping the low eight bits of each UTF-16 code unit, works for plain ASCII under WinAnsiEncoding and almost nowhere else. Producers bend the first lookup too; HotPDF's decoder also accepts a little-endian FF FE mark, which the specification never defined but which real files carry. Here is where the characters of the sample value land (codes from ISO 32000-1 Annex D):
| Character | Unicode | PDFDocEncoding (in /V) | WinAnsiEncoding (font) | StandardEncoding (font) |
|---|---|---|---|---|
| é | U+00E9 | E9 | E9 | no code |
| — | U+2014 | 84 | 97 | D0 |
| ‘ | U+2018 | 8F | 91 | 60 |
| ’ | U+2019 | 90 | 92 | 27 |
| ' (straight) | U+0027 | 27 | 27 | A9 |
| ` (grave) | U+0060 | 60 | 60 | C1 |
Read the StandardEncoding column slowly. It is the built-in encoding of the Latin Type 1 base fonts, and even below 0x80 it is no ASCII identity: 0x27 selects quoteright, 0x60 selects quoteleft, and the straight apostrophe sits up at 0xA9, so a generator that passes ASCII through untouched draws every typed apostrophe as a curly closing quote, while one that searches only the upper half of the table for curly quotes never finds them. StandardEncoding is also missing from the names /Encoding may take (MacRomanEncoding, MacExpertEncoding, WinAnsiEncoding); a font ends up with it by omitting /Encoding, or by an encoding dictionary without /BaseEncoding, and leaning on the font program. The other two columns disagree in a quieter way. WinAnsiEncoding and PDFDocEncoding share é but part company across most of 0x80–0x9F, where 0x92 is ’ in WinAnsi and ™ in PDFDocEncoding and 0x97 is an em dash in one and Š in the other. Hand a Windows-1252 byte string to a field as its value and /V reads Café Š ‚quoted™ before any appearance is drawn
Where the question mark comes from
A question mark in place of é can come from the font or from the runtime, and on screen you cannot tell which. The font route is legitimate: the /DA font is a bare Helvetica with no /Encoding and no embedded font file, its encoding is therefore StandardEncoding, and StandardEncoding has no code for é at all. The runtime route is less obvious. Delphi and Free Pascal convert between AnsiString, WideString and UnicodeString through the process code page without asking, and a font byte such as $E9 that passes through one of those conversions on its way into the content stream can come out as $3F. We traced that pattern in the Free Pascal build of the appearance generator, where the field still read Café while the Tj operand said Caf?; escaping parentheses and backslashes afterward changed nothing, because the byte was gone before escaping ran. So font bytes have to stay out of host string types. A literal string may carry any byte as a three-digit octal escape (§7.3.4.2), and HotPDF writes every code at or above 128 that way, $E9 as \351, leaving an operand that is pure ASCII; the same rule covers the text it writes when synthesizing missing annotation appearances before flattening. If you emit content yourself, it fits in a dozen lines:
// Serializes font codes (already chosen through the font's encoding)
// as an ASCII-only PDF literal string. Bytes in, ASCII out: no host
// code page conversion can touch the result
function PdfLiteralFromCodes(const Codes: TBytes): AnsiString;
var
I: Integer;
B: Byte;
begin
Result := '(';
for I := 0 to High(Codes) do
begin
B := Codes[I];
case B of
40, 41, 92: // ( ) \
Result := Result + '\' + AnsiChar(B);
0..31, 127..255: // three-digit octal, ISO 32000-1 §7.3.4.2
Result := Result + '\' +
AnsiChar(Ord('0') + (B shr 6)) +
AnsiChar(Ord('0') + ((B shr 3) and 7)) +
AnsiChar(Ord('0') + (B and 7));
else
Result := Result + AnsiChar(B);
end;
end;
Result := Result + ')';
end;
// Expected: codes 43 61 66 E9 (Café under WinAnsiEncoding) give (Caf\351)HotPDF chooses each code from the font the form actually carries: an explicit /WinAnsiEncoding is reverse-mapped over its defined positions only, a bare Latin base font without an embedded program is searched across all 256 StandardEncoding codes, and /Differences, embedded Type 1 vectors and TrueType fonts with a WinAnsi or MacRoman encoding go through tables built from glyph names. A character the font has no code for is drawn, and measured, as a question mark. Names are decoded before any comparison as well, so under §7.3.5 the state /Yes#20Please in /AP matches a /V of Yes Please
The input side deserves equal care, because THotPDF.SetFormFieldValue(const FieldName, Value: AnsiString) and its index overload store the bytes you pass exactly as given, and readers decode them as PDFDocEncoding. A value written as FEFF plus an even number of hex digits is recognized and saved as a hexadecimal UTF-16BE string, which carries any Unicode text past the system code page. The same convention, applied to field names and inherited values, is covered in setting form field values in a loaded PDF. Here it is with the encoding consequences spelled out:
uses
SysUtils, HPDFDoc;
// FEFF-prefixed UTF-16BE hex, which SetFormFieldValue stores as a
// hexadecimal text string instead of raw code-page bytes
function PdfTextHex(const S: UnicodeString): AnsiString;
var
I: Integer;
begin
Result := 'FEFF';
for I := 1 to Length(S) do
Result := Result + AnsiString(IntToHex(Ord(S[I]), 4));
end;
procedure FillNote;
var
Pdf: THotPDF;
begin
Pdf := THotPDF.Create(nil);
try
if Pdf.LoadFromFile('order-form.pdf') <= 0 then Exit;
// A literal 'Café — ‘quoted’' passed to an AnsiString parameter is
// converted through the system code page (warning W1058); character
// codes keep the source file encoding out of it as well
Pdf.SetFormFieldValue('Order.Note', PdfTextHex('Caf' + #$00E9 + ' ' +
#$2014 + ' ' + #$2018 + 'quoted' + #$2019));
// Text and choice fields get a regenerated /AP /N right here
Pdf.SaveLoadedDocument('order-form-filled.pdf');
finally
Pdf.Free;
end;
end;
// Expected Tj operand when /Helv has /Encoding /WinAnsiEncoding:
// (Caf\351 \227 \221quoted\222)
// Expected when /Helv relies on built-in StandardEncoding:
// (Caf? \320 `quoted') - the ` and ' select ‘ and ’ in that fontHow does auto-size choose a font size?
A /DA font size of 0 asks the writer to pick a size that fits the field (ISO 32000-1 §12.7.3.3), and most wrong answers come from measuring the wrong string. The Unicode value is the wrong string once a character turns into a question mark or changes code. The escaped operand is worse still, since \351 is four bytes of transport for one glyph. Take a single-line field 100 points wide and 20 tall holding Café — ‘quoted’ in Helvetica with WinAnsiEncoding: the fifteen glyphs add up to 7170 units in the Helvetica AFM, or 7.17 points of width per point of font size, and with 2 points of padding on each side the 96 usable points allow about 13.38 pt, while the height alone would allow 20. Width wins. Expected: roughly 13.38 pt. An estimate of half an em per escaped byte sees 27 bytes and settles near 7.1 pt, about half of what the box can hold. HotPDF measures the emitted codes against the font's own widths (standard AFM advances for the Latin base fonts, the declared /Widths for other simple fonts) and tests both directions at every candidate size, so each line's advance must fit the width and each line box, a quarter em below the baseline to three quarters above it, must fit the height; multiline fields are re-wrapped at each candidate first, because a smaller size moves the line breaks
Push-button captions go wrong the same way, only horizontally. Save (draft) is twelve glyphs but fourteen bytes once escaped as Save \(draft\), so at 10 pt the half-em estimate says 70 points where the Helvetica AFM says 52.24, and on an 80-point button the caption starts at x = 5 instead of 13.88, nearly 9 points left of center. HotPDF sums each character's Helvetica advance before escaping, and a caption with non-ASCII characters is encoded to the matching WinAnsi glyphs and measured by those glyphs
A form filled earlier by another tool may still carry appearances built from raw /V bytes or measured from escaped strings. Regenerating every text and choice appearance from the stored values rewrites them through the current font encoding and fit:
// Regenerates /AP /N for every text and choice field and reports the
// fields where no widget received an appearance (no usable /Rect, etc.)
procedure RefreshFieldAppearances(Pdf: THotPDF; Missed: TStrings);
var
I: Integer;
begin
for I := 0 to Pdf.GetFormFieldCount - 1 do
if Pdf.GetFormFieldType(I) in [lfftText, lfftChoice] then
if not Pdf.EnsureLoadedFieldAppearanceStream(I) then
Missed.Add(UTF8ToString(Pdf.GetFormFieldName(I)));
end;The Windows component and the Linux headless document
Everything above about single-byte codes describes the THotPDF component that you use from Delphi and C++Builder with the VCL, and from Lazarus on Windows, which reuses the font the form already declares in /DR and therefore lives inside that font's encoding. On Linux, THPDFHeadlessDocument.SetTextFieldValueWithAppearance(FieldIndex, Text, TrueTypeFont, Options) takes the value as a UnicodeString plus the bytes of a TrueType font you supply, and writes a Type0 font with CID glyph indices and a /ToUnicode map, so no single-byte encoding can run out of codes and no AnsiString sits in the path. Setting FontSize to 0 in THPDFHeadlessTextAppearanceOptions requests the same fit, with multiline text sized against width and height together
Which font should /DA name?
If you control the template, give the /DA font an explicit /Encoding /WinAnsiEncoding or embed a Unicode-capable font. Do not leave a Latin form leaning on the built-in StandardEncoding of a bare Helvetica. WinAnsi covers the accented letters, curly quotes and dashes that real names and addresses contain, and every viewer reads it the same way, whereas StandardEncoding cannot show é at all and moves the apostrophe, which is an odd foundation for a form that strangers will type into. Six characters settle the question for an existing template: fill one field with é, —, ‘, ’, a straight apostrophe and a grave accent, decompress its /AP /N stream, and read the Tj codes against the table above
For forms that must carry names beyond Western European Latin, skip the single-byte route and fill through an embedded Unicode font; editions and downloads are on the HotPDF Delphi PDF component page