Pull emoji or a Japanese family-register name out of a PDF as text, and the output shows a box, a question mark, or nothing at all where the character should be. PDFium Component's Character[] property is usually why: it reads each glyph through FPDFText_GetUnicode, which returns a full Unicode code point as an unsigned 32-bit value, then exposes it to Delphi as a single 16-bit WideChar. Any code point past U+FFFF cannot make that trip in one piece, and the corruption never shows up while you are looking at the rendered page, because rendering and text extraction run through separate code paths in PDFium — a document can display its emoji perfectly and still hand you rubbish the moment you read Character[] in a loop and build a string out of it
The Basic Multilingual Plane and why WideChar stops at U+FFFF
Delphi's WideChar is a 16-bit type that can only hold one UTF-16 code unit. Unicode's Basic Multilingual Plane, the range U+0000 to U+FFFF, fits inside that exactly, which is why Latin, Cyrillic, Greek, and the common CJK Unified Ideographs block all round-trip through a single WideChar without incident. Two families of characters routinely fall outside it in real documents: emoji, many of them in the Emoticons block starting at U+1F600, and rare CJK ideographs from CJK Unified Ideographs Extension B, the range U+20000 through U+2A6DF reserved for less common Chinese, Japanese, and Korean characters including many personal and place names. UTF-16 handles anything above U+FFFF with a surrogate pair — two 16-bit code units, a high surrogate in the range $D800 to $DBFF followed by a low surrogate in $DC00 to $DFFF, that together encode one code point — and the maths behind that pairing is fixed enough to demonstrate directly in Pascal
function ToSurrogatePair(CodePoint: LongWord; out Hi, Lo: WideChar): Boolean;
var
V: LongWord;
begin
Result := CodePoint > $FFFF;
if Result then
begin
V := CodePoint - $10000;
Hi := WideChar($D800 + (V shr 10));
Lo := WideChar($DC00 + (V and $3FF));
end;
end;Feed U+1F600, the grinning-face emoji, through that function and the result is a high surrogate of $D83D and a low surrogate of $DE00, two 16-bit values, not one. Neither half means anything on its own; a lone $D83D sitting in a string with no $DE00 behind it is a dangling surrogate, and most text-handling code that encounters one either drops it, substitutes a replacement glyph, or raises an error
Why does FPDFText_GetUnicode return a value Character[] cannot hold?
FPDFText_GetUnicode returns a LongWord, a full 32-bit value, because PDF text encoding already carries the complete Unicode scalar value for every glyph. A PDF's ToUnicode CMap maps character codes to Unicode text, and when a glyph represents what is informally called an astral-plane character — anything past the Basic Multilingual Plane — that mapping is a full code point, not a 16-bit fragment. PDFium decodes it back to a scalar value internally and returns it across the DLL boundary through FPDFText_GetUnicode, and that boundary is exactly where a 32-bit value has to become something a Delphi property can hand back to your code
The obvious implementation is WideChar(FPDFText_GetUnicode(TextPage, Index)), and it is also the wrong one. A hard cast from a 32-bit value into a 16-bit type keeps only the low 16 bits and throws the rest away silently, with no exception and no range check. For U+1F600 that means keeping $F600 and losing the fact that the true value was ever above U+FFFF, which produces a code unit that is not even a valid dangling surrogate, just an unrelated Basic Multilingual Plane character that happens to share those low bits. Concatenate a few thousand of those into a string and downstream code has no way left to tell a corrupted character from a legitimate one
What Character[] and Charcode[] return for astral-plane code points now
PDFium Component's Character[] and Charcode[] properties return U+FFFD, the Unicode replacement character, whenever the underlying code point exceeds U+FFFF, instead of silently truncating it. That guard sits directly inside the property getter behind Character[]
function TPdf.GetCharacter(Index: Integer): WideChar;
var
Code: LongWord;
begin
LoadTextPage;
Code := FPDFText_GetUnicode(FTextPage, Index);
if Code > $FFFF then
Result := #$FFFD // astral-plane code point: cannot fit in one WideChar
else
Result := WideChar(Code);
end;Returning U+FFFD instead of a truncated fragment is a deliberate, narrow fix rather than a redesign. Character[] and Charcode[] are typed as WideChar on both TPdf and TPdfView, and widening that return type to carry a full code point would break every existing caller that expects one glyph per index to mean one 16-bit value. U+FFFD is the Unicode Standard's own designated placeholder for exactly this situation, so a caller that checks for it gets a defined, documented signal instead of silently wrong data. One boundary case worth knowing: U+FFFD is also a legitimate character in its own right, so on the rare document that already contains a genuine replacement-character glyph, that index is indistinguishable from a truncated astral character by value alone
How do you extract emoji and CJK Extension B text correctly in Delphi?
Call Text instead of walking Character[] whenever the actual textual content matters, because Text reads through FPDFText_GetText and returns a full WString with proper surrogate pairs for every astral-plane character in range, rather than one fixed-width value per index. Pdf.Text(0, MaxInt), or the shorthand Pdf.Text, extracts an entire page correctly in one call, and Pdf.Text(StartIndex, Count) pulls a smaller range the same way. Character[] still earns its place when you only need position, font, or flag data at an index and never touch the code point itself — CharacterOrigin[], FontSize[], and CharacterMapError[] do not care whether the underlying glyph was astral
function ExtractLineSafely(Pdf: TPdf): WString;
var
I: Integer;
begin
Result := '';
for I := 0 to Pdf.CharacterCount - 1 do
if not (Pdf.CharacterGenerated[I] or Pdf.CharacterMapError[I]) then
Result := Result + Pdf.Text(I, 1); // full code point, never a truncated WideChar
end;The skip-generated-and-unmapped check in that loop is the same pattern used for plain-text extraction in extracting text from PDF documents with PDFium Component; the only change is the last line, which trades a direct Character[I] append for a one-index call into Text so astral characters arrive as complete surrogate pairs instead of replacement placeholders
Where this actually bites: chat exports, personal names, and embedded CJK fonts
Emoji show up wherever a PDF captures informal communication: exported chat logs, app-store review dumps, ticketing-system transcripts saved to PDF for a compliance archive. CJK Extension B shows up in a narrower but higher-stakes place, personal and place names, because Japanese family registers, Chinese household registration records, and Taiwanese identity documents are classic sources of characters that never made it into the common CJK block. A payroll or identity-verification pipeline that extracts names from scanned government paperwork is exactly the kind of workload where a silently mangled character turns into a failed match rather than a cosmetic glitch
Rare CJK ideographs also tend to travel with font problems, not just encoding ones, because a font has to carry a glyph for a U+20000-range code point before anything can render at all, and few installed system fonts do. Anyone already walking FontIsEmbedded[] per character the way reading PDF font properties with PDFium Component describes should check the same index for both problems together: an index that returns U+FFFD from Character[] and reports a non-embedded font is a document that will neither extract nor print that character correctly, and the fix belongs upstream in how the PDF was produced, not in your extraction code
The Character[], Charcode[], and Text properties described here are part of the standard PDFium Component for Delphi and C++Builder