A Japanese PDF goes through text extraction and comes back as stray Latin letters, kanji that never appeared on the page, or a line that stops dead after a wide gap. In PDFlibPas, CJK text extraction splits each string by the font CMap codespace and resolves every CID through the descendant CIDSystemInfo before it produces any Unicode
Take the two bytes 30 42 inside a string operand. With a simple font they are two characters, 0 and B. Under a Type 0 font with Identity-H they form one two-byte code, 0x3042, which is CID 12354, and an extractor that prints that number as a code point produces U+3042, the hiragana あ: a real character, so nobody flags it. If the descendant CIDFont declares Adobe-Japan1, CID 12354 names whatever character the Japan1 collection assigns to it, and Japan1 numbering has no arithmetic relationship with Unicode. The font dictionaries decide which reading is right. An Identity-H Type 0 font over a CFF CIDFont bound to Adobe-Japan1 is one of the most common shapes Japanese producers write, and a shortcut tuned on Identity-ordered subsets misreads them
From bytes to text: four lookups and a width
Each character in a composite font passes through a fixed chain of lookups, every stage driven by its own dictionary entry. The table lines the stages up with the entry ISO 32000-1 points to and the usual misreading at each step
| Stage | Input and output | Governed by | Common misreading |
|---|---|---|---|
| Split bytes into codes | String bytes to codes of one to four bytes | Codespace ranges of the Type 0 /Encoding CMap (ISO 32000-1 §9.7.6.2, Adobe Technical Note #5014) | Treating range endpoints as integers, or assuming every code is two bytes |
| Code to CID | Code to CID | cidrange and cidchar entries, or Identity-H / Identity-V, where the code equals the CID (§9.7.5.2) | Reading Identity-H as code equals Unicode |
| CID to meaning | CID to a character collection | Registry, Ordering and Supplement in the descendant CIDSystemInfo (§9.7.3) | Using a collection CID as a glyph index in a substitute font |
| Code to Unicode | Complete source bytes to Unicode | /ToUnicode, else the Registry-Ordering-UCS2 map of a known Adobe collection (§9.10.2) | Printing the CID number as a code point, or using the Encoding CMap as a Unicode map |
| CID to width | CID to horizontal advance | /W and /DW in the CIDFont (§9.7.4.3) | Widths rebuilt from a font program whose hmtx was misread |
Why does a codespace range not contain every code between its endpoints?
Because a multi-byte codespace range is a rectangle in byte space. ISO 32000-1 §9.7.6.2 says a code matches a range only when it has the same length as the bounds and each of its bytes lies between the corresponding bytes of the lower and upper bound. Shift-JIS lead bytes and trail bytes have separate legal ranges, so <8140> <82BF> means a first byte of 0x81–0x82 and a second byte of 0x40–0xBF. The code <8201> sits numerically between the endpoints and is still outside the range, since 0x01 is below the trail-byte floor. A validator that tests First <= Code <= Last on 16-bit integers accepts it. The same habit lets a cidrange pass a coverage check against <8140> <82FC> while the strip 81FD–823F inside it is not valid at all
The ranges <8140> <82BF> and <81C0> <82FF> overlap as integer intervals yet share no code, because their second-byte bounds, 40–BF and C0–FF, never meet. Real overlap is forbidden: §9.7.6.2 states that codespace ranges shall not overlap, and that rule is what guarantees a code matches at most one range. Producers break it anyway. The usual case is an embedded CMap that inherits a predefined CMap through usecmap and redeclares part of the parent range at the same width. PDFlibPas tolerates that case as the union of the two ranges, because a code of a given length decodes to the same value whichever of them it matches. A shorter code that is a prefix of a longer one is another matter. With a one-byte <00> <FF> range declared beside a two-byte range starting at <8140>, the one-byte range claims 0x81 before the two-byte range is ever tried, the longer codes become unreachable, and PDFlibPas rejects the CMap. For bytes that match no range, §9.7.6.3 prescribes the recovery: return to where the bad code started, choose the codespace with the longest partial match on the leading bytes, prefer the one with shorter codes on a tie, and use the shortest code length when even the first byte matches nothing. PDFlibPas implements that algorithm as written, so one corrupt code cannot push every later code out of alignment
Identity-H gives you the CID; CIDSystemInfo gives it meaning
Identity-H says every code is two bytes and the code equals the CID (ISO 32000-1 §9.7.5.2), with Identity-V doing the same for vertical writing. What the CID stands for lives in the CIDSystemInfo dictionary of the descendant CIDFont, whose Registry, Ordering and Supplement name a character collection (§9.7.3). Adobe-Identity-0 means the CIDs are private to this one font program; for an embedded CIDFontType2 with the default identity CIDToGIDMap they are glyph indices into the subset, and only a ToUnicode stream can turn them into text. Adobe-Japan1 means the opposite. CID 12354 is a Japan1 character whichever font program draws it, and used as a glyph index in a substitute font it paints an unrelated glyph. PDFlibPas binds the effective Encoding CMap to the descendant collection by Registry and Ordering, so a Japan1 CID keeps its Japan1 meaning whichever CMap selected it. Vertical fonts add WMode and the W2 and DW2 metrics on top of this, covered in the guide to vertical Japanese and Chinese text in Delphi
uses
System.SysUtils, System.Math, PDFlibrary;
procedure ListCompositeFonts(const FileName: string);
var
PDF: TPDFlib;
Fonts: array of TPDFlibDocumentFont;
Total, I: Integer;
Verdict: string;
begin
PDF := TPDFlib.Create;
try
if PDF.LoadFromFile(FileName, '') <> 1 then
raise Exception.CreateFmt('Cannot load %s (error %d)',
[FileName, PDF.LastErrorCode]);
SetLength(Fonts, 64);
Total := PDF.EnumDocumentFonts(Fonts);
if Total > Length(Fonts) then
begin
SetLength(Fonts, Total);
Total := PDF.EnumDocumentFonts(Fonts);
end;
for I := 0 to Min(Total, Length(Fonts)) - 1 do
begin
// 9 = CIDFontType0 (CFF based), 10 = CIDFontType2 (TrueType based),
// 11 = CJK font with built-in metrics
if (Fonts[I].FontType < 9) or (Fonts[I].FontType > 11) then
Continue;
if Fonts[I].HasToUnicode then
Verdict := 'ok: ToUnicode present'
else if Pos('Identity', Fonts[I].Encoding) = 1 then
Verdict := 'RISK: Identity encoding without ToUnicode'
else
Verdict := 'check: predefined CMap without ToUnicode';
Writeln(Format('%-36s %-16s embedded=%-5s %s',
[Fonts[I].BaseFont, Fonts[I].Encoding,
BoolToStr(Fonts[I].Embedded, True), Verdict]));
end;
finally
PDF.Free;
end;
end;Run an inventory like this before trusting any extracted CJK string. Composite fonts without ToUnicode are the ones to watch. With an Identity encoding on top, only the descendant collection can still recover text. The record does not carry CIDSystemInfo, so a RISK line is your cue to open the descendant dictionary and read its Ordering: Identity there means the characters cannot be recovered without a ToUnicode stream, while Japan1 or GB1 means the collection route in the next section still applies. Because EnumDocumentFonts returns the full count even when the array is too small, the example grows the array and calls again
What should an extractor do when a CJK font has no ToUnicode?
Use ToUnicode whenever it exists, keyed on the complete source bytes of each code, and fall back only along the route ISO 32000-1 §9.10.2 lays out. That route covers composite fonts that use a predefined CMap other than Identity-H and Identity-V, or whose descendant CIDFont uses Adobe-GB1, Adobe-CNS1, Adobe-Japan1 or Adobe-Korea1: map the code to a CID, build the name Registry-Ordering-UCS2 (Adobe-Japan1-UCS2, for instance), and look the CID up in that map. Treating the Encoding CMap as a Unicode map is the common wrong turn. Identity-H is a statement about CIDs and says nothing about Unicode, and a CID printed as a code point looks like text without being text. When neither ToUnicode nor a known collection is available, report the character as unmapped. We would rather hand a downstream indexer a visible gap than a wrong kanji that looks right, because the gap gets noticed while the substitute gets indexed and searched. PDFlibPas works the same way. Its compiled-in Adobe UCS2 tables resolve ToUnicode streams that inherit from them through UseCMap, and an unknown base name leaves the mapping incomplete. The extraction calls themselves are introduced in the overview of text, image and font extraction with PDFlibPas. The loop below adds a scan for replacement characters, control codes and private-use values, the usual residue of codes that had nowhere to go
uses
System.SysUtils, PDFlibrary;
function LooksUnmapped(C: WideChar): Boolean;
begin
Result := (C = #$FFFD) or (C < #$0020) or
((C >= #$E000) and (C <= #$F8FF));
end;
procedure ExtractCJKPages(const FileName: string);
var
PDF: TPDFlib;
Page, ListID, Count, I, J, Bad: Integer;
S: WideString;
begin
PDF := TPDFlib.Create;
try
if PDF.LoadFromFile(FileName, '') <> 1 then
raise Exception.CreateFmt('Cannot load %s', [FileName]);
for Page := 1 to PDF.PageCount do
begin
PDF.SelectPage(Page);
ListID := PDF.ExtractPageTextBlocks(4); // 4 = split words
if ListID = 0 then
Continue;
try
Count := PDF.GetTextBlockCount(ListID);
for I := 1 to Count do // blocks are 1-based
begin
S := PDF.GetTextBlockText(ListID, I);
Bad := 0;
for J := 1 to Length(S) do
if LooksUnmapped(S[J]) then
Inc(Bad);
if Bad > 0 then
Writeln(Format('page %d block %d font %s: %d of %d chars suspect',
[Page, I, PDF.GetTextBlockFontName(ListID, I), Bad, Length(S)]))
else
Writeln(S);
end;
finally
PDF.ReleaseTextBlocks(ListID);
end;
end;
finally
PDF.Free;
end;
end;The TJ gap that swallows the rest of a line
Text that vanishes after a wide gap in a CJK line is usually a byte-alignment problem. A TJ array mixes strings with numbers, and a number such as -1200 pushes the next glyph to the right by 1.2 times the font size (ISO 32000-1 §9.4.3), a gap most extractors turn into a space. The naive way to do that is to append a 0x20 byte to the raw buffer that collects the array strings and decode the buffer afterward, which is harmless with a simple font and ruinous with two-byte codes: the buffer 30 42 30 44, the space byte and 30 46 30 48 decodes as 0x3042, 0x3044, 0x2030, 0x4630 and a dangling 0x48. Every code after the gap pairs the wrong bytes, most of those accidental codes map to nothing in a subset font, and the rest of the array drops out of the output. The standard points at the same pitfall from the other side, since word spacing applies only to the single-byte code 32 (§9.3.3), so a two-byte 0x0020 in an Identity-H string is not a space at all. PDFlibPas decodes each string of a composite font through its codespace first and only then joins the decoded text with a space where the gap warrants one
Widths, and a uint16 that is easy to read as signed
Widths are the last link. Every word-gap decision and block boundary is computed from glyph advances. A CIDFont takes its advances from /W, with /DW covering every CID the array omits and defaulting to 1000 (ISO 32000-1 §9.7.4.3). Those numbers come from whatever tool read the font program. In an OpenType font, hhea.numberOfHMetrics says how many full advance-and-bearing records hmtx holds, and glyphs past that count reuse the last advance. During PDFlibPas development a large pan-CJK font came back with every glyph at the default width, and the font itself turned out to be fine. The cause sat at the end of the hhea table: nearly every field before numberOfHMetrics is a signed 16-bit value, so a reader walking the table with one signed read after another treats the last count the same way, yet that count is a uint16. In fonts with more than 32767 glyphs the count goes negative, the reader skips hmtx entirely, and every advance falls back to the default. Latin fonts and modest CJK fonts never show the mistake. PDFlibPas reads the count as unsigned. A PDF written from such metrics carries a /W array in which proportional kana, Latin letters and full-width ideographs all claim one advance. Extraction then places characters on a grid the page does not follow, merging words in one place and splitting them in another. A long run of identical 1000-unit entries for a proportional font in /W is the giveaway
When the next garbled CJK page arrives, read the descendant CIDSystemInfo and the ToUnicode entry of every font first, since those two entries explain most of the output, and the PDFlibPas Delphi PDF Library applies the decoding rules above to every page it extracts