ToUnicode Keying: NBSP and Soft Hyphen Extraction in Delphi

A search for "two-x" in a freshly generated PDF can come back with zero hits while the page plainly shows those five characters. Dump the extracted string as code points and the reason is right there: the hyphen came back as U+00AD SOFT HYPHEN, and the space after it as U+00A0 NO-BREAK SPACE. Both look identical on screen. The difference lives in the ToUnicode CMap, the table a reader consults to turn character codes back into Unicode, and it comes from one font glyph being shared by two code points

Plain string tools make the mismatch worse. A CSV export splits the line differently, a diff tool flags two lines that read the same in every viewer, and a spell checker treats "two-x" with a soft hyphen as an unknown word. Unicode normalization only half-rescues you: NFKC folds U+00A0 into an ordinary space because NBSP carries a compatibility decomposition, but U+00AD has no decomposition at all, so the soft hyphen survives every normalization form

Why do extracted spaces come back as U+00A0?

Extracted spaces turn into U+00A0 when the ToUnicode CMap is keyed by glyph and one glyph can be reached from two code points. In Arial, glyph 3 serves both U+0020 and U+00A0, and the hyphen glyph serves both U+002D and U+00AD. A CMap built from the glyph side has to say something about CID 0003, and PDFium's generator in FPDFText_LoadFont ends up saying two things: a bfchar entry that maps it to U+0020, and an array-form bfrange that maps the same CID to U+00A0

1 beginbfchar
<0003> <0020>
endbfchar
1 beginbfrange
<0003> <0010> [<00A0> ...]
endbfrange

ISO 32000-1 §9.10.3 describes how bfchar and bfrange map codes to Unicode, but it does not say which entry wins when two of them claim the same CID. Readers pick a precedence rule of their own. A reader that keeps the lowest or first mapping extracts a plain space; PDFium's reader keeps the last one, and since the array range follows the bfchar block, every space written through a glyph-keyed CMap extracts as NBSP and every hyphen as a soft hyphen. The same text can therefore extract differently in two viewers of the same file

The pairs follow a pattern. 0x20/0xA0 and 0x2D/0xAD differ only in the high bit, which is what a font produces when its cmap sends Latin-1 look-alikes to the outline already used for the ASCII character. Greek capital Omega (U+03A9) and the Ohm sign (U+2126) share a glyph in Arial too. If extraction code that used to pass now fails on characters you cannot see, print the code points instead of trusting the debugger's string view; the basics of reading text out of a page are in extracting text from PDF documents with PDFium in Delphi

How PDFium Component keys system fonts by code point

PDFium Component never asks the engine to generate that CMap for system fonts that carry a Unicode cmap. When TPdf.AddText needs a font, the component reads the font's own sfnt cmap table, preferring a format 12 subtable and falling back to format 4. Code points come back sorted and de-duplicated, and CID k+1 is assigned to the k-th code point, with CID 0 left as .notdef. An explicit CIDToGIDMap sends each CID to its glyph. U+0020 and U+00A0 therefore get two different CIDs that draw the same outline, and the ToUnicode CMap the component writes maps each CID to exactly one code point

The font is loaded through FPDFText_LoadCidType2Font, the same entry point behind glyph-level writing in CID Type 2 font embedding with explicit CID-to-GID maps. When FPDFText_SetText later writes a string, the reverse lookup lands on a single CID per character. Nothing about that depends on the reader's precedence rule, because there is no second mapping for the rule to choose between. The saved file carries the component's own ToUnicode stream as well, so extraction after a reload sees the same one-to-one table the live document used

The cost is a larger CIDToGIDMap: two bytes per code point the font covers. Suppose a font maps about 3,000 code points. The map then comes to roughly 6 KB, handed to the engine with each font. That trade is worth making every time. A few kilobytes buy extraction that reads the same in PDFium, in viewers that keep the first mapping, and in whatever indexer your customer runs next year

The round trip below writes a string that contains every troublesome pair, extracts it from the unsaved document, saves, reloads and extracts again. Both strings should match the input exactly; the CodePoints helper is there only to make a mismatch readable:

uses
  SysUtils, PDFium;

const
  // Space/U+00A0 and hyphen/U+00AD share one Arial glyph, as do
  // Greek Omega (U+03A9) and the Ohm sign (U+2126)
  Sample: WString = 'two-x'#$00A0'y'#$00AD'z '#$03A9#$2126;

function CodePoints(const S: WString): string;
var
  I: Integer;
begin
  Result := '';
  for I := 1 to Length(S) do
    Result := Result + 'U+' + IntToHex(Ord(S[I]), 4) + ' ';
end;

var
  Pdf: TPdf;
  Live, Reloaded: WString;
begin
  Pdf := TPdf.Create(nil);
  try
    Pdf.CreateDocument;
    Pdf.AddPage(1, 595, 842);
    Pdf.AddText(Sample, 'Arial', 12, 72, 770);
    Live := Pdf.Text;                  // live, unsaved document
    Pdf.SaveAs('codepoints.pdf');
  finally
    Pdf.Free;
  end;

  Pdf := TPdf.Create(nil);
  try
    Pdf.FileName := 'codepoints.pdf';
    Pdf.Active := True;
    Pdf.PageNumber := 1;
    Reloaded := Pdf.Text;              // after a full save and reload
  finally
    Pdf.Free;
  end;

  if (Live <> Sample) or (Reloaded <> Sample) then
    Writeln('Mismatch: ', CodePoints(Live), '/ ', CodePoints(Reloaded));
end.

Keep the NBSP and the soft hyphen in the sample string even if your documents rarely contain them. A test that writes only ASCII passes against a glyph-keyed CMap too, because nothing in it ever asks the reader to choose between two mappings

Why can one bfrange entry wipe out a whole block?

A single bfrange whose CID run crosses an xxFF boundary makes PDFium discard the entire block it sits in. ISO 32000-1 §9.10.3 only lets the last byte of the destination vary within a range, and the CID side has a matching trap in PDFium's parser: it derives the high CID as (low and $FFFFFF00) or (high and $FF). A run from CID 00FE to 0101 is read as 00FE to 0001, low greater than high, and the whole block is marked invalid. SetText succeeds, the page renders correctly, and extraction returns U+0000 for every character in that block

A code-point-keyed font is exactly where this bites. CIDs are dense and sequential, so any font with more than 255 code points produces CID runs that would cross 00FF, 01FF and so on if the writer simply merged consecutive entries. The CMap that PDFium Component writes ends a run before either the code point or the CID reaches a low byte of FF, and it splits the entries into blocks of at most 100, the limit the CMap grammar puts on each beginbfrange and beginbfchar section

Code points above U+FFFF get individual bfchar entries whose destinations are UTF-16 surrogate pairs, since incrementing a surrogate pair inside a range has no defined meaning. The surrogate side of the story is in emoji, CJK and surrogate pair handling in Delphi. A bfchar-only CMap would sidestep the boundary problem entirely, at several times the size, which is why ranges stay the main form and the split rules carry the burden

Repairing ToUnicode conflicts in files from other tools

RepairSubsetToUnicodeCMaps in the FPdfCompress unit fixes the glyph-keyed conflict in files the component did not write. TPdf.SaveAs runs it on every full, non-incremental save, and you can call it directly on any stream. For each CID that carries two destinations, the bfchar entry wins. With no bfchar for the CID, a pair that differs only in the high bit resolves to the smaller base-Latin code point. Any other conflict keeps its first mapping

The repair edits the conflicting hex values in place, with replacements of equal length. Block layout, CID coverage and every other byte of the stream stay as they were, so the CMap remains in the shape PDFium's reader already accepted. A cleanly rebuilt CMap would look tidier in a hex dump. It would also change the structure of a stream a strict reader already accepted, and that is a gamble with no upside when only a handful of destination values are wrong

uses
  Classes, FPdfCompress;

var
  Source, Dest: TFileStream;
begin
  Source := TFileStream.Create('from-other-tool.pdf',
    fmOpenRead or fmShareDenyWrite);
  try
    Dest := TFileStream.Create('repaired.pdf', fmCreate);
    try
      // Equal-length edits only; files without a repairable conflict,
      // and cross-reference-stream or object-stream files, are copied as-is
      RepairSubsetToUnicodeCMaps(Source, Dest);
    finally
      Dest.Free;
    end;
  finally
    Source.Free;
  end;
end.

The routine rebuilds a classic cross-reference table, so documents that use cross-reference streams, hybrid references or object streams pass through byte for byte. Incremental saves (saIncremental) skip the repair by design, because an incremental revision must stay append-only. For text the component writes itself, neither case matters: its code-point-keyed fonts never produce a conflict for the repair to find

Where the code-point-keyed path applies

The code-point-keyed path covers every font that exposes a Unicode cmap subtable, and PDFium's glyph-keyed loading handles the rest. Details worth knowing before you rely on it:

• Symbol fonts that carry only a (3,0) cmap, and any font the CID path cannot load, go through FPDFText_LoadFont, where a glyph shared by two code points is subject to the reader's precedence rule
• With only a format 4 subtable the map covers the Basic Multilingual Plane; a format 12 subtable on platform (3,10) or the Unicode platform adds supplementary-plane code points
• Glyph 0 and surrogate code points are left out, and the entry count is capped at 65535 so every CID fits in two bytes above zero
• TrueType Collections need extra care: GDI GetFontData returns the whole .ttc, and FPDFText_LoadCidType2Font has no face index parameter, so passing the collection through would embed face 0, SimSun, when you asked for NSimSun from simsun.ttc. The component matches the family name against the font's name table (nameID 1 and 16) and hands the requested face over as a standalone sfnt; a collection it cannot parse is passed through whole and uses face 0
• If no system font of the requested name can be read, AddText hands the font name to FPDFPageObj_NewTextObj, which looks it up among PDFium's standard fonts

Fold invisible characters at the index, keep them in the text

Once extraction returns the real code points, a different question comes up: what should a search index do with a genuine NBSP or soft hyphen that the author typed on purpose? Fold them in the indexing layer and leave TPdf.Text alone. The extracted string is the record of what the document says; an index is a lossy view built for matching, and the folding rules belong to it

The helper below is a function you add to your own code. It maps NBSP to a space and drops the soft hyphen, which per the Unicode Standard only marks a place where a line may break. It deliberately leaves U+2126 untouched. NFC normalization would turn the Ohm sign into Omega, because U+2126 has a canonical singleton decomposition to U+03A9, and whether a datasheet search should match those two is a product decision rather than an extraction one:

uses
  SysUtils, Classes, PDFium;

// Your own helper for a search index; not part of PDFium Component
function FoldForIndex(const S: string): string;
begin
  Result := StringReplace(S, #$00A0, ' ', [rfReplaceAll]);
  Result := StringReplace(Result, #$00AD, '', [rfReplaceAll]);
end;

procedure IndexDocument(const FileName: string; Lines: TStrings);
var
  Pdf: TPdf;
  P: Integer;
begin
  Pdf := TPdf.Create(nil);
  try
    Pdf.FileName := FileName;
    Pdf.Active := True;
    for P := 1 to Pdf.PageCount do
    begin
      Pdf.PageNumber := P;
      Lines.Add(FoldForIndex(Pdf.Text));   // exact text stays in Pdf.Text
    end;
  finally
    Pdf.Free;
  end;
end;

Run IndexDocument on the codepoints.pdf file from the first example and the indexed line should read "two-x yz ΩΩ", eleven characters with an Ohm sign as the last one, while Pdf.Text still returns all twelve original characters, NBSP and soft hyphen included

Code-point-keyed fonts, the CMap repair and text extraction all sit on the same page model in Delphi, C++Builder and Lazarus; the font and text API is covered in full on the PDFium Component for Delphi product page