ToUnicode Fix: NBSP and Soft Hyphen PDF Extraction in Delphi

PDFium Component for Delphi embeds the system fonts used by TPdf.AddText as CID fonts keyed by Unicode code point, so every CID carries exactly one ToUnicode mapping. That is what stops extracted spaces coming back as U+00A0 (no-break space) and hyphens as U+00AD (soft hyphen), both from the live document and from the saved file

The symptom is nasty because it is invisible. A search index misses "two-x" because the stored string contains a soft hyphen, a CSV export splits differently, a diff tool flags lines that look identical in every viewer. Nothing in the rendered page is wrong; only the Unicode behind the glyphs is

Why do extracted spaces come back as U+00A0?

Extracted spaces turn into U+00A0 because the ToUnicode CMap that PDFium generates in FPDFText_LoadFont is keyed by glyph, and one glyph can be reached from two code points. In Arial, glyph 3 serves both U+0020 and U+00A0, and the hyphen glyph serves both U+002D and U+00AD. The generated CMap therefore maps the same CID twice, once through a bfchar entry and once through an array-form bfrange, and whichever entry the reader's precedence rule favors becomes the extracted text

1 beginbfchar
<0003> <0020>
endbfchar
1 beginbfrange
<0003> <0010> [<00A0> ...]
endbfrange

For a long time this contradiction was harmless, since the PDFium reader let the lowest mapping win. An upstream change switched the reader to last-wins, and from that build on every space written with AddText extracted as NBSP and every hyphen as a soft hyphen. Note the pattern in the pairs: 0x20/0xA0 and 0x2D/0xAD differ only in the high bit, which is exactly what you would expect from a font whose cmap sends Latin-1 look-alikes to the same outline. If your extraction code was fine yesterday and now fails on invisible characters, dump the code points rather than trusting the debugger view; the basics of pulling text out are covered in extracting text from PDF documents with PDFium in Delphi

uses
  SysUtils, PDFium;

const
  // Space/U+00A0 and hyphen/U+00AD share one Arial glyph, as do
  // Greek Omega (U+03A9) and the Ohm sign (U+2126)
  Sample: WString = 'two-x'#$00A0'y'#$00AD'z '#$03A9#$2126;

function CodePoints(const S: WString): string;
var
  I: Integer;
begin
  Result := '';
  for I := 1 to Length(S) do
    Result := Result + 'U+' + IntToHex(Ord(S[I]), 4) + ' ';
end;

var
  Pdf: TPdf;
  Live, Reloaded: WString;
begin
  Pdf := TPdf.Create(nil);
  try
    Pdf.CreateDocument;
    Pdf.AddPage(1, 595, 842);
    Pdf.AddText(Sample, 'Arial', 12, 72, 770);
    Live := Pdf.Text;                  // live, unsaved document
    Pdf.SaveAs('codepoints.pdf');
  finally
    Pdf.Free;
  end;

  Pdf := TPdf.Create(nil);
  try
    Pdf.FileName := 'codepoints.pdf';
    Pdf.Active := True;
    Pdf.PageNumber := 1;
    Reloaded := Pdf.Text;              // after a full save and reload
  finally
    Pdf.Free;
  end;

  if (Live <> Sample) or (Reloaded <> Sample) then
    Writeln('Mismatch: ', CodePoints(Live), '/ ', CodePoints(Reloaded));
end;

Why patching the CMap after saving was not enough

Patching the saved file fixes only the saved file, and only if the patch keeps the CMap structure byte-for-byte intact. The first fix, RepairSubsetToUnicodeCMaps in the FPdfCompress unit, runs after every non-incremental TPdf.SaveAs and resolves each conflicting CID: the bfchar entry wins, a pair that differs only in the high bit resolves to the smaller base-Latin code point, and anything else keeps its first mapping

The interesting part is the negative result. Rebuilding the conflicting CMap cleanly, in either start-code or array form, looked like the obvious move, and PDFium rejected every rebuilt CMap outright, falling back to Identity. The only output the native reader accepted was an equal-length in-place replacement of the conflicting hex values, with block layout and CID coverage untouched. The second lesson was humbler: our note at the time blamed the in-memory case on the live document having no ToUnicode stream at all. Calling the DLL directly disproved that, since the live document carries the same ambiguous stream, which meant the real fix had to happen before PDFium ever generated the CMap. The repair routine stays in the library as a defense for PDFs produced by other PDFium-based tools

uses
  Classes, FPdfCompress;

var
  Source, Dest: TFileStream;
begin
  Source := TFileStream.Create('from-other-tool.pdf',
    fmOpenRead or fmShareDenyWrite);
  try
    Dest := TFileStream.Create('repaired.pdf', fmCreate);
    try
      // Equal-length edits only; files without a repairable conflict,
      // and cross-reference-stream or object-stream files, are copied as-is
      RepairSubsetToUnicodeCMaps(Source, Dest);
    finally
      Dest.Free;
    end;
  finally
    Source.Free;
  end;
end;

Keying the font by code point instead of by glyph

The root fix is to stop asking PDFium to generate the CMap at all. TPdf.LoadCachedFont now hands the system font bytes to TPdf.LoadUnicodeKeyedCidFont, which reads the font's own sfnt cmap table, preferring a format 12 subtable and falling back to format 4. Code points come back sorted and de-duplicated, and CID k+1 is assigned to the k-th code point, with CID 0 left as .notdef. An explicit CIDToGIDMap sends each CID to its glyph, so U+0020 and U+00A0 get two different CIDs that draw the same outline, and the ToUnicode CMap maps each CID to one code point only. The font is then loaded through FPDFText_LoadCidType2Font, the same entry point behind glyph-level writing in CID Type 2 font embedding with explicit CID-to-GID maps

// Condensed from TPdf.LoadUnicodeKeyedCidFont
SetLength(CidToGidMap, (Length(Entries) + 1) * 2);   // CID 0 = .notdef
for I := 0 to High(Entries) do
begin
  CidToGidMap[(I + 1) * 2]     := Byte(Entries[I].GlyphID shr 8);
  CidToGidMap[(I + 1) * 2 + 1] := Byte(Entries[I].GlyphID and $FF);
end;
ToUnicode := BuildUnicodeKeyedCidCMap(Entries);       // one CID, one code point
Result := FPDFText_LoadCidType2Font(Document, @Data[0], Length(Data),
  PAnsiChar(ToUnicode), @CidToGidMap[0], Length(CidToGidMap));

When FPDFText_SetText later writes a string, the reverse lookup lands on a single CID per character, so NBSP, soft hyphen and the Ohm sign each survive as themselves under either precedence rule, in memory and after any save. Because the saved file carries the component's own ToUnicode stream rather than an engine-generated one, RepairSubsetToUnicodeCMaps finds nothing to fix in it

Why can one bfrange entry wipe out a whole block?

A single bfrange whose CID run crosses an xxFF boundary makes PDFium discard the entire block it sits in. ISO 32000-1 §9.10.3 only lets the last byte of the destination vary within a range, but the CID side has its own trap: PDFium's HandleBeginBFRange derives the high CID as (low and $FFFFFF00) or (high and $FF). A run from CID 00FE to 0101 is therefore read as 00FE to 0001, low greater than high, and the whole block is marked invalid. The failure is silent: SetText succeeds, the page renders perfectly, and extraction returns U+0000 for every character in that block

BuildUnicodeKeyedCidCMap ends a run before either the code point or the CID reaches a low byte of FF, keeps every block within the 100-entry limit of the CMap grammar, and writes supplementary-plane code points as individual bfchar entries with UTF-16 surrogate-pair destinations, since incrementing a surrogate pair inside a range has no defined meaning; the surrogate side of that story is in emoji, CJK and surrogate pair handling in Delphi. A bfchar-only CMap would sidestep the boundary problem entirely, at several times the size

What does the code-point-keyed font not cover?

The code-point-keyed path covers every font that exposes a Unicode cmap subtable, and falls back to the old glyph-keyed behavior for the rest. The boundaries worth knowing before you rely on it:

• Symbol fonts with only a (3,0) cmap, and any font the CID path fails to load, go through FPDFText_LoadFont as before, so a glyph shared by two code points can still extract ambiguously there
• Without a format 12 subtable the map is limited to the BMP, and the entry count is capped at 65535 so every CID fits in two bytes above zero
• Incremental saves (saIncremental) skip RepairSubsetToUnicodeCMaps by design, because an incremental revision must stay append-only; the code-point-keyed fonts make that irrelevant for text the component writes itself
• TrueType Collections need extra care: GDI GetFontData returns the entire .ttc, and FPDFText_LoadCidType2Font has no face index parameter, so requesting NSimSun from simsun.ttc used to embed and render SimSun, face 0. The component now matches the family name against the name table (nameID 1 and 16) and extracts the requested face as a standalone sfnt before the cmap is parsed; if parsing fails, the collection bytes pass through and the behavior reverts to face 0

Text writing, font embedding and extraction share one page model across Delphi, C++Builder and Lazarus, and the full API is described on the PDFium Component for Delphi product page