PDF Library for Delphi can match text by canonical equivalence rather than by code unit, so a query typed as a precomposed character finds content stored as a base letter plus a combining mark, and the reverse. Two search options control it: soCanonicalEquivalent enables Unicode normalisation during matching, and soGraphemeClusters constrains every hit and every wildcard step to whole grapheme clusters
The bug this fixes is one of the most reported and least understood in document search. A user searches for a name, sees no results, copies the name out of the document, pastes it into the search box, and finds it. Nothing is broken in an obvious way: the two strings look identical, print identically, and compare unequal, because one is U+00E9 and the other is U+0065 followed by U+0301
Why does the same word compare unequal?
Unicode allows several encodings for the same abstract character. Latin letters with diacritics exist as precomposed code points and as base plus combining sequences. Hangul syllables exist as precomposed syllables and as decomposed jamo. Which one a PDF contains depends on the producer, the platform, and sometimes the font, and none of that is visible to the person doing the search
The reason simple case folding does not solve this is structural rather than incidental. Case folding and accent folding are one-to-one at the code unit level: the folded string has the same length as the original, so a match position in the folded text is a match position in the original. Normalisation is not one to one. One precomposed character becomes two or three code units, a decomposed sequence collapses back to one, and after that transformation, positions no longer line up with the text you extracted
Keeping hit coordinates pointing at the original text
This is the part that determines whether normalised search is usable rather than merely correct. Every code unit produced by normalisation records the start and end position of the original UTF-16 text that produced it. Recursive decompositions inherit the source range of their parent, compositions merge the ranges of their inputs, and when a match is found the library scans the mapping interval for the smallest start and largest end
The effect is that MatchStart, MatchLength, the context strings and both replacement entry points all continue to address the original extracted text, not the normalised intermediate. Without that mapping, a normalised search could tell you a hit exists but not reliably where it was, which makes highlighting wrong and redaction dangerous
The normaliser itself is self-contained: compact tables for canonical decomposition, composition and canonical combining class from Unicode 15.1, with Hangul handled by the algorithmic rules rather than by table entries. Nothing is loaded from an external data file and no platform normalisation API is called, so a Windows service, a Linux daemon and an FPC build all produce identical results on the same input
Searching with canonical equivalence
Options are a set, so canonical equivalence combines with the existing behaviours such as whole word matching, wildcards and diacritic-insensitive folding:
uses
PDFlibrary;
var
Lib: TPDFlib;
Hits: array of TPDFlibSearchHit;
Found, I: Integer;
begin
Lib := TPDFlib.Create;
try
Lib.LoadFromFile('contracts.pdf', '');
SetLength(Hits, 500);
Found := Lib.SearchText('Bäcker', [soCanonicalEquivalent, soWholeWord],
'', Hits); // empty page range = whole document
for I := 0 to Found - 1 do
Log(Format('page %d: "%s" at %d (%d chars)',
[Hits[I].Page, Hits[I].MatchText, Hits[I].MatchStart,
Hits[I].MatchLength]));
finally
Lib.Free;
end;
end;Normalisation is opt-in for a reason. Building the NFD text and its position mapping costs work, and most searches over ASCII-only documents never need it. When the option is used, each text block caches two transformed forms, one with combining marks removed and one without, so a batch of queries over the same block normalises once rather than once per query. Case folding continues to travel the cheaper one-to-one path unchanged
What breaks without grapheme cluster boundaries?
Code units are not characters, and characters are not what users perceive. A flag emoji is two regional indicator code points. A family emoji is several code points joined by zero-width joiners. An Indic conjunct is a consonant, a virama and another consonant. A letter with two stacked accents is three code points. Matching or cutting in the middle of any of these produces a fragment that renders as garbage
soGraphemeClusters constrains both ends of every hit, literal or wildcard, to complete extended grapheme cluster boundaries. The segmentation implements the extended rules: CR and LF pairing, control characters, Hangul syllable classes, Extend and SpacingMark, Prepend, emoji ZWJ sequences, regional indicator pairing and Indic conjunct breaks. A boundary is never produced inside a surrogate pair, which alone eliminates a whole class of corrupted results on any content beyond the basic multilingual plane
The option also governs wildcard consumption, which is where a naive implementation would still cut incorrectly. The single-character wildcard advances exactly one complete cluster, and backtracking for the run wildcard moves only between cluster boundaries:
// Without soGraphemeClusters, "?" can consume half a cluster and
// return a hit whose text ends in a dangling combining mark
Found := Lib.SearchText('c?té',
[soWildcards, soCanonicalEquivalent, soGraphemeClusters], '', Hits);
// The same boundaries protect replacement, so redaction and
// content rewriting never split an emoji or an accented letter
Replaced := Lib.SearchAndReplaceText('naïve', 'plain',
[soCanonicalEquivalent, soGraphemeClusters], '1-20');Choosing options for a real workload
Three combinations cover most cases. For an internal document search box, soCanonicalEquivalent plus soDiacriticInsensitive gives the forgiving behaviour users expect, matching both encoding forms and both accented and unaccented spellings. For legal or compliance search, where a false positive has a cost, use soCanonicalEquivalent with soCaseSensitive and soWholeWord and leave accent folding off, so equivalence is exact and encoding-independent
For anything that modifies the document, add soGraphemeClusters without exception. A search that returns a slightly wrong range only misleads a reader; a replacement or redaction that uses the same wrong range writes the mistake into the file. The consequences of getting removal ranges wrong are covered in true redaction and content removal
When throughput matters, prefer the batch entry points. SearchTextBatch runs every non-empty query while each page's text blocks are resident, which avoids re-extracting a page per query and reuses the cached normalisation, and the streaming variants emit hits without a caller-sized buffer. The extraction model underneath is described in text search and page element enumeration
Scripts where this is not optional
For Korean, canonical equivalence is the difference between finding a name and not finding it, because precomposed syllables and decomposed jamo are both common in real documents. For Vietnamese, stacked diacritics make composition form entirely producer-dependent. For Indic scripts, conjunct handling decides whether a hit boundary lands in a legible place. For Japanese and Chinese, the search side is comparatively simple, though the layout side is not, as described in vertical writing for Japanese and Chinese
The rule of thumb is short: if the corpus contains any language other than English, turn canonical equivalence on and measure the cost before deciding it is too expensive. In most document sets it is not, and the alternative is a search feature that quietly fails on exactly the names your users care most about finding
Unicode-aware search, extraction, redaction and text rewriting share one engine for Delphi, C++Builder and Free Pascal; the complete feature list is on the PDF Library for Delphi page