In a TrueType font that keeps the standard Macintosh glyph order, the word "Total" can sit in a PDF content stream as <0037005200570044004F> Tj. Those five two-byte values are glyph numbers inside the embedded font, and nothing in the string says they draw T, o, t, a and l. Read those bytes as UTF-16 and you get the string 7RWDO, while the page itself shows Total on screen, because drawing a glyph only needs the code-to-glyph mapping and getting text back needs a code-to-Unicode mapping that the file may never have carried
Why does extracted text come out as gibberish?
A content stream records character codes, and their meaning lives entirely in the font. The Tj and TJ operators (ISO 32000-1 §9.4.3) carry byte strings that are interpreted through whatever font the preceding Tf selected: byte 0x41 is the letter A under WinAnsi, an arbitrary glyph in a subset font, or half of a two-byte CID in a composite CJK font. ISO 32000-1 §9.10 treats text extraction as exactly this decoding problem, mapping each code back to Unicode from whatever the font dictionary provides, and it says plainly that a conforming file is not required to provide enough information to do it
That clause is behind every "copy-paste from this PDF produces garbage" report. A producer that embeds a subset font with no /ToUnicode table has written a valid file that renders correctly. The glyphs are all there. What is missing is the reverse map, which an extractor has to rebuild from secondary evidence: the font's declared encoding, glyph names, Adobe's published character collections, the cmap table inside the embedded font program. How many of those an API tries, and what it hands you when all of them fail, is what separates extraction libraries in practice
Reading-flow text with ExtractLoadedPageText
HotPDF Delphi Component returns the reading-flow text of a loaded page through function ExtractLoadedPageText(PageIndex: Integer; out AText: UnicodeString): boolean. Page indexes are zero-based, the result is a native Delphi UnicodeString, and the function returns False when the page has no readable content stream instead of raising. It works on any PDF you load: the invoice a customer emailed, the report a scanning bureau delivered, the contract produced by software nobody can name anymore
var
Pdf: THotPDF;
PageCount, I: Integer;
PageText, AllText: UnicodeString;
begin
Pdf := THotPDF.Create(nil);
try
PageCount := Pdf.LoadFromFile('invoice.pdf'); // returns the page count
AllText := '';
for I := 0 to PageCount - 1 do
if Pdf.ExtractLoadedPageText(I, PageText) then
AllText := AllText + PageText + #13#10;
finally
Pdf.Free;
end;
end;
For the whole document in one string, ExtractLoadedPagesText(PageRange, PageSeparator) takes a one-based range such as '1,3-5,7-' (an empty string means every page) and puts PageSeparator, a form feed #12 by default, between pages. Large documents go better through ExtractLoadedPagesTextToStream, which writes UTF-8 page by page to a TStream and returns how many pages it processed
The text holds what the page shows. Glyphs whose box center falls outside the visible area (the crop box within the media box) are left out, so printer's marks on a trimmed page do not leak into your index. Text inside Form XObjects that the page paints is included, clipped to each form's /BBox. A character painted twice at the same spot, the old trick for faking a bold face, is read once; fake bold and duplicate glyphs in text extraction walks through how that case is detected
Line breaks follow a simple geometric rule. A new line starts when the glyph origin moves across the writing direction by more than half the text height, taking the larger of this glyph's and the previous glyph's ascent-to-descent height in page units. Measuring the glyph box instead of the Tf size operand is the part that matters: plenty of producers size text through the text matrix and pass 1 to Tf, and a threshold built on that operand would break at every raised superscript and put each character of rotated text on its own line. Word spaces work the same way. Where a producer separates words by moving the text position with a TJ adjustment or a Td instead of showing a space, a space is added once the gap after the previous glyph's own width exceeds 0.15 of the glyph height, and never between two Chinese, Japanese or Korean characters. Codes the decoder cannot resolve become spaces, so word boundaries survive even when individual glyphs do not. Horizontal text comes out in content-stream order; vertical CJK runs are regrouped into columns read right to left, top to bottom. Word spaces and line breaks in page text covers those thresholds
Grid, columns or structure order
When position carries meaning (tables, forms, code listings, anything you will diff, grep or parse by column), ExtractLoadedPageTextLayout is the better call. It clusters glyphs into baselines, sorts each baseline by X, and reproduces horizontal and vertical whitespace on a monospaced character grid sized from the page's median glyph advance and font size. Wide gaps on a baseline become runs of spaces, large gaps between baselines become blank lines
uses System.IOUtils;
...
var
Grid: UnicodeString;
begin
if Pdf.ExtractLoadedPageTextLayout(0, Grid) then
TFile.WriteAllText('page1.txt', Grid, TEncoding.UTF8);
// Columns, indentation and table alignment survive as
// spaces and blank lines on a character grid
end;
Both calls decode through the same machinery, so choosing one costs nothing in fidelity. What the grid will not do is reflow prose: a two-column article comes out as both columns side by side, faithfully, which is right for a diff and wrong for a summarizer. Two other calls cover that gap. ExtractLoadedPageTextColumns(PageIndex, AText, ColumnGapMultiplier) splits each row wherever the gap between neighboring glyphs exceeds the multiplier times the median advance (3.0 by default) and separates the pieces with a tab. ExtractLoadedPageStructureText(PageIndex, AText, UntaggedGlyphCount) walks the structure tree of a tagged PDF (ISO 32000-1 §14.8.4) and returns the marked-content runs in authoring order; it returns False when the page has no usable structure tree, and UntaggedGlyphCount tells you how many glyphs it could not place
For text that a person or a language model will read, try structure order first and fall back to ExtractLoadedPageText. The structure tree is the one ordering the author declared on purpose, and every geometric method is a guess by comparison. Keep the grid for the jobs where columns are data. Structure-order text extraction has the details of how marked content is matched to the tree
How HotPDF decodes character codes to Unicode
HotPDF Delphi Component resolves each character code through a priority-ordered fallback chain, and a strategy that cannot answer hands the code to the next one. The font's /ToUnicode CMap (ISO 32000-1 §9.10.3) comes first, because it is the only mapping the producer wrote specifically for extraction. After it come glyph names from an /Encoding /Differences array, read through the Adobe Glyph List; then the /Encoding CMap itself, embedded as a stream or named, where a Uni…-UTF16 name means the code already is UTF-16. Composite fonts continue through the Adobe character collections (Adobe-GB1, Adobe-CNS1, Adobe-Japan1, Adobe-Korea1, Adobe-KR) from CID to Unicode, then through /CIDToGIDMap into the cmap table of the embedded font program, read in reverse from glyph to Unicode. Simple fonts fall back to the WinAnsi, MacRoman, Standard and MacExpert tables and finally to that same embedded cmap. A code that exhausts the chain resolves to 0
The embedded-cmap step is what rescues many subset fonts that ship without /ToUnicode: the font program usually still contains its own Unicode cmap, and reversing it recovers most of the text. It is a reconstruction, though. Two Unicode characters that share one glyph will both come back as whichever the cmap lists, so treat recovered text as good for search and indexing and check it before using it as a legal record
The Adobe collection data lives on disk. HotPDF ships the CMap and cid2code.txt files under its resources\CMap folder, parses each file at most once per process and caches the result; the Adobe-GB1 map alone is roughly 2 MB of text you do not want to parse per page. The predefined CMaps also exist as built-in tables, so a font encoded with a named CMap such as UniGB-UTF16-H still decodes without the folder. The CID-to-Unicode tables for Identity-H CJK fonts do need it. The CMapResourcePath property names the folder that holds the collection subfolders; left empty, HotPDF looks for resources\CMap next to the executable and one or two levels above it
Pdf := THotPDF.Create(nil);
try
// Folder that contains Adobe-GB1-6, Adobe-CNS1-7, Adobe-Japan1-7 ...
Pdf.CMapResourcePath := 'C:\MyService\resources\CMap';
Pdf.LoadFromFile(FileName);
...
finally
Pdf.Free;
end;
Set the path explicitly in any service, ISAPI module or plugin DLL. In those hosts ParamStr(0) names the host process, which is rarely the folder you deployed to, and a missing folder does not raise; CJK pages simply come back with gaps where the collection tables would have filled in. Complex script text shaping with HotPDF meets the same code-versus-glyph split at writing time
Two CMap syntax traps
CMap files look trivially parseable, and two details trip up most first attempts. The record count comes before the section keyword: a section reads 2 beginbfchar, never beginbfchar 2. A parser that expects the count after the keyword swallows the number as a stray token and then finds zero entries in every section. HotPDF's reader ignores the count entirely and loops until the matching endbfchar or endbfrange, which also copes with real-world files whose declared counts are wrong
The second trap is that bfchar and bfrange targets are UTF-16BE strings. The destination <D83DDE00> means U+1F600, a surrogate pair that has to be recombined into one code point, and reading those four bytes as a big-endian integer yields a meaningless value for every character outside the Basic Multilingual Plane, emoji included. HotPDF parses the hex literal to raw bytes first and then recombines UTF-16BE code units, which also handles the multi-character targets that ligature mappings produce, such as one fi glyph mapping to two letters
Counting unresolved glyphs with ExtractLoadedPageGlyphs
ExtractLoadedPageGlyphs(PageIndex, AGlyphs) exposes the glyph records underneath the text calls as a THPDFGlyphArray. Each THPDFGlyphRecord carries the raw Code, its CodeByteWidth (1, 2 or 4, decided by the CMap's codespacerange), the resolved Unicode code point plus a UnicodeSequence for ligatures, the active FontKey and FontSize, the user-space X and Y origin, the Advance, a four-corner glyph quad, and GlyphEndX / GlyphEndY, the end of the glyph's own width that the word-space rule measures from. That is enough for positioned highlighting or a layout algorithm of your own
var
Glyphs: THPDFGlyphArray;
I, Unresolved: Integer;
begin
if Pdf.ExtractLoadedPageGlyphs(0, Glyphs) then
begin
Unresolved := 0;
for I := 0 to High(Glyphs) do
if Glyphs[I].Unicode = 0 then
Inc(Unresolved);
if Unresolved > 0 then
ShowMessageFmt('%d of %d glyphs have no Unicode mapping',
[Unresolved, Length(Glyphs)]);
end;
end;
Counting Unicode = 0 records is the honest way to judge a document before trusting its text downstream. Expect the count to differ slightly from what the text calls see. This call returns every glyph in the page's own content stream, including ones outside the crop box and duplicates painted for fake bold, and does not descend into Form XObjects. Each record also stores TokenIndex, ItemIndex and ByteOffset, which tie the character back to its source operand; search and replace in an existing PDF is built on those three fields
Some files have no text to give. A scanned page is one large image with no text operators, so extraction correctly returns an empty string, and the route forward is OCR, starting with extracting the page images from the loaded PDF. Encrypted documents extract normally once they are opened with LoadFromFile(FileName, Password), because the streams are decrypted before the interpreter reads them. Font and content streams are decoded through the same filter chain as the rest of the loaded document, and a /ToUnicode stream that cannot be decoded drops to the next strategy while the rest of the page still extracts. The same object reader backs editing metadata on loaded documents, so one intake pass can read text and fix metadata
Reading-flow, grid, column and structure-order extraction, plus glyph-level access, all ship in the HotPDF Delphi Component for Delphi and C++Builder as plain Object Pascal with no external DLLs or OS text services, so when an odd file lands in your queue you can step into the decoder and see which strategy answered for each code