The content-stream fragment [(Reference)-333(Manual)] TJ paints two words with a clear gap between them, yet it contains no space character at all. The gap is pen movement: ISO 32000-1 §9.4.3 tells the reader to shift the next glyph right by 333 thousandths of the font size before drawing Manual. An extractor that copies characters and ignores geometry returns ReferenceManual, and a search index fed from it will never match the phrase “Reference Manual”. TeX output, many Distiller files and most justified layouts separate words this way
Line breaks go wrong from the other side. A raised superscript splits its line in two, a diagonal DRAFT watermark arrives one character per line, and a trimmed proof leads with the printer’s slug line that no viewer ever displays. None of these files is broken. Each one uses a legal way of placing text that a character-copying extractor misreads
Word spaces rebuilt from glyph geometry
THotPDF.ExtractLoadedPageText in HotPDF Delphi Component decides word breaks from glyph positions, and a space character in the stream is only one of the signals it reads. Along the previous glyph’s writing direction it measures the distance from the end of that glyph’s own width to the origin of the current glyph, and inserts one space when the distance exceeds 0.15 of the current glyph’s box height, measured from ascent to descent in user space. For the TJ fragment above, a third of an em sits comfortably past that line for any ordinary text face, so the page text reads Reference Manual
Two conditions hold the space back. Nothing is added when the text so far already ends in a blank or the next glyph is one (space, tab, no-break space, ideographic space), so producers that do show spaces never get doubled ones. Nothing is added between two CJK characters either. Ideographs, kana, hangul and full-width forms all count, because justification stretches a Chinese or Japanese line apart without that stretch meaning a word boundary. Separately from both, a glyph box with zero height falls back to the Tf size as its height. Every number the rule uses is stored on the glyph records, so a file whose spacing looks wrong can be replayed glyph by glyph
uses
SysUtils, HPDFDoc, HPDFContentStream;
// Prints the gaps that ExtractLoadedPageText would turn into spaces.
// Horizontal text only; ExtractLoadedPageGlyphs returns the page
// content stream without Form XObject text and without the crop filter
procedure DumpWordGaps(Pdf: THotPDF; PageIndex: Integer);
var
Glyphs: THPDFGlyphArray;
I: Integer;
Height, Gap: Double;
begin
if not Pdf.ExtractLoadedPageGlyphs(PageIndex, Glyphs) then
Exit;
for I := 1 to High(Glyphs) do
begin
if Glyphs[I].Vertical or Glyphs[I - 1].Vertical then
Continue;
// ascent-to-descent height of the glyph box, in user space
Height := Sqrt(Sqr(Glyphs[I].QuadX[3] - Glyphs[I].QuadX[0]) +
Sqr(Glyphs[I].QuadY[3] - Glyphs[I].QuadY[0]));
if Height <= 0 then
Height := Abs(Glyphs[I].FontSize);
// gap from the end of the previous glyph's own width
Gap := Glyphs[I].BaselineStartX - Glyphs[I - 1].GlyphEndX;
if (Height > 0) and (Gap > 0.15 * Height) then
Writeln(Format('U+%.4x gap %.2f height %.2f: space',
[Glyphs[I].Unicode, Gap, Height]));
end;
end;
Why measure from the glyph’s own width instead of the pen position?
Because the pen position after a glyph already contains spacing that has nothing to do with word breaks. ISO 32000-1 §9.4.4 defines the horizontal displacement as the glyph width times the font size, plus character spacing Tc, plus word spacing Tw, all scaled by horizontal scaling Tz. Each glyph record carries both ends: BaselineEndX / BaselineEndY hold that full displacement, while GlyphEndX / GlyphEndY hold only the font’s advance at the current size and Tz. The two diverge on files that tighten tracking with a negative Tc and hand the space back through a TJ adjustment after each glyph. Measured from the pen, every hand-back looks like a gap, and a Chinese term such as “95后” comes out as “9 5 后”. Measured from the glyph’s own width, the adjustment only returns what Tc took away, and the term stays intact
The threshold is tied to the glyph box height instead of the Tf size for a related reason. Word exports often write 1 Tf and carry the real size in a scaled Tm, so the font size operand reads 1 while the text is 10 points tall. A rule keyed to that operand would put the threshold at 0.15 point on such a page, and ordinary kerning pairs would start reading as word breaks. The box height is measured after every matrix has been applied, so 1 Tf with a scaled matrix and a plain 10 Tf produce the same text
Two edge cases follow straight from the geometry. A heading set with very loose letter spacing, where Tc alone opens more than 0.15 of the text height between letters, extracts as S P A C E D. That is what the page looks like, and we would rather return it than add a heuristic that guesses which wide gaps are tracking and which are words; a guess of that kind starts swallowing real word breaks in tightly justified columns, which is a worse failure for search. Pieces drawn out of order on one baseline produce a negative gap and join without a space. In running body text that case is rare
When does a new line start in the extracted text?
ExtractLoadedPageText starts a new line when the move from the previous glyph’s origin to the current one, projected onto the normal of the previous writing direction, exceeds half the larger box height of the two glyphs. Movement along the baseline never counts toward a line break, however far the pen jumps
A rule that compares the raw Y move with half of the Tf size fails in two directions. With 1 Tf and a scaled Tm the threshold shrinks to half a unit, so a superscript raised by a text rise of 0.4 breaks the line. So does the small baseline jitter some layout engines leave between runs. The same rule ignores X entirely. Text under a rotated Tm, such as that diagonal watermark, steps down the page with every glyph and comes out one character per line, while the projection treats a 45-degree run exactly like a horizontal one
Taking the larger of the two heights matters when sizes mix on one line. A 36-point display word followed by an 8-point caption a few points lower is judged against half the large glyph’s height, so the pair stays together; judged against the small glyph alone, the caption would land on a line of its own. The text rise is part of the glyph origin, which is why a superscript’s raised baseline is measured against that same half height
Vertical text in writing mode 1 (§9.7.4.3) takes a separate path. Those glyphs are grouped into columns on their origin X, with half the font size as the column tolerance, then read right to left and top to bottom, with a line break at each column change. A page that mixes a vertical Japanese body with a horizontal running header also gets a line break at every switch between the two directions
Which glyphs make it into the page text
ExtractLoadedPageText returns the text a viewer shows. It keeps only glyphs whose box centre falls inside GetLoadedPageVisibleBox, which is the CropBox clipped to the MediaBox (§14.11.2), so slug lines and other printer’s marks set as text outside the trim area drop out. ExtractLoadedPageGlyphs keeps returning every glyph of the page content stream, slug line included, for the jobs that need it. The filter is a geometric test on one point per glyph. Text hidden by a clipping path, painted white or covered by an image still comes out, because deciding real visibility would mean rendering the page
uses
SysUtils, HPDFDoc, HPDFContentStream;
var
Pdf: THotPDF;
Raw, Merged: THPDFGlyphArray;
PageText: UnicodeString;
L, B, R, T: Single;
begin
Pdf := THotPDF.Create(nil);
try
Pdf.LoadFromFile('trimmed-proof.pdf');
if Pdf.GetLoadedPageVisibleBox(0, L, B, R, T) then
Writeln(Format('Visible box: %.1f %.1f %.1f %.1f', [L, B, R, T]));
// every glyph of the page content stream, slug line included
if Pdf.ExtractLoadedPageGlyphs(0, Raw) then
Writeln(Length(Raw), ' glyphs in the page content stream');
// page stream plus Form XObject text, in painting order, before the crop
if Pdf.ExtractLoadedPageGlyphsWithForms(0, Merged) then
Writeln(Length(Merged), ' glyphs including forms');
// what the page shows: forms spliced in, outside-crop glyphs dropped
if Pdf.ExtractLoadedPageText(0, PageText) then
Writeln(PageText);
// the same spacing rules applied to the unfiltered stream, for a diff
Writeln(HPDFAssemblePageText(Raw));
finally
Pdf.Free;
end;
end.
Text painted through Form XObjects is part of the page text. Headers, stamps and watermarks very often live in forms, and an extractor that reads only the page content stream never sees them. HotPDF records each Do together with the CTM in effect, interprets the form at its /Matrix times that CTM (§8.10.1), and splices the form’s glyphs in at the position of the Do, recursing into nested forms. A form without its own /Resources borrows those of the stream that paints it, the older arrangement §7.8.3 describes. Glyphs whose centre falls outside the form’s /BBox are left out, since that box clips the form when it is painted. A form that paints itself, directly or through another form, is not entered a second time, and nesting stops at 12 levels
Form glyphs carry TokenIndex = -1 and ItemIndex = -1. The same reason keeps ExtractLoadedPageGlyphs limited to page-stream glyphs: search, replace and redaction write their changes back through TokenIndex, and a form glyph in that array would point them at bytes in a different stream. When you need the merged list at glyph level, ExtractLoadedPageGlyphsWithForms returns it in painting order, before the visible-box filter runs
One more pass runs before the spacing rules. Overprinted logos and bold faked by painting the same text twice with a small offset would otherwise extract with every letter doubled. When a glyph paints the same character at the same size within a tenth of that size along the baseline, and a fifth across it, of an earlier glyph, only one copy is kept; spaces are never removed this way. The article on fake bold and duplicate glyphs in text extraction walks through that case
Text state across q and Q
Text after a Q decodes wrongly in any extractor that saves only the CTM on q. The text state parameters, namely font, size, Tc, Tw, Tz, TL, rendering mode and rise, belong to the graphics state (§9.3.1), so Q has to restore them along with everything else on the stack (§8.4.2). One pattern exposes the mistake quickly: a two-byte Identity-H font is selected inside q … Q for a single table cell, and the text after the Q shows one-byte WinAnsi strings without a Tf of its own. An interpreter that keeps the inner font reads those bytes in pairs, and the dot leaders of a table of contents turn into a string of unrelated CJK characters. The HotPDF interpreter pushes all eight text state values together with the CTM on every q and pops them on every Q
Everything above applies on every page, so a whole document can go to a UTF-8 file in one call. ExtractLoadedPagesTextToStream takes a one-based page range (an empty string means every page), a separator written between pages, and a flag for the byte order mark, and it returns the number of pages extracted. Output is appended at the stream’s current position, so nothing larger than one page’s text is held in memory
uses
SysUtils, Classes, HPDFDoc;
var
Pdf: THotPDF;
Output: TFileStream;
Pages: Integer;
begin
Pdf := THotPDF.Create(nil);
try
Pdf.LoadFromFile('report.pdf');
Output := TFileStream.Create('report.txt', fmCreate);
try
// empty range = every page; form feed between pages; UTF-8 BOM
Pages := Pdf.ExtractLoadedPagesTextToStream(Output, '', #12, True);
Writeln(Pages, ' pages extracted');
finally
Output.Free;
end;
finally
Pdf.Free;
end;
end.
Picking the text API for the job
For search and indexing, ExtractLoadedPageText is the right feed: it follows content-stream order and reflects what the page shows. The decoding chain beneath it, from /ToUnicode maps to CID fonts, is covered in extracting text from loaded PDFs with HotPDF. Content-stream order is weakest on tagged documents whose producer wrote the running header last or interleaved two columns; for those, structure-order text extraction walks the structure tree in authoring order instead of inferring it from geometry. Data locked in tables is better served by typed table extraction across page breaks, which returns cells with their values. Code that edits the page belongs on ExtractLoadedPageGlyphs, whose token indexes point at the bytes it will rewrite
The glyph record fields used in these examples, along with the rest of the text extraction API and a trial build, are documented on the HotPDF Delphi PDF Component product page