A two-column conference paper sorted by vertical position comes back with a first line like Abstract We measure the 2 Related work Earlier studies: the opening words of the left column and the opening words of the right column, glued together because both sit on the same baseline. Nothing went wrong in the parser. A PDF page offers two defensible orders for its text, the sequence in which the content stream paints it and the position of every glyph on the page, and for many layouts those two orders disagree. An extractor that returns positioned blocks instead of one string lets you see which order you got and switch when it is the wrong one for the page in front of you
What GetStructuredText returns
TPdf.GetStructuredText in PDFium Component returns a TPdfStructuredTextPage for the current page: an array of blocks, each block an array of lines, each line an array of styled spans. Every level carries its own Text and a page-space Bounds rectangle in PDF points, so a heading, a single line or a bold phrase can be located on the page without a second lookup. The page record adds Width, Height, CharacterCount and two quality counters described further down
The flattened Page.Text is built from the blocks rather than from the raw character stream. Lines inside a block are joined with a single CRLF and blocks are separated by an empty line, which means a plain-text dump of Page.Text already shows where the extractor drew its block boundaries. That makes it the quickest thing to print when a page reads strangely
All of this is driven by a small options record. TPdfStructuredTextOptions.Default sets ReadingOrder to roPhysicalLayout, turns on IncludeFontInfo and IncludeSemantics, and sets MaxCharacters to 1,000,000. The reading order field is the one that changes the shape of the output most
How the two reading orders build lines and blocks
roContentOrder keeps the producer's drawing sequence, while roPhysicalLayout rebuilds order from glyph positions, and the difference shows up first in how lines are formed. In content order, characters are taken in the order the page draws them. A new line starts at an explicit line break or when the next character no longer overlaps the current line vertically. A line then joins the block directly before it only if the two overlap horizontally by at least a fifth of the narrower width and the vertical gap is no more than about two and a quarter line heights; otherwise it opens a new block. Blocks stay in drawing order
Physical layout throws the drawing sequence away. Every character with a known box is sorted by the vertical centre of that box, top of the page first, then by its left edge. Characters whose vertical extents overlap enough share a line, and that test looks only at height. Horizontal distance plays no part in it. Each line then joins whichever existing block anywhere on the page fits it best by the same overlap-and-gap rule, and the finished blocks are sorted by their top edge, with the left edge breaking ties within half a point
That second rule is where the opening example comes from. When the two columns of a paper share a baseline grid, the first line of the left column and the first line of the right column overlap vertically, so physical layout puts them on one line. The gap at the gutter is wider than a quarter of the font size, so the extractor inserts one space and keeps going. Every line of the body does the same, the merged lines all overlap one another horizontally, and the whole two-column body becomes a single wide block that reads straight across the gutter
Content order handles that page well whenever the producer wrote column one before column two, which is what word processors and typesetting systems normally do. The left column's lines stack into blocks, the first line of the right column has no horizontal overlap with them and opens a fresh block, and the blocks come out in the order the author wrote them. Physical layout earns its place on a different kind of page: invoices, statements, forms and slides, where a label and its value sit on the same baseline and reading the row left to right is exactly what you want. It is also the mode to use when the drawing sequence is meaningless, for instance text painted font by font or a page assembled from many small overlays
For a mixed collection, split the choice by document type. Articles, reports and manuals from a known authoring tool go through roContentOrder; forms and transactional documents stay on the roPhysicalLayout default. The following routine prints the blocks of one page in content order, with headings marked when the file is tagged
uses
SysUtils, PDFium;
procedure DumpBlocks(Pdf: TPdf; PageNumber: Integer);
var
Options: TPdfStructuredTextOptions;
Page: TPdfStructuredTextPage;
B, L: Integer;
begin
Pdf.PageNumber := PageNumber; // 1-based
Options := TPdfStructuredTextOptions.Default;
Options.ReadingOrder := roContentOrder; // column-by-column producers
Page := Pdf.GetStructuredText(Options);
if Page.Source = rosStructure then
Writeln('block kinds come from the structure tree')
else
Writeln('block kinds are inferred from layout');
for B := 0 to High(Page.Blocks) do
begin
if Page.Blocks[B].Kind = cfHeading then
Writeln(Format('[%d] H%d %s',
[B, Page.Blocks[B].HeadingLevel, Page.Blocks[B].Text]))
else
begin
Writeln(Format('[%d] x=%.0f..%.0f, %d line(s)',
[B, Page.Blocks[B].Bounds.Left, Page.Blocks[B].Bounds.Right,
Length(Page.Blocks[B].Lines)]));
for L := 0 to High(Page.Blocks[B].Lines) do
Writeln(' ', Page.Blocks[B].Lines[L].Text);
end;
end;
end;
How do you detect a line that runs across the gutter?
A merged line betrays itself through its spans. GetStructuredText starts a new span whenever the next character is not the direct successor of the previous one in the page's original character sequence, in addition to the style changes covered below. Inside one column, consecutive words are usually consecutive in the source. At the gutter of a merged line the source index jumps by hundreds of characters, and the horizontal gap between the two spans is far wider than any word space
Those two signals together make a reliable test, and you can run it on the default output before deciding whether a page needs content order instead. The snapshot of character data that GetStructuredText reads is cached per page, eight pages by default, so asking for the same page a second time with a different ReadingOrder regroups data already in memory instead of extracting the page again
// LineCrossesGutter is a helper you add yourself: it flags a line holding two
// neighbouring spans that are far apart on the page and not adjacent in the
// source character sequence
function LineCrossesGutter(const Line: TPdfStructuredTextLine;
MinGap: Double): Boolean;
var
S: Integer;
Prev, Cur: TPdfStructuredTextSpan;
begin
Result := False;
for S := 1 to High(Line.Spans) do
begin
Prev := Line.Spans[S - 1];
Cur := Line.Spans[S];
if (Prev.SourceStartIndex < 0) or (Cur.SourceStartIndex < 0) then
Continue; // alternate-text blocks
if (Cur.Bounds.Left - Prev.Bounds.Right > MinGap) and
(Cur.SourceStartIndex <>
Prev.SourceStartIndex + Prev.SourceCharacterCount) then
Exit(True);
end;
end;
function ExtractWithFallback(Pdf: TPdf): TPdfStructuredTextPage;
var
Options: TPdfStructuredTextOptions;
B, L, Lines, Crossing: Integer;
begin
Options := TPdfStructuredTextOptions.Default; // roPhysicalLayout
Result := Pdf.GetStructuredText(Options);
Lines := 0;
Crossing := 0;
for B := 0 to High(Result.Blocks) do
for L := 0 to High(Result.Blocks[B].Lines) do
begin
Inc(Lines);
if LineCrossesGutter(Result.Blocks[B].Lines[L], 18) then
Inc(Crossing);
end;
// a quarter of the lines crossing a gap looks like columns, not a form row
if (Lines > 0) and (Crossing * 4 >= Lines) then
begin
Options.ReadingOrder := roContentOrder;
Result := Pdf.GetStructuredText(Options);
end;
end;
Eighteen points is a quarter of an inch, comfortably wider than a word space in body text and narrower than most gutters. The one-in-four threshold keeps an invoice with a few label-and-amount rows on physical layout while a genuine two-column body, where nearly every line crosses, switches over. If the producer wrote the page row by row across both columns, the source indices stay contiguous, the test never fires, and neither order will separate the columns for you; that page needs the column mode of the readable-content API, described in reading two columns with a spanning heading
What does IncludeSemantics change, and what does it leave alone?
IncludeSemantics labels blocks; it does not reorder them. With the option on, GetStructuredText reads the page's readable content and gives each block the kind of the fragment it overlaps by the largest area. The available kinds are cfParagraph, cfHeading with a HeadingLevel, cfListItem, cfTableCell, cfCaption, cfFigure, cfLink, cfOther for any other tag, and cfPlain when nothing matched. Each block also picks up a Language and a StructElementIndex that ties it to an element of the structure tree
The Source field says where the label came from: rosStructure when the page has a usable structure tree in the sense of ISO 32000-1 §14.7, rosHeuristic when the kinds were inferred. Page.Source gives the same answer for the page as a whole. A heading reported with rosStructure is a heading because the author tagged it; one reported with rosHeuristic is an educated guess, and an indexing pipeline that weights titles should treat the two differently. Block order, in both cases, still comes from ReadingOrder. The semantic pass uses the readable-content defaults, so a column setting applied through ReadableContentOptions does not reach GetStructuredText
Figures get one extra rule. On a tagged page, a Figure element or alternate text that was not matched to any text block is appended as a block of its own, holding the alternate description as its text. Such a block has a single line and a single span with SourceStartIndex set to -1, because no glyph on the page produced it. Code that maps spans back to characters has to skip that value, which is why the gutter test above checks for it. Structure tree checks themselves are covered in PDF/UA structure tree validation
Spans, bounds and the way back to characters
Each TPdfStructuredTextSpan carries its text, its bounds, FontName, FontSize, FontWeight and Angle, plus SourceStartIndex and SourceCharacterCount. With IncludeFontInfo on, a span ends wherever any of the four style values changes, so a sentence with three bold words in the middle arrives as three spans and emphasis can be rebuilt from FontWeight instead of guessed from font names. With IncludeFontInfo off, the style fields stay empty and spans break only where the source sequence is interrupted
The angle matters on pages with stamps. A diagonal DRAFT watermark lives in the same coordinate space as the body text, and in physical layout its letters can land on body lines that they cross vertically. Spans with a non-zero Angle are easy to drop before you join text for indexing
The two source fields are what turn a search hit into something you can draw. They index the page's character sequence, so a phrase found in a block converts directly into character boxes for a highlight, using the approach in visual text line selection with character boxes. When the reflowed or HTML form of the page is the goal, the same blocks feed the reflow model and its HTML export
A character budget and two quality counters
MaxCharacters rejects a page outright when it holds more characters than allowed. GetStructuredText checks the count before it allocates any working arrays, and an oversized page raises EPdfError with the message Structured text character budget exceeded; a partial page is never returned. The value must be positive. The per-page character snapshot the method reads has its own default ceiling of 1,000,000 characters and 100,000 text objects, reported as Text snapshot budget exceeded, so setting MaxCharacters above one million does not let a larger page through
UnmappedCharacterCount counts characters for which PDFium could produce no Unicode value at all. Those characters appear as U+FFFD in every Text field. The usual cause is a subset font with a custom encoding and no /ToUnicode CMap, the case ISO 32000-1 §9.10.2 describes, where text renders correctly and cannot be extracted reliably. The counter measures missing mappings only: a glyph whose /ToUnicode entry names the wrong code point counts as mapped, so a page of plausible-looking garbage can still report zero
GeometryFailureCount counts characters without a usable box, line breaks excluded. In content order such characters simply follow their neighbours. In physical layout they have no position to sort by, so they are placed after every positioned character and end up attached to the bottom line of the page. A page with a non-zero geometry count and odd fragments on its last line is showing exactly that
// LogRejected and LogQuality are your own logging routines
function ExtractForIndex(Pdf: TPdf; PageNumber: Integer;
out Page: TPdfStructuredTextPage): Boolean;
var
Options: TPdfStructuredTextOptions;
begin
Result := False;
Pdf.PageNumber := PageNumber;
Options := TPdfStructuredTextOptions.Default;
Options.MaxCharacters := 200000; // intake limit for this pipeline
try
Page := Pdf.GetStructuredText(Options);
except
on E: EPdfError do
begin
LogRejected(PageNumber, E.Message); // character or snapshot budget
Exit;
end;
end;
if Page.UnmappedCharacterCount * 20 > Page.CharacterCount then
LogQuality(PageNumber, Format('%d of %d characters have no Unicode value',
[Page.UnmappedCharacterCount, Page.CharacterCount]));
if (Options.ReadingOrder = roPhysicalLayout) and
(Page.GeometryFailureCount > 0) then
LogQuality(PageNumber, Format('%d characters without a box; check the last line',
[Page.GeometryFailureCount]));
Result := True;
end;
For a page you only need as a string, extracting text from PDF documents covers the simpler calls, and reading and writing BDC and MCID marked content goes one layer below the structure tree. The structured page model, the readable-content reader and the reflow export all ship in the Delphi and Lazarus component described on the PDFium Component product page