PDFlibPas version 3.539.22 decodes JBIG2 custom Huffman tables natively: the pure Pascal decoder in PDFlibJBIG2.pas parses the Tables segment (type 53), assigns canonical prefix codes in table-line order as ITU-T T.88 Annex B.3 requires, consumes custom table references in selector order for symbol dictionaries and text regions, and bounds every read by the declared segment length rather than by whatever bytes happen to follow
The file that forced this work was unremarkable on the surface. A scanned contract, JBIG2-compressed with Huffman symbol coding rather than the far more common arithmetic coding, and with the encoder shipping its own code tables instead of the standard tables B.1 through B.15. Two independent decoders disagreed on its refinement pixels, and the PDFlibPas decoder of the time produced text that looked like it had been through a shredder: glyph fragments shifted by a few pixels, one column of every character missing. Nothing raised an error. That is the shape of bug that survives for years, because a decoder that rejects a file gets a support ticket, while a decoder that renders it slightly wrong gets a customer who assumes the scan was bad
What does a JBIG2 Tables segment actually contain?
A Tables segment is a compact description of one Huffman table: one flags byte, two signed 32-bit bounds, and then a run of (prefix length, range length) pairs that partition the interval between the bounds, as laid out in T.88 §7.4.13 and Annex B.2. Bit 0 of the flags byte is HTOOB and says whether the table has an out-of-band code. Bits 1 to 3 plus one give HTPS, the number of bits used to write each prefix length; bits 4 to 6 plus one give HTRS, the width of each range length field. Bit 7 is reserved, and PDFlibPas refuses the segment if it is set rather than guessing what a future revision meant by it. HTLOW and HTHIGH follow as signed 32-bit integers, which is the first place a decoder can go wrong: reading them as unsigned makes a table whose low bound is negative, which is perfectly normal for delta-coded symbol widths, look like it starts at four billion. Every field goes through a local ReadField helper that checks the request against the bit position where the segment data ends before touching the reader, because a table that reads past its segment would be consuming the next segment header as prefix lengths
// TCodeTableSegment.readSegment, PDFlibJBIG2.pas
EndBit := (Int64(decoder.reader.bytePointer) +
segmentHeader.getSegmentDataLength) * 8;
Flags := ReadField(8);
if (Flags and $80) <> 0 then
raise EJBIG2DecodeError.Create('reserved custom Huffman table flag');
PrefixBits := ((Flags shr 1) and 7) + 1; // HTPS
RangeBits := ((Flags shr 4) and 7) + 1; // HTRS
LowValue := Integer(ReadField(32)); // signed HTLOW
HighValue := Integer(ReadField(32)); // signed HTHIGH
if LowValue >= HighValue then
raise EJBIG2DecodeError.Create('invalid custom Huffman range bounds');
CurrentValue := LowValue;
while CurrentValue < HighValue do
begin
PrefixLength := ReadField(PrefixBits);
RangeLength := ReadField(RangeBits);
if RangeLength > 32 then
raise EJBIG2DecodeError.Create('invalid custom Huffman range length');
AddLine(CurrentValue, PrefixLength, RangeLength);
Inc(CurrentValue, Int64(1) shl RangeLength);
end;
AddLine(LowValue - 1, ReadField(PrefixBits), jbig2HuffmanLOW);
AddLine(HighValue, ReadField(PrefixBits), 32);
if (Flags and 1) <> 0 then
AddLine(0, ReadField(PrefixBits), jbig2HuffmanOOB);The two lines appended after the loop are the escape lines from Annex B.2: the lower range line starts at HTLOW minus one and counts downward, the upper range line starts at HTHIGH with a fixed 32-bit range, and the optional OOB line has no value at all. PDFlibPas marks them with the sentinel range lengths jbig2HuffmanLOW ($FFFFFFFD) and jbig2HuffmanOOB ($FFFFFFFE), the same convention its fifteen built-in standard tables use, so the decoding loop does not care whether a table came from the specification or from the file
Why must prefix codes be assigned in table-line order?
Because the encoder never writes the codes. A JBIG2 Tables segment carries only prefix lengths, and both sides reconstruct the actual bit patterns with the canonical procedure in Annex B.3: count how many lines have each length, assign codes of length one first, then shift left and continue, and within one length hand out codes in the order the lines appear. Any deviation from that order silently produces a different table. The decoder will not notice, because every bit pattern it generates is still a valid prefix code, just not the one the encoder used, and the output is a plausible-looking bitmap assembled from the wrong symbols
// THuffmanDecoder.buildTable, PDFlibJBIG2.pas
FillChar(Counts, SizeOf(Counts), 0);
for I := 0 to length - 1 do
begin
if table[I].prefixLen > 32 then
raise EJBIG2DecodeError.Create(
'Huffman prefixes longer than 32 bits are not supported');
Inc(Counts[table[I].prefixLen]);
end;
Active := 0;
Code := 0;
for Bits := 1 to 32 do
begin
Starts[Bits] := Active;
Positions[Bits] := Active;
Inc(Active, Counts[Bits]);
if Code + UInt64(Counts[Bits]) > (UInt64(1) shl Bits) then
raise EJBIG2DecodeError.Create('oversubscribed Huffman prefix codes');
Code := (Code + UInt64(Counts[Bits])) shl 1;
end;
SetLength(Result, Active + 1);
for I := 0 to length - 1 do // stable: declaration order kept
if table[I].prefixLen > 0 then // within each prefix length
begin
Result[Positions[table[I].prefixLen]] := table[I];
Inc(Positions[table[I].prefixLen]);
end;
Code := 0;
for Bits := 1 to 32 do
begin
for I := Starts[Bits] to Positions[Bits] - 1 do
begin
Result[I].prefix := Cardinal(Code);
Inc(Code);
end;
Code := Code shl 1;
end;
Result[Active].rangeLen := jbig2HuffmanEOT;THuffmanDecoder.buildTable is a counting sort rather than a comparison sort for one reason: a counting pass over Counts, Starts and Positions is stable by construction, so lines of equal prefix length land in the result in the order they were declared, which is precisely the ordering Annex B.3 assigns codes by. Lines with prefix length zero are dropped before code assignment, because B.3 defines them as unused rather than as one-bit codes. Two guards sit in the same loop. The oversubscription check catches a table whose lengths claim more codes than a prefix code of that depth can hold, which is the Kraft inequality expressed as an integer comparison; without it, a hostile table produces a code that matches two lines and the decoder picks whichever it scans first. The 32-bit ceiling exists because prefix is a Cardinal and the matcher in decodeInt accumulates bits into one. T.88 allows longer prefixes on paper, PDFlibPas rejects them by name, and no real encoder has been seen emitting one. The value arithmetic needs the same care as the code arithmetic: THuffmanTable.val is an Int64, and the lower range line is decoded as val - readBits(32), a 32-bit unsigned offset subtracted from HTLOW minus one. With Integer intermediates that subtraction wraps, and the wrapped value is then accepted as a symbol width. The 64-bit path computes the true value, checks it against the signed 32-bit range, and raises if it does not fit, which turns a silent corruption into an explicit refusal
Why did custom tables never fire before 3.539.22?
Two defects hid each other. The first was a one-line setter bug: TTextRegionHuffmanFlags.setFlags received its argument under the same name as the field it stored into, so Self.flagsAsInt := flagsAsInt assigned the uninitialised field to itself and every selector read back as zero, which sent text regions asking for custom tables through standard tables F, H and K instead. The second defect meant that fixing the first alone would still have produced corrupted symbols. When a Huffman symbol dictionary stores its symbols as an uncompressed collective bitmap, the last byte of each row is partial, and the old copy loop treated padding, which holds the number of valid bits, as the position of the lowest valid bit; a 63-pixel-wide row copied one bit from its final byte instead of seven. The corrected loop runs for bitPointer := 7 downto ((8 - padding) and 7), and synthetic fixtures at 7-bit and 9-bit widths pin both sides of the byte boundary. With the selectors reading correctly, tables are handed out in the order the specification lists them, which T.88 §7.4.3.1.2 fixes for text regions as FS, DS, DT, RDW, RDH, RDX, RDY and RSIZE and §7.4.2.1.1 fixes for symbol dictionaries as DH, DW, BMSIZE and AGGINST. Each two-bit selector means standard table 0 or 1, reserved for 2 on fields with only two standard tables, and custom for 3, and every custom selection consumes the next Tables segment among the referred-to segments in referral order. NextCustomHuffmanTable does exactly that walk and raises missing custom Huffman table reference when a region refers to fewer tables than its selectors demand. One more line belongs to the same fix: a Huffman symbol dictionary whose input and new symbols add up to one computes a symbol code length of zero from the log2 formula, while the Huffman variant of the format writes every symbol ID with at least one bit, so if sdHuffman and (symbolCodeLength = 0) then symbolCodeLength := 1 in TSymbolDictionarySegment keeps the refinement and aggregate path from reading zero bits per symbol ID
What does the segment boundary guarantee?
PDFlibPas treats the segment data length in every header as a contract that both directions must honour: a segment may not read past its declared end, and it may not finish short and leave the next header at an unpredictable offset. The rules that fall out of that contract are individually small. A data length with bit 31 set is the unknown-length marker of T.88 §7.2.7, and handleSegmentDataLength maps it to a negative value that readSegments rejects outright rather than scanning ahead for a terminator. Every referred-to segment number must be smaller than the current segment number and must already exist, so a forward or dangling reference fails before any region tries to resolve it. END_OF_PAGE and END_OF_FILE must declare zero bytes of data. A Profiles segment (type 52) carries a 32-bit count followed by that many 32-bit identifiers and no pixels at all, so it is checked as 4 plus 4 times the count against the declared length, skipped, and kept in the segment list only so that later segments can still refer to it by number. An unknown profile identifier is not an unknown encoding, and treating it as one would reject files that decode perfectly well
// TJBIG2StreamDecoder.readSegments, PDFlibJBIG2.pas
DataLength := segmentHeader.getSegmentDataLength;
if DataLength < 0 then
raise EJBIG2DecodeError.Create(Context +
'unknown or oversized segment length is not supported');
if DataLength > Length(reader.Data) - reader.bytePointer then
raise EJBIG2DecodeError.Create(Context + 'truncated segment data');
DataEnd := reader.bytePointer + DataLength;
for I := 0 to noOfReferredToSegments - 1 do
if (referredToSegments[I] >= segmentHeader.getSegmentNumber) or
(findSegment(referredToSegments[I]) = nil) then
raise EJBIG2DecodeError.Create(Context + 'invalid segment reference');
// ... create the segment object for this type ...
reader.SegmentEnd := DataEnd;
segment.readSegment;
if reader.bytePointer > DataEnd then
raise EJBIG2DecodeError.Create(Context +
'decoded data exceeds declared segment length');
if reader.bytePointer < DataEnd then
begin
reader.bytePointer := DataEnd; // MMR may leave the EOFB unread
reader.bitPointer := 7;
end;The tail of that loop is where an earlier version of the decoder went wrong on MMR-coded regions. An MMR decoder knows it is finished when the last pixel of the last row is produced, which can happen before it has consumed the EOFB terminator that T.88 §6.2.5.7 places at the end of the data. The old code assumed the reader was positioned at the next header, so the leftover terminator bytes were parsed as a segment number and the stream failed a few bytes later with a misleading error. Now the declared end wins: reading past it is an error, stopping short is normal, and the reader is moved to DataEnd with the bit pointer reset so the next header is read from where the file said it would be. The same discipline shows up wherever PDFlibPas parses untrusted PDF structures: the declared length is the boundary, and the decoder does not go looking for a friendlier one
Where does Huffman refinement read its bitmap size?
Before the arithmetic decoder starts, and from a field that only exists in Huffman mode. When a text region instance carries refinement (RI is non-zero) and SBHUFF is set, T.88 §6.4.11 has the decoder read RDW, RDH, RDX and RDY with their selected tables, then BMSIZE with the RSIZE table, then align to a byte boundary, and only then run the generic refinement decoding over exactly BMSIZE bytes. Arithmetic-mode text regions have no such field, and a decoder that shares one code path for both modes will skip it, start the arithmetic decoder two or more bytes early, and refine every symbol against garbage. The symbol dictionary path with REFAGG and a single refinement instance, described in §6.5.8.2.2, has the same BMSIZE field with the same consequences. In PDFlibPas the upper bound on that size is TStreamReader.SegmentEnd, the end of the current segment as set by readSegments, not the end of the whole stream, because a BMSIZE that can only be satisfied by borrowing bytes from the following segment is malformed and validating it against the stream length would let the arithmetic decoder read into the next header. The lower bound of two bytes reflects the initial byte pair the arithmetic decoder always consumes, and after refinement the reader jumps to RefinementEnd regardless of how far the arithmetic decoder read ahead, since its final position is not the position of the next Huffman-coded field
// TJBIG2Bitmap text region decoding, Huffman refinement path
RefinementSize := huffmanDecoder.decodeInt(huffmanRSizeTable).intResult;
huffmanDecoder.consumeRemainingBits;
if (RefinementSize < 2) or
(RefinementSize > huffmanDecoder.reader.SegmentEnd -
huffmanDecoder.reader.bytePointer) then
raise EJBIG2DecodeError.Create('invalid refinement bitmap size');
RefinementEnd := huffmanDecoder.reader.bytePointer + RefinementSize;
arithmeticDecoder.start;
// ... readGenericRefinementRegion ...
if huffmanDecoder.reader.bytePointer > RefinementEnd then
raise EJBIG2DecodeError.Create('refinement data exceeds declared size');
huffmanDecoder.reader.bytePointer := RefinementEnd;
huffmanDecoder.reader.bitPointer := 7;What was verified, and what still gets refused
The sample that started this, a 500 by 473 pixel JBIG2 image with custom tables and Huffman refinement, now decodes to a bitmap with zero differing pixels against an independent decoder, and the synthetic 7-bit and 9-bit collective bitmap fixtures produce the expected rows on both. The two independent decoders that disagreed on the original sample still disagree with each other; PDFlibPas matches one of them, and the honest statement is that the native output agrees with one independent implementation and with the specification as read, not that every decoder in the world agrees. The malformed side of the suite covers:
- a reserved flag bit or a reserved selector value
- a table truncated in the middle of a line
- oversubscribed prefix lengths and prefixes longer than 32 bits
- a region whose selectors ask for more custom tables than it refers to
- confirmation that stale output is cleared after a failed decode rather than left in place for the caller to mistake for a result
Three limits remain deliberate. Random-access stream organisation, where all segment headers precede all segment data, raises JBIG2 random-access organisation is not supported as soon as the file header flags are read, because no representative sample exists to validate it against and a half-implemented path is worse than a named refusal. Custom tables are capped at 65,536 lines and 32-bit prefixes. And the public decoding entry, TPLJBIG2Decoder.LoadFromByteArray, returns the first page bitmap in stream order through getPageAsJBIG2Bitmap(0), the first page-information segment encountered, rather than looking up page association zero; embedded PDF streams routinely number their single page 1, and asking for page 0 by association would find nothing. The failure text lands in TPLJBIG2Decoder.LastError, the internal decoder diagnostic that carries the segment number, type and byte offset of the fault, and is not the same thing as the library-level TPDFlib.LastErrorCode. None of this touches the encoding side, which is covered in the notes on JBIG2 encoder backends and how they are linked; the read path has to accept whatever somebody else's encoder decided to emit, and it shares its rules with the rest of the image stack, including the built-in TIFF decoder and its BigTIFF and tiled-layout refusals: refuse by name, never borrow bytes across a declared boundary, and keep the arithmetic wide enough that a wrapped intermediate cannot pass for a valid answer. If you are evaluating a native JBIG2 read path for Delphi or C++Builder, the decoder and the rest of the image handling are documented on the PDF Library for Delphi page