JBIG2 Random-Access Files: Decoding Them in Delphi

PDFlibPas version 3.539.23 decodes standalone JBIG2 files that use the random-access organisation from ITU-T T.88 Annex D.2, where every segment header comes first and the segment data follows in the same order. The native Pascal decoder in PDFlibJBIG2.pas indexes the header offsets up to the mandatory end-of-file header, checks that segment numbers increase and that the declared data lengths add up to exactly the bytes that remain, and then decodes each body in header order without copying or reordering the compressed data. Before this release the same file raised a flat "random-access organisation is not supported" error the moment the header flags were read

Random-access JBIG2 files are rare, which is exactly why they hurt when they do show up. They come out of archival pipelines and document-imaging systems that want a reader to see every segment header, and therefore every page and dictionary dependency, before touching a single compressed byte. A Delphi application that batch-converts scanned archives into PDF usually meets one of these in the middle of a job, after hundreds of sequential files went through cleanly, and a decoder that stops dead on a well-formed file is only marginally better than one that renders garbage. The same release line had just finished teaching the decoder JBIG2 custom Huffman tables and canonical prefix codes, so random access was the last organisation gap left in the decoder's documented capability limits

What is the JBIG2 random-access organisation?

The random-access organisation is one of three ways T.88 Annex D allows the same segments to be laid out: sequential (D.1) interleaves each header with its data, random-access (D.2) puts all headers first and all data afterwards, and embedded (D.3) is the headerless form used inside other containers such as PDF. A standalone .jb2 file starts with the eight-byte identifier 97 4A 42 32 0D 0A 1A 0A, followed by one flags byte and, when the page count is known, a four-byte page count. Bit 0 of the flags byte selects the organisation, with 1 meaning sequential and 0 meaning random-access; bit 1 set means the number of pages is unknown and the four-byte count is absent. PDFlibPas reads these in checkHeader and setFileHeaderFlags, and the reserved bits 2 to 7 are tolerated rather than rejected

// TJBIG2StreamDecoder.setFileHeaderFlags, PDFlibJBIG2.pas
headerFlags := reader.readByte;
fileOrganisation := headerFlags and 1;          // 0 = random-access (D.2)
randomAccessOrganisation := fileOrganisation = 0;
pagesKnown := headerFlags and 2;                // 1 = page count omitted
noOfPagesKnown := pagesKnown = 0;

// TJBIG2StreamDecoder.decodeJBIG2
validFile := checkHeader;                       // 97 4A 42 32 0D 0A 1A 0A
if not validFile then
begin
  // PDF stream: no file header, embedded organisation, one page
  noOfPagesKnown := True;
  randomAccessOrganisation := False;
  noOfPages := 1;
end
else
begin
  setFileHeaderFlags;
  if noOfPagesKnown then
    noOfPages := getNoOfPages;
end;

PDF itself never carries this layout. A JBIG2Decode image stream, described in ISO 32000-1 §7.4.7, holds only the page segments in embedded organisation, with shared symbol dictionaries moved into a separate JBIG2Globals stream and no file header, end-of-page or end-of-file segments. When decodeJBIG2 fails to find the eight-byte identifier it assumes exactly that and forces sequential, single-page decoding. The native JBIG2 image export goes the other way and wraps the PDF segments in a standalone file whose flags byte is $03, sequential with an unknown page count, followed by an appended end-of-file header. So the random-access work touches only one path: standalone files handed to TPLJBIG2Decoder directly, typically before they are converted or recompressed for PDF, the job that the JBIG2 encoder backends in PDFlibPas handle on the output side

Why can a random-access file not be read in file order?

A random-access file cannot be read in file order because nothing in the byte stream marks where the header block stops and the data block begins, except the end-of-file segment header itself. JBIG2 segment headers have variable length: the referred-to segment count can be a three-bit short form or a long form with a retention bitmap, referred-to segment numbers take one, two or four bytes depending on the segment's own number, and the page association field is one or four bytes. A naive sequential reader parses the first header, reads its data length, and then treats the first bytes of the second header as that segment's data. The decoder cannot tell it has gone wrong until much later, which is why the old code refused the organisation outright instead of attempting it

How does PDFlibPas index random-access segment headers?

PDFlibPas indexes random-access headers in one pre-scan, IndexRandomHeaders, that parses every header, records only its byte offset, and stops at the first end-of-file header (segment type 51). Each header is parsed fully and discarded, so the index is an array of integers rather than a list of objects, and the pre-scan accumulates the declared data lengths as it goes. When the scan finishes, the reader sits on the first byte of the first segment's data, and that position becomes NextBodyOffset

// IndexRandomHeaders, local to TJBIG2StreamDecoder.readSegments
while not reader.isFinished do
begin
  Offset := reader.bytePointer;
  Header := TSegmentHeader.Create;
  try
    readSegmentHeader(Header);
    if reader.BufferOverrun then
      raise EJBIG2DecodeError.CreateFmt(
        'JBIG2 truncated random-access header at byte %d', [Offset]);
    if (HeaderCount > 0) and
       (Cardinal(Header.getSegmentNumber) <= Cardinal(PreviousNumber)) then
      raise EJBIG2DecodeError.Create('JBIG2 random-access segment numbers must increase');
    PreviousNumber := Header.getSegmentNumber;
    Count := Header.getSegmentDataLength;
    if Count < 0 then
      raise EJBIG2DecodeError.Create('JBIG2 unknown or oversized segment length is not supported');
    Inc(TotalLength, Count);                   // Int64 accumulator
    HeaderOffsets[HeaderCount] := Offset;      // grown in chunks
    Inc(HeaderCount);
    if Header.getSegmentType = JBIG2_END_OF_FILE then
    begin
      if Count <> 0 then
        raise EJBIG2DecodeError.Create('JBIG2 invalid end segment length');
      FoundEnd := True;
      Break;
    end;
  finally
    Header.Free;
  end;
end;
if not FoundEnd then
  raise EJBIG2DecodeError.Create('JBIG2 random-access file is missing its end-of-file header');
if TotalLength > Length(reader.Data) - reader.bytePointer then
  raise EJBIG2DecodeError.Create('JBIG2 truncated random-access segment data');
if TotalLength < Length(reader.Data) - reader.bytePointer then
  raise EJBIG2DecodeError.Create('JBIG2 trailing random-access data');
NextBodyOffset := reader.bytePointer;

Every check in that loop exists because a random-access file has less redundancy than a sequential one. Segment numbers must strictly increase, compared as unsigned values, because two headers claiming the same number make it ambiguous which body a later region's referred-to list means. The data-length field is read by handleSegmentDataLength, which maps any value with the top bit set, including the 0xFFFFFFFF "unknown length" marker, to -1; in random-access layout there is no other way to find where the next body starts, so PDFlibPas rejects that length immediately instead of scanning for an end marker. The total must match the remaining bytes exactly in both directions, and a single extra byte after the last body fails with "trailing random-access data". That strictness is deliberate: in this layout a length mismatch means every body after the error point is shifted, and a decoder that shrugs off one stray byte has no way to know whether it is harmless padding or the first symptom of misaligned data

Why did the last end-of-page segment go missing?

The last end-of-page segment went missing because the first version of the decode loop kept the sequential termination test, while not reader.isFinished, and in random-access layout the data stream runs out before the header index does. End-of-page (type 49) and end-of-file segments carry zero bytes of data, and they are normally the last headers in the file. After the final region body is consumed the reader sits exactly at the end of the buffer, so the loop exits and those zero-length segments are never dispatched, leaving the page unfinished. The fix makes the random-access loop count headers instead of bytes. Each iteration jumps the reader to the next indexed header, resets bitPointer to 7 because the previous body may have ended mid-byte, re-parses that header, then moves bytePointer to NextBodyOffset and advances it past the body. The existing segment handlers, referred-to segment checks and the Context diagnostics run unchanged, and an error message still reports the header's original byte offset, not the body position

// TJBIG2StreamDecoder.readSegments, main loop
if randomAccessOrganisation then
  IndexRandomHeaders;
while (randomAccessOrganisation and (HeaderIndex < HeaderCount)) or
      ((not randomAccessOrganisation) and (not reader.isFinished)) do
begin
  if randomAccessOrganisation then
  begin
    reader.bytePointer := HeaderOffsets[HeaderIndex];
    reader.bitPointer := 7;                    // realign after a partial byte
    Inc(HeaderIndex);
  end;
  SegmentOffset := reader.bytePointer;         // used in error context
  readSegmentHeader(segmentHeader);
  if randomAccessOrganisation then
    reader.bytePointer := NextBodyOffset;      // jump to this segment's data
  DataLength := segmentHeader.getSegmentDataLength;
  DataEnd := reader.bytePointer + DataLength;
  NextBodyOffset := DataEnd;
  // ... dispatch to the existing segment handler, then seek to DataEnd
end;

What does the random-access validation actually prove?

The validation proves that the reorganised bytes decode to the same pixels as their sequential originals, and it proves that malformed random-access input fails cleanly; it does not prove coverage of random-access files from arbitrary encoders. The shared Pascal regression uses a 235-byte synthetic file built on a custom-table fixture that must decode to a 7 by 1 row of black pixels, both with a known page count and with the count field removed, and then feeds the decoder every truncated prefix of that file, a duplicate segment number, one trailing byte and an unknown data length, asserting each time that LoadFromByteArray returns False and leaves Width and Height at zero. The real-image case is a 500 by 473 custom-table refinement image whose segments were reorganised into random-access layout with every original header and compressed byte preserved; its SHA-256 matches the reviewed sequential baseline exactly. That file is a derivative produced by an organisation transform, not a natural random-access document found in the wild, and no such natural sample was available. The suites passed at 1,598 tests for Delphi Win32, 42 for the Delphi Win64 image suite, 48 for FPC Win32 and 46 for FPC Win64, alongside the three existing sequential pixel cases

Loading a random-access .jb2 file and its limits

Application code does not change: TPLJBIG2Decoder.LoadFromByteArray detects the file header and organisation on its own, returns False on any rejected input with the reason in LastError, and exposes the decoded page through Width, Height and GetScanline, which returns one byte per pixel

uses
  SysUtils, Classes, PDFlibJBIG2;

function ReadJb2(const FileName: string): TJBIG2ByteArray;
var
  FS: TFileStream;
begin
  FS := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  try
    SetLength(Result, FS.Size);
    if Length(Result) > 0 then
      FS.ReadBuffer(Result[0], Length(Result));
  finally
    FS.Free;
  end;
end;

function CountBlackPixels(const FileName: string): Integer;
var
  Decoder: TPLJBIG2Decoder;
  Row: TJBIG2ByteArray;
  X, Y: Integer;
begin
  Result := 0;
  Decoder := TPLJBIG2Decoder.Create;
  try
    // Sequential and random-access standalone files take the same call
    if not Decoder.LoadFromByteArray(ReadJb2(FileName)) then
      raise Exception.Create('JBIG2 rejected: ' + Decoder.LastError);
    for Y := 0 to Decoder.Height - 1 do
      if Decoder.GetScanline(Y, Row) then
        for X := 0 to Decoder.Width - 1 do
          if Row[X] = 1 then
            Inc(Result);
  finally
    Decoder.Free;
  end;
end;

The boundaries are worth stating plainly. Random-access support is a file organisation feature, not a random page API: TPLJBIG2Decoder still returns the first page's bitmap, and there is no call to pick page 7 out of a 40-page file or to decode pages lazily. Segments with unknown data length are rejected in random-access files, and the existing limits on custom Huffman prefix lengths and table-entry counts are unchanged. Those limits are narrow enough that a Delphi application can route the rejected cases elsewhere by LastError, and the rest of the image pipeline, from PDF image extraction to JBIG2 encoding, is covered on the PDFlibPas Delphi PDF library product page