Technical Article

PDFlibPas JBIG2 Halftone Grids: HSKIP and Negative Offsets

The native JBIG2 halftone region decoder in PDFlibPas indexes the HSKIP skip mask as HSKIP[ng, mg], as ITU-T T.88 §6.6.5.1 defines it, and places grids that reach negative coordinates, through a negative HGX or HGY or through rotation, with a true floor shift. Both details are easy to get wrong without any error being raised: the halftone region simply comes out garbled or displaced, and both mistakes lurk behind test data that happens to be symmetric or non-negative. The second one turns on a property of Delphi and Free Pascal that bites far outside JBIG2: shr on a signed integer is a logical shift, not the arithmetic >> the standard assumes

Halftone regions are the least common JBIG2 region type, so a decoder can process thousands of scanned documents before it meets a screened photograph encoded as one. When it does, the failure is nasty: the file parses, the segment lengths add up, the page has the right size, and the region is garbage

What does a JBIG2 halftone region actually decode?

A JBIG2 halftone region is a grid of small bitmaps picked from a pattern dictionary, and the decoder's real work is computing an index for every grid cell and the pixel position where that cell lands. The pattern dictionary holds HNUMPATS patterns of HPW × HPH pixels. The halftone region segment then describes a grid of HGW columns by HGH rows and a greyscale image of the same size, coded as Gray-coded bitplanes. Each bitplane is decoded with the generic region procedure over an HGW × HGH bitmap, most significant plane first, and the planes together give each cell its pattern index

Cell placement uses fixed-point arithmetic with an 8-bit fraction. The grid origin HGX, HGY is a pair of 32-bit values, and the grid vector HRX, HRY describes the step between neighbouring cells, which allows a rotated grid. For grid row mg and grid column ng, T.88 §6.6.5 computes the pixel position as:

  • x = (HGX + mg × HRY + ng × HRX) >> 8
  • y = (HGY + mg × HRX − ng × HRY) >> 8

The skip mask enters through the optional HENABLESKIP flag. When the flag is set, §6.6.5.1 builds an HGW × HGH bitmap HSKIP and sets HSKIP[ng, mg] to 1 for every cell whose pattern lies entirely outside the region: x + HPW <= 0, x >= HBW, y + HPH <= 0 or y >= HBH. The greyscale bitplanes are then decoded with that mask as the generic region skip bitmap, so the arithmetic decoder neither reads nor updates context for a skipped cell. Decoder and encoder must agree on every bit of HSKIP, or the two arithmetic coders fall out of step

Why does a transposed HSKIP mask only break non-square grids?

A skip mask written with its coordinates swapped only shows up on a non-square grid, because a square grid keeps every swapped coordinate inside the mask. With a (column, row) pixel accessor, a builder that passes (mg, ng), row first, writes the transposed mask, while the greyscale bitplane decoder reads it as (ng, mg). If the pattern placement loop reads the mask back in the builder's swapped order, the two agree, and a review of the placement logic alone passes it. Loop variables named against what they iterate make it worse, such as a col that walks grid rows next to a Row that walks grid columns

Take a 5 × 3 grid of 4 × 4 patterns on a 16 × 8 region. With HRX = 1024 and HRY = 0, grid column 4 lands at x = 16 and grid row 2 at y = 8, both outside the region. The correct mask marks seven cells: the whole of column 4 and the whole of row 2. The swapped writes try to set pixels at row indices 3 and 4 in a mask only three rows high, and a bitmap setter that silently ignores out-of-range writes drops them. What survives is column 2, rows 0 to 2. The decoder therefore skips two cells the encoder coded, and decodes six cells the encoder skipped

PDFlibPas JBIG2 halftone skip masks for a 5 by 3 grid where the correct HSKIP[ng, mg] marks column 4 and row 2 as skipped, while the transposed writes aimed at rows 3 and 4 of a three-row mask were silently dropped and only column 2 survived, desyncing the arithmetic coders
Only a non-square grid exposes a transposed mask, and the resulting coder desync garbles the region instead of raising an error

The arithmetic decoder does not fail when that happens. It decodes extra pixels out of bits that belong to later cells, its contexts read wrong neighbours, and every pattern index after the first disagreement is noise, which is why the symptom is a garbled region instead of a few misplaced cells. On a square grid the same mistake is often invisible: no swapped coordinate leaves the mask, and when the out-of-region cells are symmetric about the diagonal, a grid that overhangs the right and bottom edges by the same number of cells for instance, the transposed mask is bit-for-bit the correct one. HENABLESKIP is also optional, must be 0 when the greyscale image is MMR-coded, and is rarely set by encoders, so the mistake has very few ways to surface. In PDFlibPas the builder writes HSKIP[ng, mg] and the placement loop reads the same order

Why do negative halftone grid offsets fail in three layers?

A halftone grid that starts left of or above its region can go wrong in three separate places in a decoder, and each mistake hides the next one. T.88 allows this geometry deliberately. An encoder that aligns its screen to the page rather than to the region, or uses a rotated grid, naturally produces negative cell corners that the region crops. All three layers have to be right together, because getting any one right on its own only changes the symptom

Layer 1: a signed field read as unsigned

T.88 §7.4.5.1.2 defines HGX and HGY as signed 32-bit values. Read them through the same 32-bit helper used for unsigned fields, one that clamps every negative result to 0, and a grid meant to start at HGX = -900 is quietly moved onto the region origin; in the example below the whole picture comes out two rows too low. The clamp also hides the other two layers: with the origin forced to be non-negative, a negative coordinate can only appear through a rotated grid with HRY > 0, where y = HGY + mg × HRX − ng × HRY drops below zero for later grid columns

Layer 2: shr is not >> 8

T.88 writes >> 8 and means an arithmetic shift, which rounds towards minus infinity. The literal translation is shr 8, and in Delphi and Free Pascal shr on a signed integer is a logical shift: the sign bit is shifted in as a zero. For an Integer holding -512, shr 8 yields 16777214 instead of -2. A pattern that should be drawn at y = -2 and cropped to its lower half is sent 16 million rows down and dropped as off-region. Nothing crashes; the top row of the halftone simply vanishes

Layer 3: comparing fixed point instead of pixels

A skip test on fixed-point values is not equivalent to one on pixel positions once the fraction is non-zero. Dodging the logical shift by testing xx + HPW × 256 <= 0 on the unshifted value looks like an equivalent of the T.88 test. With HGX = -900 and a 4-pixel pattern, that gives -900 + 1024 = 124, which is positive, so the cell is not skipped. The standard shifts first: floor(-900 / 256) = -4, and -4 + 4 = 0 meets x + HPW <= 0, so the cell lies entirely outside and must be skipped. The encoder skips it, the decoder decodes it, and the greyscale image drifts exactly as in the transposed-mask case

Three JBIG2 halftone traps for a grid at negative HGX: a signed field read through an unsigned helper clamps to zero, the T.88 right shift rendered as a logical shr sends a pattern 16 million rows down, and a skip test on fixed-point values keeps a cell the encoder skipped
Each mistake hides the next one, which is why PDFlibPas handles all three layers in one shared HalftoneGridPixel helper used by the mask builder and the placement loop

Take a 4 × 3 grid of 4 × 4 patterns at HGX = -900, HGY = -512, HRX = 1024 on a 12 × 10 region. Grid columns land at x = -4, 0, 4 and 8, so column 0 is entirely outside and belongs in HSKIP; grid rows land at y = -2, 2 and 6, so row 0 must be cropped to its lower two pixel rows rather than dropped. Getting the layers right one at a time shows the stack:

Layers handled correctlyDecoded region
NoneGrid pulled to the origin, whole picture two rows too low
Signed HGX / HGY read onlyFirst grid row missing, the rest garbled by the skip-test drift
Signed read, floor shift and pixel-space skip testIdentical, pixel for pixel, to the page computed from T.88 §6.6.5 and to two independent reference decoders

PDFlibPas puts all of this in one helper, HalftoneGridPixel, shared by the skip-mask builder and the placement loop. It accumulates the coordinate in Int64 so a large mg × HRX product cannot wrap, divides by 256 rounding towards minus infinity, and clamps to ±MaxInt div 2 so a corrupt grid cannot overflow later bitmap arithmetic. The skip test compares those pixel values against HPW, HPH, HBW and HBH, exactly as §6.6.5.1 states it

How do you write an arithmetic right shift in Delphi?

Delphi has no arithmetic shift operator, so a correct signed right shift has to be written as a floor division, and plain div is not that division. div truncates towards zero. For non-negative values truncation and floor agree, and they also agree for negative values that are exact multiples of the divisor, which is why -512 div 256 = -2 looks fine in a quick test. They disagree everywhere else: -900 div 256 is -3, while the floor is -4, and -1 div 256 is 0, while the floor is -1. A JBIG2 coordinate with a non-zero fraction is exactly the case where div gives the wrong pixel

On the Delphi Win32 and Win64 compilers, an Integer variable holding -512 shifted right by 8 gives 16777214, and an Int64 holding -512 gives 72057594037927934. Free Pascal also defines shr as a logical shift and ships SarLongint and SarInt64 in its System unit for the arithmetic version, but those functions do not exist in Delphi, so code shared between the two compilers needs its own helper:

// Floor division: rounds toward minus infinity for any sign of A and B.
// B must not be 0, and FloorDiv(Low(Integer), -1) overflows just like div
function FloorDiv(A, B: Integer): Integer;
begin
  Result := A div B;
  if (A mod B <> 0) and ((A < 0) <> (B < 0)) then
    Dec(Result);
end;

// Arithmetic right shift (the C and T.88 ">>" on signed values).
// For negative Value, not Value = -Value - 1 is non-negative, so the
// logical shr is safe there, and the outer not maps the result back
function SarInt32(Value: Integer; Shift: Integer): Integer;  // Shift 0..31
begin
  if Value >= 0 then
    Result := Value shr Shift
  else
    Result := not ((not Value) shr Shift);
end;

function SarInt64(Value: Int64; Shift: Integer): Int64;      // Shift 0..63
begin
  if Value >= 0 then
    Result := Value shr Shift
  else
    Result := not ((not Value) shr Shift);
end;

The not trick never shifts a negative number, so it does not depend on how a compiler treats the sign bit, and it never overflows, including for Low(Integer). Both helpers matched an Int64 floor reference over several million values, every shift from 0 to 31 and the Low(Integer) and High(Integer) edges on Delphi Win32, Delphi Win64 and Free Pascal x86_64. A sanity check worth keeping in any unit test that touches coordinates:

var
  V: Integer;
begin
  V := -900;
  Writeln(V shr 8);           // 16777212  logical shift, the trap
  Writeln(V div 256);         // -3        truncation toward zero
  Writeln(FloorDiv(V, 256));  // -4        what T.88 means by >> 8
  Writeln(SarInt32(V, 8));    // -4
end;
PDFlibPas number line for the coordinate -900 shifted right by 8: shr gives 16777212, div truncates to -3, while FloorDiv and SarInt32 both land on the floor value -4 that ITU-T T.88 means by the shift, which matters only when the fixed-point fraction is non-zero
Truncation and floor agree only on exact multiples, so -512 div 256 passes a quick test and -900 div 256 picks the wrong pixel

Math.Floor(V / 256) also returns -4, but its detour through Double loses precision for Int64 values above 253, so integer geometry should stay in integers

Which PDFlibPas calls run the halftone decoder?

The JBIG2 halftone decoder runs when PDFlibPas renders a page with the built-in renderer, because rendering needs pixels. RenderPageToFile and RenderPageToStream both reach it through the page's JBIG2Decode image streams, so re-rendering a halftone page is the direct way to check what the decoder produces. The same decoder handles the other JBIG2 region types, covered in JBIG2 custom Huffman tables in the pure Pascal decoder and decoding random-access JBIG2 files in Delphi, and the rendered bitmap feeds conversions such as rendering PDF pages to 1-bit monochrome

uses
  SysUtils, PDFlibrary;

var
  Lib: TPDFlib;
  Page: Integer;
begin
  Lib := TPDFlib.Create;
  try
    if Lib.LoadFromFile('scanned-halftone.pdf', '') <> 1 then
      raise Exception.CreateFmt('Load failed, error %d', [Lib.LastErrorCode]);
    for Page := 1 to Lib.PageCount do
      // Rendering decodes every JBIG2 region, halftones included
      if Lib.RenderPageToFile(150, Page, PDF_RENDER_PNG,
        Format('page-%.3d.png', [Page])) <> 1 then
        Writeln('Page ', Page, ' was not rendered');
  finally
    Lib.Free;
  end;
end.

Image extraction normally takes a different path. GetPageImageList returns JBIG2 images in native form, and SaveImageListItemDataToFile or GetImageListItemDataToString hands you a standalone JBIG2 file built from the stream bytes: the file header, the JBIG2Globals data and an end-of-file segment around the page data. Property 400 of GetImageListItemIntProperty reports 6 for such an item. Nothing is decoded on that path, so an extracted .jb2 that looks correct in another viewer while the rendered page shows noise points at the decoding path rather than at the stream:

var
  ListID, I: Integer;
begin
  Lib.SelectPage(1);
  ListID := Lib.GetPageImageList(0);
  if ListID = 0 then
    Exit;
  try
    for I := 1 to Lib.GetImageListCount(ListID) do
      if Lib.GetImageListItemIntProperty(ListID, I, 400) = 6 then  // standalone JBIG2
        Lib.SaveImageListItemDataToFile(ListID, I, 0,
          Format('page1-image%d.jb2', [I]));
  finally
    Lib.ReleaseImageList(ListID);
  end;
end;

When masks or colour conversion force a rendered fallback, the item comes back as a decoded bitmap and the halftone decoder does run. More on image lists is in Delphi PDF text, image and font extraction

Quick reference: JBIG2 halftone grid rules

  • Index the skip mask as HSKIP[ng, mg], grid column first, and read it back in the same order wherever cells are placed (T.88 §6.6.5.1)
  • Test any halftone or grid code with a non-square grid and an asymmetric set of out-of-region cells, because a square grid can hide a transposed index completely
  • Read HGX and HGY as signed 32-bit values (T.88 §7.4.5.1.2), never through an unsigned helper that clamps negatives
  • Translate the standard's >> 8 as a floor division by 256, not as shr 8 and not as div 256
  • Run the skip test on shifted pixel positions; the fixed-point form differs whenever the fraction is non-zero, as HGX = -900 with a 4-pixel pattern shows
  • Accumulate grid coordinates in Int64 and clamp before handing them to bitmap code, so a corrupt grid cannot overflow

PDFlibPas renders, extracts and edits PDF documents from Delphi and C++Builder with a native Pascal JBIG2 decoder that handles halftone skip masks, negative grid origins and rotated grids as T.88 specifies. See the PDFlibPas Delphi PDF library for features, editions and a trial download