Technical Article

UTF-8 in Data Matrix and PDF417: The ECI 26 Marker

A warehouse label stores the consignee Müller GmbH, Köln as UTF-8 in a Data Matrix symbol, and every handheld scanner on the floor reads it back as Müller GmbH, Köln. Setting the PDFlibPas property BarcodePayloadEncoding to bpeUTF8WithECI puts an ECI 000026 marker in front of the data, which tells Data Matrix and PDF417 readers to decode the bytes as UTF-8

Why does a UTF-8 payload decode as Müller?

The symbol carries no charset declaration, so the reader falls back to the symbology's default single-byte character set, which for Data Matrix is ISO-8859-1. A UTF-8 ü is the two bytes C3 BC, which ISO-8859-1 maps to à and ¼; ö becomes ö the same way. The bytes themselves are intact. Extended Channel Interpretation is the barcode world's charset declaration. The AIM ECI registry assigns numbers to character sets, 000026 among them for UTF-8, and a symbol that opens with an ECI designator tells a conforming reader to decode everything after it in that character set. To the symbol the designator is ordinary data. It takes two codewords in both symbologies discussed here, counts toward capacity, and is protected by the same Reed-Solomon error correction for 2D barcodes as the payload behind it

How identical UTF-8 bytes decode two ways in PDFlibPas barcodes from Delphi: no ECI marker means an ISO-8859-1 guess, ECI 000026 means UTF-8
the symbol stores bytes only, so the ECI 000026 designator is what tells the reader the bytes are UTF-8 rather than an ISO-8859-1 guess

ECI codeword positions in Data Matrix and PDF417

Each symbology has a fixed slot for the marker. Put it anywhere else and the symbol either fails to decode or decodes with garbage at the front:

Codeword slots for the ECI UTF-8 marker in PDFlibPas Data Matrix, GS1 Data Matrix and PDF417 symbols from Delphi, each fixed ahead of the payload
every symbology pins the ECI designator to a fixed leading slot, and PDFlibPas writes it before position-dependent encodations run
  • Data Matrix (ISO/IEC 16022): codeword 241, then the ECI number plus one in the single-codeword form used for values below 127, so 27 for UTF-8. It opens the data stream
  • GS1 Data Matrix: FNC1 (codeword 232) must stay in first position, so PDFlibPas places the marker right after it if you ask for one; GS1 element strings are drawn from a subset of ASCII, though, and read the same without it
  • PDF417 (ISO/IEC 15438): codeword 0 is the symbol length descriptor, and the marker follows as codewords 927 and 26, ahead of the first data codeword

Base 256 is the hard case. That encodation, the compact Data Matrix mode for arbitrary bytes, scrambles each codeword with a pseudo-random value derived from its absolute position in the symbol, (149 × position mod 255) + 1. Prepend the marker after scrambling and every byte behind it was scrambled for a position two places earlier than the one the reader sees, so the whole payload unscrambles to wrong values. EDIFACT encodation has a milder version of the same dependency — its unlatch position shifts by two when the marker goes in front. PDFlibPas writes the marker into the codeword stream before either step runs, and the capacity check that picks the symbol size already includes it. The label code for both symbologies:

uses
  SysUtils, PDFlibrary;

procedure PrintConsigneeLabel(const FileName: string);
var
  Pdf: TPDFlib;
begin
  Pdf := TPDFlib.Create;
  try
    Pdf.NewDocument;                                // creates and selects page 1
    Pdf.SetPageSize('A4');
    Pdf.SetMeasurementUnits(1);                     // millimetres
    Pdf.SetOrigin(1);                               // Top is measured from the top edge

    Pdf.BarcodePayloadEncoding := bpeUTF8WithECI;   // UTF-8 bytes + ECI 000026
    if Pdf.DrawDataMatrixSymbol(20, 20, 0.5, 'Müller GmbH, Köln',
      1,      // Encoding: 1 = ASCII encodation
      0,      // SymbolSize: 0 = first size in the table that fits
      100) <> 1 then                                // normal orientation, default 1-module quiet zone
      raise Exception.CreateFmt('Data Matrix failed (%d)', [Pdf.LastErrorCode]);

    if Pdf.DrawPDF417SymbolEx(20, 50, 'Zustellung: Müller GmbH, Köln',
      0, 0, 0,  // orientation, columns, rows: automatic
      -1,       // error level automatic
      0.3, 3.0) <> 1 then
      raise Exception.Create('PDF417 failed');

    // An ECI-aware scanner should return the German text for both symbols
    Pdf.BarcodePayloadEncoding := bpeSystemANSI;    // back to the default
    Pdf.SaveToFile(FileName);
  finally
    Pdf.Free;
  end;
end;

NewDocument already creates page 1 and selects it, so the symbols land on the first page without a NewPage call. SetOrigin(1) moves the origin to the top-left corner, which makes Top the distance of the symbol's top edge from the top of the page. Under the default bottom-left origin, a Top of 10 mm puts that edge 10 mm above the bottom of the sheet, and an 11 mm symbol hangs off it. DrawDataMatrixSymbol returns 1 on success. For data it cannot encode — a payload that outgrows the largest symbol once the marker is counted, or the fixed size you asked for — it returns 0, sets LastErrorCode to 302 and draws nothing, so the example treats anything other than 1 as an error

Should every label carry the ECI marker?

No, and the default reflects that. BarcodePayloadEncoding takes three values, and each fits a different reader population:

  • bpeSystemANSI (the default) converts the text through the system ANSI code page and writes no marker, which is what label layouts built with the PDFlibPas QR, PDF417 and Data Matrix functions expect
  • bpeUTF8 writes UTF-8 bytes without a marker, for closed systems in which your own software reads the barcode and already assumes UTF-8
  • bpeUTF8WithECI writes UTF-8 bytes and labels them with ECI 000026

The price of labelling everything is reader support. ECI processing is a configuration option on many scanners, and depending on that setting a device may refuse the symbol or pass the designator to the host as text: the AIM convention is a backslash and six digits, so the application receives \000026Müller GmbH, Köln. Turn the marker on per symbol, for payloads that contain non-ASCII text, and roll it out together with the scanner profile change. A single-byte payload is still a fair choice when every name fits the code page your readers assume. ü is byte FC in both Windows-1252 and ISO-8859-1, so a German address written under bpeSystemANSI on a Western European Windows system decodes correctly on a Data Matrix reader that has never heard of ECI. Japanese text has no such fallback

Symbol size: auto selection, fixed templates and rectangles

The marker's two codewords rarely act alone. In ASCII encodation the consignee line under bpeSystemANSI holds ü and ö as single bytes above 127 at two codewords each, and the whole line costs 19 codewords, which fits a 20×20 symbol with its 22 data codewords. As UTF-8 the two letters become four high bytes and the line costs 23. The marker brings it to 25. That needs 22×22, and the step comes from the UTF-8 bytes alone, since 23 already overflows 20×20. A template that fixes SymbolSize at 9 (20×20) gets 0 and error 302 from DrawDataMatrixSymbol as soon as the encoding changes, and nothing is drawn in the barcode slot

Auto sizing can change the shape. With SymbolSize at 0, PDFlibPas walks the ISO/IEC 16022 size list in order and takes the first entry whose data capacity holds the stream. The six rectangular sizes sit in that list between the squares. A payload that filled a 26×26 symbol (44 data codewords) needs 46 with the marker, and the next entry that holds 46 is 16×48 with 49, ahead of 32×32 with 62. At 0.5 mm per module a 13 mm square becomes an 8 × 24 mm strip. A full 22×22 does the same thing: 30 codewords plus two land in 16×36. Small payloads stay square — three codewords that filled 10×10 become five and fit 12×12, which precedes 8×18 in the list. If the label reserves a square slot, pin SymbolSize to the square you can afford (16 for 32×32) and let error 302 tell you when a name is too long for it; auto sizing suits layouts that can take either shape

PDF417 grows by a different rule. With ErrorLevel at -1, the encoder plans error correction from the recommended minimum levels in ISO/IEC 15438, which step up at 41, 161 and 321 codewords, length descriptor included. A 39-codeword stream becomes 41 with the marker, the minimum level moves from 2 to 3, and the error-correction block it plans for grows from 8 to 16 codewords. That is ten extra codewords. With automatic columns and rows, the encoder lays out the rectangle from that larger total, so a payload sitting just below a step gains more area than its two marker codewords suggest

Which encodation keeps a UTF-8 Data Matrix small?

Compare the two costs directly. ASCII encodation packs a digit pair into one codeword and plain ASCII at one per character, but every byte above 127 costs two: an Upper Shift (235) and the byte minus 127. Base 256 pays a latch and a length field once — one length codeword up to 249 bytes, two beyond that — and then one codeword per byte, with no digit packing. For text without digit runs, Base 256 wins as soon as a payload holds three high bytes. The Müller line already does, at 21 codewords against 23. Long numeric runs push the balance back toward ASCII, because every digit pair saves a codeword there. The helper below counts both costs and draws with the cheaper mode; pure ASCII gets no marker at all, since it reads identically under every character set:

// Draw a Data Matrix for Unicode text with the cheaper encodation
function DrawUnicodeDataMatrix(Pdf: TPDFlib; Left, Top, ModuleSize: Double;
  const Text: WideString): Integer;
var
  Bytes: UTF8String;
  I, HighBytes, AsciiCost, Base256Cost, Mode: Integer;
  Saved: TPDFlibBarcodePayloadEncoding;
begin
  Bytes := UTF8Encode(Text);
  HighBytes := 0;
  AsciiCost := 0;
  I := 1;
  while I <= Length(Bytes) do
  begin
    if (I < Length(Bytes)) and (Bytes[I] >= '0') and (Bytes[I] <= '9') and
       (Bytes[I + 1] >= '0') and (Bytes[I + 1] <= '9') then
    begin
      Inc(AsciiCost);                    // digit pair: one codeword
      Inc(I, 2);
      Continue;
    end;
    if Ord(Bytes[I]) > 127 then
    begin
      Inc(HighBytes);
      Inc(AsciiCost, 2);                 // Upper Shift + (byte - 127)
    end
    else
      Inc(AsciiCost);
    Inc(I);
  end;
  if Length(Bytes) <= 249 then
    Base256Cost := Length(Bytes) + 2     // latch + one length codeword
  else
    Base256Cost := Length(Bytes) + 3;    // latch + two length codewords

  Saved := Pdf.BarcodePayloadEncoding;
  try
    if HighBytes = 0 then
    begin
      Pdf.BarcodePayloadEncoding := bpeSystemANSI;  // pure ASCII: no marker
      Mode := 1;
    end
    else
    begin
      Pdf.BarcodePayloadEncoding := bpeUTF8WithECI;
      if Base256Cost < AsciiCost then
        Mode := 3                        // Base 256
      else
        Mode := 1;                       // ASCII
    end;
    Result := Pdf.DrawDataMatrixSymbol(Left, Top, ModuleSize, Text, Mode, 0, 100);
  finally
    Pdf.BarcodePayloadEncoding := Saved;
  end;
end;

For 東京都千代田区 the UTF-8 form is 21 bytes, all above 127. With the marker, Base 256 needs 25 data codewords and ASCII encodation 44, which by the capacity table is a 22×22 symbol against a 26×26 one at the same module size. That 26×26 is full. One more codeword and it jumps to 16×48. GS1 payloads (mode 7) stay outside the helper because their element strings are ASCII. QR codes take their data mode through the EncodeOptions argument of DrawQRCode, and Aztec symbols drawn by PDFlibPas carry an ECI 26 flag automatically whenever the payload bytes are valid UTF-8 with at least one non-ASCII character

ASCII versus Base 256 encodation costs for PDFlibPas Data Matrix barcodes in Delphi, with worked codeword totals and the mode the helper picks
base 256 wins once three or more high bytes appear, while digit-heavy payloads swing the cost back toward ASCII encodation

For the Müller line, the marker costs two codewords, while the move from 20×20 to 22×22 — 10 mm to 11 mm at 0.5 mm modules — is caused by the UTF-8 bytes. If your label slot has that extra millimetre, set BarcodePayloadEncoding to bpeUTF8WithECI on the TPDFlib instance of the PDFlibPas Delphi PDF Library just before the consignee symbol is drawn, and set it back to bpeSystemANSI for the legacy symbols on the same page