LlosLab Software Development BlogEngineering notes for PDF, Delphi, and document software developers.

Technical Article

FPC Codepage Traps Break PDF/A XMP Metadata in Delphi

PDFium Component

The PDFium Delphi Component assembles the XMP packet for PDF/A output by concatenating UTF-8 fragments into an AnsiString, and on Free Pascal 3.2.2 that packet quietly stopped being valid UTF-8 the moment a document title carried a non-ASCII character. ISO 19005-1 6.7.2 requires the metadata stream to be valid UTF-8, so the file failed validation. Version 3.103.1 fixes the encoder itself, in StringToUtf8. The interesting part is not the patch. It is that one unchanged source line produced correct bytes under Delphi, correct bytes in a Lazarus LCL application, and corrupt bytes in a plain Free Pascal console program compiled from the identical unit. Three separate Free Pascal string behaviours have to line up before that makes sense, and every one of them is defensible on its own

Why does the same metadata code emit different bytes on Delphi and FPC?

Because string is not the same type on the two compilers. FPC 3.2.2 in {$MODE Delphi} compiles string to an AnsiString tagged with DefaultSystemCodePage, while Delphi compiles it to UnicodeString. Every metadata field in TPdfASaveOptions is declared string, so Title, Author, Subject, Keywords, Creator and Producer carry UTF-16 code units on one compiler and single-byte characters plus a codepage label on the other. Same record, same field, different payload. The values themselves arrive from the document as UTF-16. TPdf.GetTitle and its siblings return WString, which is WideString on FPC and string on Delphi, and SaveAsPdfAToStream fills any blank option field from the Info dictionary before injecting markers. That assignment is a narrowing conversion on Free Pascal, and the RTL performs it through the target string codepage. In an LCL program LazUTF8 has already set DefaultSystemCodePage to CP_UTF8, so the narrowing produces UTF-8 and everything downstream happens to be correct. In a plain console program the same narrowing lands on the ANSI codepage, and StringToUtf8 then copied those octets through unchanged because it assumed they were already UTF-8. Six save bridges share this shape: SaveAsPdfAToStream, SaveAsPdfUaToStream, SaveAsPdfEToStream, SaveAsPdfXToStream, SaveAsPdfRToStream and SaveAsPdfVTToStream, each with its own option record

// PDFium.pas: the document accessors are always UTF-16
//   WString = WideString on FPC, = string (UnicodeString) on Delphi
function TPdf.GetTitle: WString;

// FPdfPdfa.pas: the save option record carries metadata as `string`
TPdfASaveOptions = record
  Conformance: TPdfAConformance;
  IccProfileData: TBytes;
  Title: string;      // UnicodeString on Delphi
                      // AnsiString + DefaultSystemCodePage on FPC
  Author: string;
  Subject: string;
  Keywords: string;
  Creator: string;
  Producer: string;
  CreationDate: string;
  ModDate: string;
  DocumentId: TBytes;
  InstanceId: TBytes;
  class function Default: TPdfASaveOptions; static;
end;

// SaveAsPdfAToStream backfills blank fields from the Info dictionary.
// The narrowing is now spelled out instead of left implicit:
if Eff.Title = '' then
  Eff.Title := WStringToStr(GetTitle);

Routing all six bridges through one WStringToStr helper does not change what the RTL does, but it puts the conversion where a reader can see it, and it cleared 92 implicit-conversion warnings that had been masking exactly this class of problem. This is the mirror image of the Delphi-side corruption described in our notes on Delphi and FPC cross-compiler pitfalls in PDFium builds, where a concatenation on Delphi destroys a high byte that Free Pascal preserves

Three Free Pascal behaviours that defeat the obvious fix

The obvious fix is to call UTF8Encode and be done. That fails three times over on FPC 3.2.2 in mode Delphi, and each failure is silent

var
  W: UnicodeString;
  S: string;
  U: UTF8String;
  R: RawByteString;
  Xmp: AnsiString;
begin
  // Trap 1: in mode Delphi a UTF8String *variable* is a plain AnsiString,
  // so the assignment transcodes the octets straight back to the host codepage
  U := UTF8Encode(W);

  // Trap 2: S is already an AnsiString, so UTF8Encode does nothing at all
  R := UTF8Encode(S);                 // no decode, no encode, no error
  R := UTF8Encode(UnicodeString(S));  // this one really encodes

  // Trap 3: concatenation unifies every operand to the destination codepage,
  // and a RawByteString destination is not an exception
  Xmp := Xmp + R;
end;

Trap one means the encoded result has to stay in the AnsiString or RawByteString it was produced in. Pass it through a UTF8String temporary on the way out and you have undone the work. Trap two is the one that hides longest, because UTF8Encode(S) compiles, runs, returns a value of the right length and performs no conversion whatsoever when its argument is already an AnsiString; only widening to UnicodeString first makes the call decode anything. Trap three is why a correct encoder can still produce a broken document: BuildXmpBytes accumulates the packet in a local Xmp: AnsiString, and Free Pascal converts every operand of a concatenation to the destination variable's codepage, folding the multi-byte sequences back down to single ANSI bytes on the way in

What does SetCodePage with False actually guarantee?

SetCodePage(RawByteString(Result), DefaultSystemCodePage, False) relabels the string without touching a byte. The third parameter is Convert; passing False means "assume the payload is already in the target codepage and just change the tag". That is a lie about the content, told deliberately: the octets really are UTF-8, but tagging them as the host codepage is what stops the concatenation in trap three from converting them. They join the XMP buffer as raw bytes and come out the far side unchanged

function StringToUtf8(const S: string): AnsiString;
begin
{$IFDEF UNICODE}
  // Delphi: UTF8Encode already yields CP_UTF8-tagged octets, and
  // concatenation into an AnsiString keeps them
  Result := AnsiString(UTF8Encode(S));
{$ELSE}
  // FPC: widen first, or UTF8Encode is a no-op on an AnsiString argument
  Result := UTF8Encode(UnicodeString(S));
  // Relabel without transcoding, so the octets survive concatenation into
  // the ANSI-tagged buffers that assemble XMP packets and PDF string objects
  if Length(Result) > 0 then
    SetCodePage(RawByteString(Result), DefaultSystemCodePage, False);
{$ENDIF}
end;

Be clear about the boundary. The retag is FPC-only, and it is not a general blessing on mixing tagged and untagged strings. It works here because exactly one consumer pattern exists downstream: append to an AnsiString, then write the buffer out as bytes. Anything that tried to interpret the retagged value as text on the host codepage would read mojibake, and correctly so. The reverse direction is handled the other way round and is identical on both compilers: tag the incoming buffer CP_UTF8 with SetCodePage(..., False), then call UTF8ToString

Why did the regression tests carry the same trap?

Because a test that builds its expected bytes from a source literal is testing the compiler, not the library. A constant such as #$C3#$A9 written in a Pascal source file carries the compile-time codepage of that file, and when it is passed to an AnsiString parameter the RTL re-encodes it, which is precisely the conversion under test. The expectation has to be assembled at run time, byte by byte, and compared byte by byte, because = on two AnsiString values with different tags reconciles the codepages before comparing and returns a cheerful false negative

function BytesPattern(const Values: array of Byte): AnsiString;
var
  I: Integer;
begin
  SetLength(Result, Length(Values));
  for I := 0 to High(Values) do
    Result[I + 1] := AnsiChar(Values[I]);
end;

procedure TPdfATests.StringToUtf8_AnsiCodePageString_EncodesUtf8;
var
  Saved: Word;
  Wide: WideString;
  Narrowed: string;
  Encoded, ExpectedUtf8: AnsiString;
begin
  Saved := DefaultSystemCodePage;
  try
    SetMultiByteConversionCodePage(1252);
    Wide := WideChar($0043) + WideChar($0061) + WideChar($0066) + WideChar($00E9);
    Narrowed := Wide;                    // the narrowing under test
    Encoded := StringToUtf8(Narrowed);
  finally
    SetMultiByteConversionCodePage(Saved);
  end;
  // 'Caf' + U+00E9 as UTF-8, built at run time so no literal can be re-encoded
  ExpectedUtf8 := BytesPattern([$43, $61, $66, $C3, $A9]);
  AssertTrue('StringToUtf8 must emit UTF-8 for a string carrying a non-UTF-8 codepage',
    SameOctets(Encoded, ExpectedUtf8));
end;

The harness itself is an LCL program, so DefaultSystemCodePage is CP_UTF8 and the bug is invisible until the test switches it. SetMultiByteConversionCodePage(1252) inside a try..finally reproduces the plain console environment for the duration of one test. The end-to-end check goes further and asserts both directions: the XMP packet produced by marker injection must contain $43 $61 $66 $C3 $A9 and must not contain $43 $61 $66 $E9, so a future regression that reverts to raw single-byte output fails loudly instead of producing a file that merely looks plausible in a hex dump. If you work with non-Latin metadata, the same widening discipline governs the cases in emoji and CJK text that break WideChar handling in Delphi

Where else the narrowing lands

XMP is the visible casualty, but any TBytes to string bridge in the same codebase had the same exposure. Two more were corrected in v3.103.1: Utf8BytesToString and StringToUtf8Bytes in FPdfProduction, which round-trip the XFA datasets packet through a string so that MergePdfXfaDatasets can substitute bound values, and BytesToUtf8 in FPdfTrustedList, which decodes European trusted-list XML after stripping the byte-order mark. Both now stage the buffer in a RawByteString, tag it CP_UTF8 without converting, and decode with UTF8ToString. One module was already immune, and the reason is worth copying. The XFDF writer declares its own text type as XFDFString, which resolves to WideString under FPC and UnicodeString under Delphi, so its encoder never sees a codepage-tagged AnsiString at all. That is the structural fix: keep text in a UTF-16 type until the exact point of serialisation, and let a single narrow function own the conversion to bytes. Every bug in this family came from a string field sitting in the middle of a pipeline that was otherwise UTF-16 at one end and octets at the other

What to check in your own dual-compiler PDF code

If you ship Object Pascal that runs on both compilers and writes metadata into a standards-conformant PDF, four checks find most of this class before a validator does

  • Grep for UTF8Encode with a string argument. On FPC that call is a no-op, and it is the single highest-yield line to audit
  • Treat every UTF8String variable as suspect in mode Delphi. It is a plain AnsiString there, and assigning encoded bytes to one transcodes them back
  • Run at least one regression under SetMultiByteConversionCodePage with a single-byte codepage. An LCL test harness runs at CP_UTF8 and will never reproduce a plain console program
  • Build expected byte vectors at run time and compare them octet by octet. Source literals and = both go through codepage reconciliation and will hide the defect you are hunting

None of this is exotic Free Pascal trivia. It is the ordinary cost of a language that kept a byte-oriented string type alive alongside a UTF-16 one, and the two compilers made reasonable but different choices about which of them string should mean. The practical consequence for PDF work is narrow and sharp: metadata that reads fine in your IDE can reach the XMP packet as invalid UTF-8, and ISO 19005-1 6.7.2 does not care which compiler put it there. If you are building an archival pipeline, the encoding layer deserves as much attention as the rest of the PDF/A archival compliance workflow that surrounds it. The PDFium Delphi Component ships these conversions as part of the library, so SaveAsPdfA and its five standards siblings emit conformant UTF-8 metadata on Delphi, on Lazarus and on plain Free Pascal builds without any codepage configuration from the caller. Full API documentation and the current release are at the PDFium Delphi Component product page