Chunked zlib on FPC: Why FlateDecode Needs One Stream

A chunked FlateDecode writer on Free Pascal has to keep a single zlib state alive across every chunk. The tempting shortcut, compressing each 64 KB chunk on its own, hands every chunk its own zlib header and checksum, so a 1 MiB attachment ends up as sixteen zlib members glued end to end. ISO 32000-1 FlateDecode expects exactly one zlib stream, and a standard decoder stops at the first end-of-stream marker, which means everything after the first 65,536 bytes silently disappears. PDF Library for Delphi keeps one zlib state alive across every chunk in both DeflateStream and InflateStream

The trap only exists on the FPC side of a cross-compiler unit, and only on the chunked streaming path, which is precisely why it is easy to miss: the Delphi branch and the string-based helpers each produce a single zlib stream, and small test payloads never reach the chunked code at all. It is a close cousin of the traps in five FPC porting pitfalls that Delphi tends to hide, except that here Delphi is not covering for anything. The shortcut simply rests on the wrong mental model of what a zlib stream is

Why does a chunked zlib stream get truncated by other PDF readers?

Because a zlib stream as defined by RFC 1950 is one container, not a sequence of them, and a conforming inflater treats the first Adler-32 trailer as the end of the data. The format is a two-byte header, one continuous RFC 1951 deflate bit stream whose last block carries the final-block flag, and a four-byte Adler-32 checksum over all the uncompressed bytes. ISO 32000-1 §7.4.4 defines /FlateDecode in exactly those terms. When inflate reaches the trailer it returns Z_STREAM_END and leaves any remaining input unread in avail_in. Nothing is wrong from its point of view, so it raises no error, and the bytes after the trailer are simply ignored. Concatenated members are a legitimate idea in gzip (RFC 1952 allows several members in one file), which is probably where the intuition came from, but zlib has no such rule and PDF never asked for one

The naive FPC DeflateStream reads the source 64 KB at a time and hands each chunk to ZFPCCompress, a helper that runs its own deflateInit2, compresses with Z_FINISH, and calls deflateEnd. Every chunk therefore comes out as a complete, valid, self-terminating zlib stream, and the function concatenates them into one AnsiString before writing it out. The result looks like Flate data, carries a correct header, and decodes without an error, but it decodes to the first 65,536 bytes only. The read side has the mirror-image trap: calling ZFPCInflate once per 64 KB of compressed input starts a fresh inflateInit2 each time. The second chunk begins in the middle of a deflate bit stream with no zlib header, so a new inflater rejects it, and a perfectly ordinary single-member stream from any other producer is decoded only as far as its first 64 KB of compressed bytes carry it

// Naive chunked FPC DeflateStream (simplified):
// ZFPCCompress does deflateInit2 / deflate(Z_FINISH) / deflateEnd,
// so every 64 KB chunk becomes a separate zlib member
Repeat
  ReadCount:= Source.Read(Input[1], ChunkSize);
  If (ReadCount> 0) Then
    Compressed:= Compressed+ ZFPCCompress(Copy(Input, 1, ReadCount), 6, False);
Until (ReadCount< ChunkSize);
If (Compressed<> '') Then
  Target.WriteBuffer(PAnsiChar(Compressed)^, Length(Compressed));

Which PDF Library for Delphi calls reach the chunked path?

On FPC, the chunked path writes every embedded file of 1 MiB or more, and reads every Flate stream extracted through the streaming API once its compressed size passes one chunk. TPDFStream.ReadFromStream decides how to encode incoming data. When Deflate is true and Stream.Size >= 1048576 it streams through DeflateStream; below that threshold it reads the whole source into memory and calls DeflateStr, the single-shot helper that compresses everything in one call. The ASCII85-plus-Flate filter chain skips the size test and always goes through DeflateStream, so on that path any payload larger than 64 KB already spans several chunks. The public entry points that feed ReadFromStream with compression switched on are the embedded-file writers:

• TPDFlib.EmbedFile and TPDFlib.AddEmbeddedFile, which read a file from disk into an /EmbeddedFile stream
• TPDFlib.AddAssociatedFileFromStream and TPDFlib.AddAssociatedFileFromFile, the PDF/A-3 associated-file writers used for e-invoice XML and other source data
• On the read side, TPDFlib.GetEmbeddedFileContentToStream and GetEmbeddedFileContentToFile, which decode through TPDFStream.WriteDecodedToStream and from there through InflateStream

Getting this wrong fails quietly at every layer. The writer stores /Params /Size and an MD5 /CheckSum computed from the original file, so the attachment dictionary advertises the full size even when the stream holds sixteen members. Any conforming reader, a Delphi build of the library included, stops cleanly at the first Z_STREAM_END and returns exactly 65,536 bytes. GetEmbeddedFileContentToStream returns 1, because it reports whether decoding raised an error, not whether the output matches /Size. Anyone who has chased a large-document problem through merging and splitting gigabyte PDFs knows this pattern: the file opens, the page count is right, and the damage only shows when someone opens the attachment

One deflate state across every chunk

The DeflateStream in PDFlibZLib.pas initialises one paszlib.TZStream, feeds every chunk to deflate with Z_NO_FLUSH, and only at the end drains the compressor with Z_FINISH until it returns Z_STREAM_END. That produces exactly one header, one deflate bit stream whose back-references can reach across chunk boundaries, and one Adler-32 over the whole input. The FPC branch has the same structure as the Delphi branch. It also writes output as it is produced, instead of concatenating the entire compressed result into an AnsiString first, so the writer never builds a second full copy of the compressed data in memory before it copies it to the target

// FPC DeflateStream with one zlib state (error paths trimmed)
If (deflateInit2(strm, Level, Z_DEFLATED, 15, 8, Z_DEFAULT_STRATEGY)<> Z_OK) Then
  Exit;
Try
  Repeat
    ReadCount:= Source.Read(Input[0], ChunkSize);
    If (ReadCount> 0) Then
    Begin
      strm.next_in:= Pointer(Input);
      strm.avail_in:= ReadCount;
      While (strm.avail_in> 0) Do
      Begin
        strm.next_out:= Pointer(Output);
        strm.avail_out:= ChunkSize;
        Status:= deflate(strm, Z_NO_FLUSH);   // same state, no member break
        Produced:= ChunkSize- strm.avail_out;
        If (Produced> 0) Then
          Target.WriteBuffer(Output[0], Produced);
        If (Status<> Z_OK) Then
          Break;
      End;
    End;
  Until (ReadCount< ChunkSize);
  Repeat                                    // one trailer for the whole input
    strm.next_out:= Pointer(Output);
    strm.avail_out:= ChunkSize;
    Status:= deflate(strm, Z_FINISH);
    Produced:= ChunkSize- strm.avail_out;
    If (Produced> 0) Then
      Target.WriteBuffer(Output[0], Produced);
  Until (Status= Z_STREAM_END);
Finally
  deflateEnd(strm);
End;

InflateStream follows the same pattern: one inflateInit2, an inner loop that keeps calling inflate until the current chunk is consumed and the output buffer is no longer full, and a stop on Z_STREAM_END. There is one side effect worth knowing about. Because the writer calls deflateInit2 itself instead of going through a fixed-level helper, it honours PLDeflateLevel, so a level set through TPDFlib.SetCompressionLevel(1..9) applies to large embedded files on FPC as well. That matters if you already tune compression for archival output as described in reducing PDF file size in Delphi

How do you verify that a Flate stream is a single zlib member?

Inflate it with a plain zlib decoder and check two things when it returns Z_STREAM_END: the decoded length equals the source length, and avail_in is zero. Leftover input after the end marker is the signature of a concatenated stream. The routine below is the FPC version of that check, built directly on paszlib so it does not trust the code under test, followed by a probe that pushes 200,000 bytes, four 64 KB chunks, through DeflateStream

uses Classes, SysUtils, paszlib, PDFlibZLib;

function IsSingleZlibMember(Packed: TMemoryStream; out Decoded: Int64): Boolean;
var
  strm: TZStream;
  Buf: array[0..65535] of Byte;
  Status: Integer;
begin
  Result:= False;
  Decoded:= 0;
  FillChar(strm, SizeOf(strm), 0);
  if inflateInit2(strm, 15) <> Z_OK then
    Exit;
  try
    strm.next_in:= Packed.Memory;
    strm.avail_in:= Cardinal(Packed.Size);
    repeat
      strm.next_out:= @Buf;
      strm.avail_out:= SizeOf(Buf);
      Status:= inflate(strm, Z_NO_FLUSH);
    until Status <> Z_OK;
    Decoded:= strm.total_out;
    // One member ends exactly at the last input byte
    Result:= (Status = Z_STREAM_END) and (strm.avail_in = 0);
  finally
    inflateEnd(strm);
  end;
end;

// Push 200,000 bytes through DeflateStream with the default 64 KB chunk
// and require one member that decodes back to the full length
Source.Position:= 0;
DeflateStream(Source, Packed);
if not (IsSingleZlibMember(Packed, Decoded) and (Decoded = Source.Size)) then
  raise Exception.Create('DeflateStream produced more than one zlib member');

At the application level, the useful assertion is the one the library does not make for you: compare what comes out of an attachment with the /Params /Size recorded when it went in. GetEmbeddedFileIntProperty with tag 5 returns that recorded size, embedded-file indexes are 1-based, and the payload needs to be at least 1 MiB to exercise the streaming path. Run the same test on every compiler you ship with, since this class of mistake passes on Delphi and fails only on FPC

procedure CheckLargeAttachmentRoundTrip(const PayloadFile, OutFile: string);
var
  PDF: TPDFlib;
  Extracted: TMemoryStream;
  I, Declared: Integer;
begin
  PDF:= TPDFlib.Create;
  try
    PDF.NewDocument;
    PDF.AddStandardFont(4);
    PDF.DrawText(80, 100, 'Large attachment round trip');
    // 1 MiB or more takes the chunked DeflateStream path in ReadFromStream
    if PDF.EmbedFile('Payload', PayloadFile, 'application/octet-stream') <> 1 then
      raise Exception.Create('EmbedFile failed');
    if PDF.SaveToFile(OutFile) <> 1 then
      raise Exception.Create('SaveToFile failed');
  finally
    PDF.Free;
  end;

  PDF:= TPDFlib.Create;
  Extracted:= TMemoryStream.Create;
  try
    if PDF.LoadFromFile(OutFile, '') = 0 then
      raise Exception.Create('LoadFromFile failed');
    for I:= 1 to PDF.EmbeddedFileCount do
    begin
      Extracted.Clear;
      if PDF.GetEmbeddedFileContentToStream(I, Extracted) <> 1 then
        raise Exception.CreateFmt('Attachment %d could not be decoded', [I]);
      Declared:= PDF.GetEmbeddedFileIntProperty(I, 5);   // /Params /Size
      if Extracted.Size <> Declared then
        raise Exception.CreateFmt('Attachment %d truncated: %d of %d bytes',
          [I, Extracted.Size, Declared]);
    end;
  finally
    Extracted.Free;
    PDF.Free;
  end;
end;

Edge cases on the FPC branch

The FPC branch uses lenient decoding rules. When MaxOutput is reached, FPC InflateStream truncates at the limit and returns, while the Delphi branch raises ERangeError, and FPC still accepts partially decoded output when inflate reports a data error, because some PDF producers emit truncated or checksum-broken streams. A PDF that already contains concatenated members, whatever wrote it, stops at the first Z_STREAM_END in every reader. Do not try to heal such a file by decoding and re-encoding the stream inside the library, since that only makes the 64 KB truncation permanent. Re-embed the attachment from its original source instead. The FPC loop treats a read that returns less than a full chunk as the end of the data, which holds for streams such as TFileStream and TMemoryStream, so a custom TStream passed to AddAssociatedFileFromStream is safest copied into a TMemoryStream first

The single-state FPC DeflateStream and InflateStream ship in PDF Library for Delphi, which targets Delphi, C++Builder, and Free Pascal from one source tree, so a large attachment comes back from an FPC build byte for byte, just as it does from Delphi