Chunked zlib on FPC: Why FlateDecode Needs One Stream

Before v3.539.24, the Free Pascal build of PDF Library for Delphi compressed large streams in 64 KB chunks and gave every chunk its own zlib header and checksum, so a 1 MiB attachment became sixteen zlib members glued end to end. ISO 32000-1 FlateDecode expects exactly one zlib stream, and a standard decoder stops at the first end-of-stream marker, which means everything after the first 65,536 bytes silently disappeared. The fix keeps one zlib state alive across every chunk in both DeflateStream and InflateStream

The bug only existed on the FPC side of the unit, and only on the chunked streaming path, which is exactly why it survived: the Delphi branch was always correct, the string-based helpers were always correct, and small test payloads never reached the chunked code at all. It is a close cousin of the defects in five FPC porting bugs that Delphi had been hiding, except that here the Delphi build was not covering for anything. The FPC branch had simply been written against a wrong mental model of what a zlib stream is

Why does a chunked zlib stream get truncated by other PDF readers?

Because a zlib stream as defined by RFC 1950 is one container, not a sequence of them, and a conforming inflater treats the first Adler-32 trailer as the end of the data. The format is a two-byte header, one continuous RFC 1951 deflate bit stream whose last block carries the final-block flag, and a four-byte Adler-32 checksum over all the uncompressed bytes. ISO 32000-1 §7.4.4 defines /FlateDecode in exactly those terms. When inflate reaches the trailer it returns Z_STREAM_END and leaves any remaining input unread in avail_in. Nothing is wrong from its point of view, so it raises no error, and the bytes after the trailer are simply ignored. Concatenated members are a legitimate idea in gzip (RFC 1952 allows several members in one file), which is probably where the intuition came from, but zlib has no such rule and PDF never asked for one

The old FPC DeflateStream read the source 64 KB at a time and handed each chunk to ZFPCCompress, a helper that runs its own deflateInit2, compresses with Z_FINISH, and calls deflateEnd. Every chunk therefore came out as a complete, valid, self-terminated zlib stream, and the function concatenated them into one AnsiString before writing it out. The result looked like Flate data, had a correct header, and decoded without an error, but it decoded to the first 65,536 bytes only. On the read side, InflateStream had the mirror-image defect: it called ZFPCInflate once per 64 KB of compressed input, and each call started a fresh inflateInit2. The second chunk begins in the middle of a deflate bit stream with no zlib header, so a new inflater rejects it, and a perfectly normal single-member stream from any other producer was decoded only as far as its first 64 KB of compressed bytes carried it

// FPC DeflateStream before v3.539.24 (simplified):
// ZFPCCompress does deflateInit2 / deflate(Z_FINISH) / deflateEnd,
// so every 64 KB chunk becomes a separate zlib member
Repeat
  ReadCount:= Source.Read(Input[1], ChunkSize);
  If (ReadCount> 0) Then
    Compressed:= Compressed+ ZFPCCompress(Copy(Input, 1, ReadCount), 6, False);
Until (ReadCount< ChunkSize);
If (Compressed<> '') Then
  Target.WriteBuffer(PAnsiChar(Compressed)^, Length(Compressed));

Which PDF Library for Delphi calls reached the chunked path?

On FPC, any embedded file of 1 MiB or more was written wrong, and any Flate stream extracted through the streaming API was read wrong once its compressed size passed one chunk. TPDFStream.ReadFromStream decides how to encode incoming data. When Deflate is true and Stream.Size >= 1048576 it streams through DeflateStream; below that threshold it reads the whole source into memory and calls DeflateStr, the single-shot helper that was never affected. The ASCII85-plus-Flate filter chain skips the size test and always goes through DeflateStream, so on that path any payload larger than 64 KB was already split into several members. The public entry points that feed ReadFromStream with compression switched on are the embedded-file writers:

• TPDFlib.EmbedFile and TPDFlib.AddEmbeddedFile, which read a file from disk into an /EmbeddedFile stream
• TPDFlib.AddAssociatedFileFromStream and TPDFlib.AddAssociatedFileFromFile, the PDF/A-3 associated-file writers used for e-invoice XML and other source data
• On the read side, TPDFlib.GetEmbeddedFileContentToStream and GetEmbeddedFileContentToFile, which decode through TPDFStream.WriteDecodedToStream and from there through InflateStream

The failure was quiet at every layer. The writer stores /Params /Size and an MD5 /CheckSum computed from the original file, so the attachment dictionary advertised the full size while the stream held sixteen members. A Delphi build of the same library reading that file stopped cleanly at the first Z_STREAM_END and returned exactly 65,536 bytes. GetEmbeddedFileContentToStream returned 1, because it reports whether decoding raised an error, not whether the output matches /Size. Anyone who has chased a large-document problem through merging and splitting gigabyte PDFs knows this pattern: the file opens, the page count is right, and the damage only shows when someone opens the attachment

One deflate state across every chunk

The fixed DeflateStream in PDFlibZLib.pas initialises one paszlib.TZStream, feeds every chunk to deflate with Z_NO_FLUSH, and only at the end drains the compressor with Z_FINISH until it returns Z_STREAM_END. That produces exactly one header, one deflate bit stream whose back-references can reach across chunk boundaries, and one Adler-32 over the whole input. The FPC branch now has the same structure the Delphi branch always had. It also writes output as it is produced, instead of concatenating the entire compressed result into an AnsiString first, so the writer no longer builds a second full copy of the compressed data in memory before it copies it to the target

// FPC DeflateStream from v3.539.24 (error paths trimmed)
If (deflateInit2(strm, Level, Z_DEFLATED, 15, 8, Z_DEFAULT_STRATEGY)<> Z_OK) Then
  Exit;
Try
  Repeat
    ReadCount:= Source.Read(Input[0], ChunkSize);
    If (ReadCount> 0) Then
    Begin
      strm.next_in:= Pointer(Input);
      strm.avail_in:= ReadCount;
      While (strm.avail_in> 0) Do
      Begin
        strm.next_out:= Pointer(Output);
        strm.avail_out:= ChunkSize;
        Status:= deflate(strm, Z_NO_FLUSH);   // same state, no member break
        Produced:= ChunkSize- strm.avail_out;
        If (Produced> 0) Then
          Target.WriteBuffer(Output[0], Produced);
        If (Status<> Z_OK) Then
          Break;
      End;
    End;
  Until (ReadCount< ChunkSize);
  Repeat                                    // one trailer for the whole input
    strm.next_out:= Pointer(Output);
    strm.avail_out:= ChunkSize;
    Status:= deflate(strm, Z_FINISH);
    Produced:= ChunkSize- strm.avail_out;
    If (Produced> 0) Then
      Target.WriteBuffer(Output[0], Produced);
  Until (Status= Z_STREAM_END);
Finally
  deflateEnd(strm);
End;

InflateStream got the symmetric rewrite: one inflateInit2, an inner loop that keeps calling inflate until the current chunk is consumed and the output buffer is no longer full, and a stop on Z_STREAM_END. There is one side effect worth knowing about. The old chunked writer hardcoded level 6, while the new one honours PLDeflateLevel, so a level set through TPDFlib.SetCompressionLevel(1..9) now applies to large embedded files on FPC as well. That matters if you already tune compression for archival output as described in reducing PDF file size in Delphi

How do you verify that a Flate stream is a single zlib member?

Inflate it with a plain zlib decoder and check two things when it returns Z_STREAM_END: the decoded length equals the source length, and avail_in is zero. Leftover input after the end marker is the signature of a concatenated stream. The fix was verified this way: 200 KB of test data, which spans four 64 KB chunks, went through the new DeflateStream and came out as a 534-byte single zlib stream, a stock zlib decoder recovered all 200,000 bytes with no remaining input, and the same check passed on the i386 cross-compiled FPC target. The routine below is the FPC version of that check, built directly on paszlib so it does not trust the code under test

uses Classes, SysUtils, paszlib, PDFlibZLib;

function IsSingleZlibMember(Packed: TMemoryStream; out Decoded: Int64): Boolean;
var
  strm: TZStream;
  Buf: array[0..65535] of Byte;
  Status: Integer;
begin
  Result:= False;
  Decoded:= 0;
  FillChar(strm, SizeOf(strm), 0);
  if inflateInit2(strm, 15) <> Z_OK then
    Exit;
  try
    strm.next_in:= Packed.Memory;
    strm.avail_in:= Cardinal(Packed.Size);
    repeat
      strm.next_out:= @Buf;
      strm.avail_out:= SizeOf(Buf);
      Status:= inflate(strm, Z_NO_FLUSH);
    until Status <> Z_OK;
    Decoded:= strm.total_out;
    // One member ends exactly at the last input byte
    Result:= (Status = Z_STREAM_END) and (strm.avail_in = 0);
  finally
    inflateEnd(strm);
  end;
end;

// Push 200,000 bytes through DeflateStream with the default 64 KB chunk
// and require one member that decodes back to the full length
Source.Position:= 0;
DeflateStream(Source, Packed);
if not (IsSingleZlibMember(Packed, Decoded) and (Decoded = Source.Size)) then
  raise Exception.Create('DeflateStream produced more than one zlib member');

At the application level, the useful assertion is the one the library does not make for you: compare what comes out of an attachment with the /Params /Size recorded when it went in. GetEmbeddedFileIntProperty with tag 5 returns that recorded size, embedded-file indexes are 1-based, and the payload needs to be at least 1 MiB to exercise the streaming path. Run the same test on every compiler you ship with, since the original defect passed on Delphi and failed only on FPC

procedure CheckLargeAttachmentRoundTrip(const PayloadFile, OutFile: string);
var
  PDF: TPDFlib;
  Extracted: TMemoryStream;
  I, Declared: Integer;
begin
  PDF:= TPDFlib.Create;
  try
    PDF.NewDocument;
    PDF.AddStandardFont(4);
    PDF.DrawText(80, 100, 'Large attachment round trip');
    // 1 MiB or more takes the chunked DeflateStream path in ReadFromStream
    if PDF.EmbedFile('Payload', PayloadFile, 'application/octet-stream') <> 1 then
      raise Exception.Create('EmbedFile failed');
    if PDF.SaveToFile(OutFile) <> 1 then
      raise Exception.Create('SaveToFile failed');
  finally
    PDF.Free;
  end;

  PDF:= TPDFlib.Create;
  Extracted:= TMemoryStream.Create;
  try
    if PDF.LoadFromFile(OutFile, '') = 0 then
      raise Exception.Create('LoadFromFile failed');
    for I:= 1 to PDF.EmbeddedFileCount do
    begin
      Extracted.Clear;
      if PDF.GetEmbeddedFileContentToStream(I, Extracted) <> 1 then
        raise Exception.CreateFmt('Attachment %d could not be decoded', [I]);
      Declared:= PDF.GetEmbeddedFileIntProperty(I, 5);   // /Params /Size
      if Extracted.Size <> Declared then
        raise Exception.CreateFmt('Attachment %d truncated: %d of %d bytes',
          [I, Extracted.Size, Declared]);
    end;
  finally
    Extracted.Free;
    PDF.Free;
  end;
end;

What does the fix not change?

The FPC branch keeps its lenient decoding rules, and it does not repair files that earlier FPC builds already wrote. When MaxOutput is reached, FPC InflateStream truncates at the limit and returns, while the Delphi branch raises ERangeError, and FPC still accepts partially decoded output when inflate reports a data error, because some PDF producers emit truncated or checksum-broken streams. A PDF written by a pre-v3.539.24 FPC build still contains concatenated members, and the corrected reader, like every other reader, stops at the first Z_STREAM_END. Do not try to heal such a file by decoding and re-encoding the stream inside the library, since that only makes the 64 KB truncation permanent. Re-embed the attachment from its original source instead. The FPC loop also still ends on the first read that returns less than a full chunk, which is only an end-of-data signal for streams such as TFileStream and TMemoryStream, so a custom TStream passed to AddAssociatedFileFromStream is safest copied into a TMemoryStream first. The Delphi branch, the DeflateStr and InflateStr helpers, and every stream smaller than 1 MiB on the plain Flate path behave exactly as before

The corrected FPC DeflateStream and InflateStream ship in v3.539.24 of PDF Library for Delphi, which targets Delphi, C++Builder, and Free Pascal from one source tree, and where a large attachment should now come back from an FPC build byte for byte, the same way it always did from Delphi