PDF 1.5 object streams pack many small indirect objects into one Flate-compressed container, and losLab PDF Library emits them on a full save through its PackObjectStreams flag. The gain is real: hundreds of page, font and annotation dictionaries that each cost dozens of uncompressed bytes collapse into a handful of compressed blobs. The cost is that every packed object now needs a cross-reference stream to describe it
That second half is where writers break. Building an /ObjStm container is arithmetic; teaching the cross-reference machinery to point into it is a redesign. A writer that produces a perfectly valid container and then describes its members with ordinary type-1 offsets has produced a file Acrobat will open just long enough to declare damaged. The two features are one feature, and this article covers the write side of both, as defined in ISO 32000-1 §7.5.7 and §7.5.8
What an ObjStm container actually contains
An object stream is a stream whose decoded bytes are two concatenated regions, and ISO 32000-1 §7.5.7 gives the dictionary exactly three keys that matter for construction. /Type /ObjStm identifies it, /N gives the number of members, and /First gives the byte length of the header region — equivalently, the offset at which the body begins. The header is whitespace-separated pairs of object number and offset; the body is the members serialised back to back, with each offset measured from the start of the body rather than from the start of the decoded payload. Reading a fully decoded container makes it obvious: below, /First is 14 because the three header lines occupy fourteen bytes, and object 7 sits 55 bytes into the body because object 4 serialised to 54 characters plus a separator
// Decoded payload of: 12 0 obj << /Type /ObjStm /N 3 /First 14
// /Filter /FlateDecode /Length 118 >> stream
4 0
7 55
9 90
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
<< /Type /ExtGState /CA 1 /ca 1 >>
[ 0 0 595 842 ]Two membership rules are absolute and both come straight from §7.5.7. A stream object can never be a member, because a stream carries raw bytes that would have to be nested inside another stream. And a member must be a complete object value, never a bare indirect reference — a compressed object that is merely 5 0 R creates an indirection the reader cannot resolve without already knowing where it points. losLab PDF Library filters both cases out during candidate collection, along with the encryption dictionary and object 0, then packs whatever survives in groups of 200 per container. That cap is a random-access decision rather than a spec limit: a reader that wants one member has to inflate the entire container, so oversized containers make small lookups expensive
Why must ObjStm members use type-2 cross-reference entries?
Because a packed object has no file offset to record. ISO 32000-1 §7.5.8 answers this with three entry types in a binary cross-reference stream: type 0 for free objects, type 1 for ordinary in-use objects stored at a byte offset, and type 2 for compressed objects, whose two data fields hold the container object number and the member index inside it. There is no way to express a packed object in the classic plaintext xref table, which is precisely why PDF 1.5 introduced both features together
The ordering that follows trips up nearly every first implementation, including ours. Ordinary objects get type-1 entries. The /ObjStm containers themselves get type-1 entries, because a container is a perfectly normal indirect stream object written at a real offset. Only the members get type-2 entries. And the cross-reference stream is itself an indirect object in the file, so it needs its own type-1 entry pointing at the offset where it was just written — the same offset startxref records. An early version of our writer excluded container object numbers from the write loop instead of excluding members, and the result was a file with a cross-reference stream and no object streams at all: structurally coherent, semantically empty, rejected downstream. The /Size value hides a matching off-by-one, since it is the highest object number plus one and the cross-reference stream is allocated as the highest object number, so it must be counted too
Sizing the /W array: why four bytes is not enough
The /W array declares the byte width of each of the three fields, and losLab PDF Library writes it as /W [1 Field2 Field3] with field 1 fixed at one byte for the type code and field 3 fixed at two bytes, which covers generation numbers up to 65535 and member indices alike. Field 2 is the one that cannot be a constant, because it carries two unrelated quantities: in a type-1 entry it is a byte offset bounded only by file size, while in a type-2 entry it is a container object number and in a type-0 entry it is the next free object in the chain. A fixed four-byte field 2 works fine until the file crosses 4 GB, at which point every offset past the boundary silently truncates and the whole table becomes garbage. The writer therefore scans the assembled table for the largest value any field-2 slot will ever hold, including the offset of the cross-reference stream itself, and widens the field up to eight bytes
// Field 2 must hold the largest byte offset AND the largest
// ObjStm container number AND the largest free-chain target.
MaxField2Value := XRefStart;
for X := 0 to MaxObj do
begin
if XRefTable[X].InUse and (XRefTable[X].ObjStrNum > 0) then
Field2Value := XRefTable[X].ObjStrNum // type-2: container number
else
Field2Value := XRefTable[X].ObjPos; // type-1 offset / type-0 next-free
if Field2Value > MaxField2Value then
MaxField2Value := Field2Value;
end;
Field2 := 4;
while (Field2 < 8) and
(MaxField2Value > ((Int64(1) shl (Field2 * 8)) - 1)) do
Inc(Field2);
Field3 := 2; // generation numbers and member indices both fitOnce the widths are known the payload size is known exactly, so the writer preallocates the whole buffer and fills it by index; appending entries byte by byte to an AnsiString turns table construction quadratic, which nobody notices on a ten-page invoice and everybody notices on a document with two hundred thousand objects. Two further details keep strict readers happy. /Index declares which object-number ranges the table covers, and for a full rewrite that is simply [0 N] with no gaps. And every slot the writer did not actually emit must default to free rather than in-use: object 0 heads the free chain, each free slot links to the next, and a slot that once held a deleted object keeps its generation number incremented by one. The companion note on memory safety when parsing untrusted PDFs makes the same bounds argument from the read side
Why must the cross-reference stream never be encrypted?
Because a reader has to parse it before it can know how to decrypt anything. The cross-reference stream is what tells the reader where the /Encrypt dictionary lives; if its bytes were themselves encrypted, the reader would need the file key to find the object that describes the file key. losLab PDF Library enforces this in a single predicate: ShouldCryptStreamData returns False whenever the stream dictionary carries /Type /XRef, so the exemption holds no matter which path reaches the serialiser
The /ObjStm container gets the opposite treatment, and the asymmetry is deliberate. A container is encrypted whole, keyed on its own object number, exactly like any other stream. Its members are not encrypted individually — they are packed in their decrypted plaintext form, and the single pass over the assembled container covers them, strings included. Double-encrypting the members produces a file that decrypts to ciphertext, and because the outer layer succeeds the failure surfaces as a parse error deep in the object graph rather than as an authentication failure. One object then stays outside the scheme entirely: in an encrypted document the Catalog is kept as a direct type-1 object and never packed, because packing it would force the loader to inflate and decrypt an object stream in order to reach the document root, before the decryption context that the root helps establish is fully built
Turning packing on from Delphi
The public switch is PackObjectStreams, exposed as a field on TPDFlibSaveOptions, as the standalone setter SetPackObjectStreams, and as a property on the document object. It defaults to enabled and is auto-gated by version: the writer only packs when the document is already PDF 1.5 or later, and it calls the internal minimum-version guard so a packed document is bumped to 1.5 rather than mislabelled. After the save, GetLastSaveUsedObjectStreams reports whether the gate actually opened, which is the assertion you want in a regression test rather than a byte-size comparison
var
Doc: TPDFlib;
Options: TPDFlibSaveOptions;
begin
Doc := TPDFlib.Create;
try
if Doc.LoadFromFile('report.pdf', '') <= 0 then
Exit;
Doc.SetInformation(0, '1.5'); // packing is gated on PDF 1.5+
FillChar(Options, SizeOf(Options), 0);
Options.CompressContent := True;
Options.GarbageCollect := True; // drop orphans before packing
Options.PackObjectStreams := True;
if Doc.SaveToFileOptions('report-packed.pdf', Options) = 1 then
if Doc.GetLastSaveUsedObjectStreams = 1 then
Writeln('Saved with ObjStm containers and an xref stream');
finally
Doc.Free;
end;
end;Ordering matters between packing and garbage collection. Reachability analysis has to run first, because a member that survives into a container drags the container along with it — if a live object is packed, its container number is reachable by definition, and sweeping the container away strands the member with no way to locate it. Running the collector first also means dead objects never enter a container at all, which is where the compounding size win comes from. Packing complements the other size levers rather than replacing them; the walkthrough of PDF file size optimisation and font subsetting covers the levers that act on stream payloads, where object streams act on structure
Boundaries worth knowing before you enable it
Incremental saves never pack. An incremental update appends new objects and a new cross-reference section while leaving earlier revisions physically intact, so repacking existing objects into fresh containers would orphan the type-1 entries that the previous revision still references; losLab PDF Library disables packing whenever append mode is active, and the article on incremental updates and append-mode streaming covers that path in full. Documents below PDF 1.5 keep the plaintext cross-reference table unconditionally: a 1.4 consumer has no idea what /ObjStm means, and silently promoting a document because the writer preferred a smaller file would be the wrong trade to make on the caller's behalf. One optional key we deliberately do not emit is /Extends, which ISO 32000-1 §7.5.7 defines so a container can name a predecessor and readers can treat a chain of containers as a logical group. It is genuinely optional, every container we write is self-contained and independently decodable, and skipping it removes a class of cycle and dangling-reference bugs from the writer — though readers must of course still honour /Extends when they meet it in files from other producers
Object-stream packing and cross-reference stream output ship as part of losLab PDF Library for Delphi and C++Builder, alongside the garbage collector and content-stream optimiser they compose with; the product page carries the full save-options reference