Technical Article

Find PDF Memory Hogs: HotPDF Object Dependency Graphs

For a 400-page report that embeds one 2 MB font, charging each page for everything it references produces a strange total: the pages add up to 800 MB of font in a file that may be 30 MB on disk. Divide the font evenly instead and each page costs about 5 KB, so nothing looks expensive and the memory seems to come from nowhere. Neither number answers the real question, which is how many bytes would be freed if a given object went away

That figure has a name in heap analysis, retained size, and graph theory has a clean way to compute it. An object is billed to its dominator, the nearest object that every path from the root has to pass through on the way to it. Shared objects float up until they reach a node that really owns them, exclusive ones stay put, and the totals add up without double counting

Shallow size, retained size and the dominator rule in HotPDF

HotPDF computes this for a loaded document through THotPDF.BuildLoadedObjectDependencyGraph. The call returns one THPDFObjectDependencyNode per loaded indirect object and one THPDFObjectDependencyEdge per indirect reference, then runs a dominator analysis rooted at the document Catalog, the object named by the trailer's /Root entry (ISO 32000-1 §7.7.2). Each node carries two sizes. EstimatedShallowBytes is what the object costs on its own. EstimatedRetainedBytes is its own cost plus the shallow cost of every object it dominates

Back to the font. Every page references it, so no single page dominates it; the paths only converge at the page tree node above those pages, which in a flat tree is the root /Pages object. The font's 2 MB lands there and then rolls up into the Catalog. An image XObject drawn on page 12 and nowhere else is the opposite case: every path to it runs through page 12, so page 12's retained size includes the whole image. The top of the retained list is therefore a list of objects whose removal would free memory

HotPDF: Dependency graph where a dominator attributes retained bytes: an image XObject kept by one page bills that page while a font shared by every page bills the Catalog
The dominator rule bills an image XObject kept alive by a single page to that page, while a font every page references rolls up to the Catalog — retained totals therefore sum without double counting

The same arithmetic gives one cross-check for free. THPDFObjectDependencyGraphInfo.CatalogRetainedBytes equals the summed shallow size of every object reachable from the Catalog, and TotalEstimatedBytes is the summed shallow size of every loaded object, reachable or not. The gap between the two is memory held by objects the document no longer points at

Running the graph and reading a failed result

The method returns True only when the whole analysis finished, and that return value is the thing to check. A False result still fills the info record, but the arrays are not something to report from. When the object count exceeds MaxObjects (default 250,000), the method stops before building any nodes and sets ObjectCount to one more than the limit, so you learn that the limit was too small, not how large the document is. When the edge budget (default 2,000,000) or the 256-level nesting limit for direct objects is hit during the scan, nodes come back with partial shallow sizes and no retained sizes or reachability at all

uses
  SysUtils, HPDFDoc;

procedure ProfileLoadedPdf(const FileName: string);
var
  Pdf: THotPDF;
  Nodes: THPDFObjectDependencyNodeArray;
  Edges: THPDFObjectDependencyEdgeArray;
  Info: THPDFObjectDependencyGraphInfo;
begin
  Pdf := THotPDF.Create(nil);
  try
    // LoadFromFile returns the page count; 0 means the load failed
    if Pdf.LoadFromFile(FileName) = 0 then
      Exit;

    // Raised limits for an offline forensic run
    if not Pdf.BuildLoadedObjectDependencyGraph(Nodes, Edges, Info,
      1000000, 8000000) then
    begin
      // Log is your own logging routine
      Log(Format('graph incomplete: %s (limit %s, invalid %s, objects %d)',
        [string(Info.Diagnostic), BoolToStr(Info.LimitExceeded, True),
         BoolToStr(Info.InvalidObjectGraph, True), Info.ObjectCount]));
      Exit; // do not rank or sum a partial node array
    end;

    Log(Format('%d objects, %d edges (%d unresolved), %d reachable',
      [Info.ObjectCount, Info.EdgeCount, Info.UnresolvedReferenceCount,
       Info.ReachableObjectCount]));
    Log(Format('estimated %d bytes, Catalog retains %d, unreachable %d',
      [Info.TotalEstimatedBytes, Info.CatalogRetainedBytes,
       Info.TotalEstimatedBytes - Info.CatalogRetainedBytes]));

    LogTopRetained(Nodes, Edges, 10);
  finally
    Pdf.Free;
  end;
end;

Two other flags deserve a look in that failure branch. InvalidObjectGraph is set when the Catalog object cannot be found among the loaded objects, or when a direct object turns out to contain itself, which only happens in malformed input. LimitExceeded also covers a dominator analysis that has not settled after 128 passes. Diagnostic explains each case in one line. Passing zero or a negative value for MaxObjects, or a negative MaxEdges, raises EArgumentOutOfRangeException instead of returning False

When the call fails on a limit, throw the arrays away and run it again with larger bounds. Partial shallow sizes look plausible in a log and are wrong in exactly the way that sends people after the wrong object. The bounds exist because the graph has a cost of its own: every edge stores a Path string, so an eight-million-edge run allocates eight million small strings on top of the document it is measuring. It belongs on a diagnostic path someone triggers on purpose

What does EstimatedShallowBytes actually count?

EstimatedShallowBytes estimates resident memory, not bytes in the file. For each object it adds the instance size of the parsed object and its direct children, the slots held by dictionary and array lists, the length of every key, name and string, and the payload of a stream if that payload is in RAM: a memory stream, or a lazily loaded stream that has already been read into memory. A stream whose bytes still sit in the source file, or that has been spilled to a temporary file, contributes only its object overhead

That has a practical consequence. A scan-heavy file profiled straight after loading can show modest retained sizes for its images, because nothing has decoded them yet. Run the graph after the step that made the process grow (rendering, text extraction, a merge) and the materialized streams show up with their real weight. For a reproducible check, profile the same file twice, once right after LoadFromFile and once after the expensive operation, and compare the top entries

ObjectType is coarse. It reports the PDF value type, otStream or otDictionary and so on, so an image, a font program and a page content stream all read as otStream. The incoming edge is where the role shows up. THPDFObjectDependencyEdge.Path records where in the source object the reference was found, starting at $, with /Key for dictionary entries, [n] for array items and /dictionary for the dictionary of a stream. A path of $/Resources/XObject/Im3 from a page, or $/XObject/Im3 from a separate resource dictionary, means the object is used as an XObject named Im3; $/FontFile2 coming from a font descriptor means an embedded TrueType program; $/Contents[0] means page content

// Your own helper, not part of the library: ranks the top Count nodes by
// retained size and prints the path of one incoming reference for each
procedure LogTopRetained(const Nodes: THPDFObjectDependencyNodeArray;
  const Edges: THPDFObjectDependencyEdgeArray; Count: Integer);
var
  Taken: array of Boolean;
  I, J, Best: Integer;
  Via: AnsiString;
begin
  SetLength(Taken, Length(Nodes));
  for I := 1 to Count do
  begin
    Best := -1;
    for J := 0 to High(Nodes) do
      if (not Taken[J]) and ((Best < 0) or
         (Nodes[J].EstimatedRetainedBytes >
          Nodes[Best].EstimatedRetainedBytes)) then
        Best := J;
    if Best < 0 then
      Break;
    Taken[Best] := True;

    Via := '(no incoming reference)';
    for J := 0 to High(Edges) do
      if Edges[J].Resolved and
         (Edges[J].TargetObjectNumber = Nodes[Best].ObjectNumber) and
         (Edges[J].TargetGenerationNumber = Nodes[Best].GenerationNumber) then
      begin
        Via := Edges[J].Path;
        Break;
      end;

    Log(Format('%d %d obj  shallow %d  retained %d  refs in %d  via %s',
      [Nodes[Best].ObjectNumber, Nodes[Best].GenerationNumber,
       Nodes[Best].EstimatedShallowBytes, Nodes[Best].EstimatedRetainedBytes,
       Nodes[Best].IncomingReferenceCount, string(Via)]));
  end;
end;

One incoming path is enough to classify most objects. For a shared object, IncomingReferenceCount tells you how many references there are, and ImmediateDominatorObjectNumber with its generation number tells you which object owns it after the dominator analysis. That field is -1 for the Catalog and for every object the Catalog cannot reach. An edge whose Resolved flag is False points at an object number that is not loaded; those are counted in UnresolvedReferenceCount, and a high count on an otherwise healthy file usually means a damaged cross-reference section

Where do unreachable objects come from?

A node with ReachableFromCatalog set to False is loaded and costs memory, but no reference chain from the Catalog leads to it, so nothing in the document structure leads a viewer to it. Its retained size equals its shallow size, because the dominator analysis only runs over reachable objects. Summing EstimatedRetainedBytes across unreachable nodes therefore gives the same answer as TotalEstimatedBytes - CatalogRetainedBytes

Incremental updates are a common source, though not in the way people tend to assume. When a later revision redefines object 2120, the newest cross-reference entry wins and the older definition is never loaded, so superseded copies of the same object number do not appear in the graph. What does appear is an object that a later revision simply stopped referencing: an annotation dropped from a rewritten /Annots array, an image replaced by a new object number, a form field removed from the AcroForm. Nothing in ISO 32000-1 §7.5.6 obliges an update to mark those entries free, and as long as the cross-reference data lists them as in use the loader treats them as live objects. The article on object streams and incremental updates covers how revisions stack

HotPDF: Incremental update that rewires a reference to a new object and leaves the previously referenced object loaded but unreachable from the Catalog
An update that points the document at a new object leaves the previously referenced object listed and loaded but outside the Catalog's reach, and its bytes are still counted in TotalEstimatedBytes

The other two sources are producers that write objects and then fail to link them, and damaged files whose cross-reference data had to be rebuilt by scanning, which can pick up object definitions nothing references. When a large share of the estimated total sits in unreachable nodes, a clean rewrite of the file beats appending yet another revision, and the graph gives you the byte count to justify the slower save

Is the parser the one holding memory?

Before blaming the document, check how loading itself behaved. Two calls describe the parser rather than the content, and both reset at the start of every LoadFromFile or LoadFromStream, so they always describe the most recent load

GetLastParserArenaStatistics fills a THPDFParserArenaStatistics record for the block arena that holds short-lived tokens and buffers during parsing. Blocks are 64 KiB (BlockSize), and an allocation larger than a block gets a block of its own size. AllocationCount minus ReusedAllocationCount equals BlockAllocationCount, the number of times the arena had to ask the memory manager for a new block. Blocks are kept for the lifetime of the THotPDF instance and freed when it is destroyed. If one instance processes a whole batch, RetainedBytes roughly reflects the largest working set any file in that batch needed, which explains a plateau that lasts until the component is freed. TemporaryObjectElisionCount counts dictionary keys read straight into the arena instead of being built as temporary name objects

GetDocumentStringInternStatistics describes the per-document pool that deduplicates PDF names, content-stream operators and short strings. Strings of 1 to 64 bytes are eligible. The pool holds at most 65,536 entries and 4 MiB, and those caps come back in the record as MaxStringLength, MaxEntryCount and MaxRetainedBytes. A string that is empty, longer than 64 bytes, or new when the pool is full is counted in BypassCount and kept as an ordinary string. A file that generates huge numbers of unique names therefore costs roughly what it would cost without the pool, and no more

var
  Arena: THPDFParserArenaStatistics;
  Intern: THPDFDocumentStringInternStatistics;
begin
  if Pdf.GetLastParserArenaStatistics(Arena) then
    Log(Format('arena: %d blocks, %d bytes held, peak %d, ' +
      '%d new blocks for %d allocations, %d tokens',
      [Arena.RetainedBlockCount, Arena.RetainedBytes, Arena.PeakUsedBytes,
       Arena.BlockAllocationCount, Arena.AllocationCount, Arena.TokenCount]));

  if Pdf.GetDocumentStringInternStatistics(Intern) then
  begin
    Log(Format('intern: %d of %d requests hit, %d bypassed, %d bytes reused',
      [Intern.HitCount, Intern.RequestCount, Intern.BypassCount,
       Intern.ReusedBytes]));
    // Your own check, not a library call: flags a pool that hit its caps
    if (Intern.EntryCount >= Intern.MaxEntryCount) or
       (Intern.RetainedBytes + Intern.MaxStringLength >
        Intern.MaxRetainedBytes) then
      Log('intern pool at its cap: later strings were not deduplicated');
  end;
end;
HotPDF: Parser arena and string intern pool statistics with allocation reuse and a capped bypass counter
The arena record shows how much parser traffic reused recycled blocks while the intern record shows the pool earning its keep before its caps force bypasses — high reuse with a low bypass count clears the loader of suspicion

Typical PDFs repeat a small vocabulary of names and operators, so HitCount sits close to RequestCount and ReusedBytes grows with page count. A BypassCount that approaches RequestCount on a modest file is unusual. On an intake path for untrusted uploads, log it

A triage order for a memory complaint

Start with CatalogRetainedBytes. If it is close to the growth you measured in the process, the memory is in the document and the top of the retained list will name it. If it is far below, the memory lives somewhere the graph does not measure, such as your own caches, rendered bitmaps or the parser arena, and the arena record is the next thing to read

  1. Rank nodes by EstimatedRetainedBytes and read the incoming Path of the top ten. XObject targets at the top usually point to scan-heavy content where downsampling pays off; /FontFile, /FontFile2 or /FontFile3 targets point to full font programs embedded where subsets would do; large /Contents streams usually come from generated vector output such as maps or CAD exports
  2. Subtract CatalogRetainedBytes from TotalEstimatedBytes. A large remainder is stranded objects, so the fix belongs in the save step
  3. Read the arena and intern records last. They rarely explain gigabytes, but they do explain a process that stays large after every document is closed

Run those steps in that order. The first one settles most complaints, and for very large documents the structural answer is often to stop holding the whole object graph at once, which is what the direct file API workflow is built for

All three diagnostics are ordinary method calls returning plain records, so they slot into whatever logging or telemetry a Delphi or C++Builder application already has. HotPDF ships as a native VCL component with full source; documentation and a trial download are on the HotPDF Delphi PDF component page