Why Signed AES-256 PDFs Load Slowly in Delphi: HotPDF Fixes

HotPDF loads signed, AES-256 encrypted PDFs in Delphi much faster than it used to, because three separate costs were removed from the load path: unbuffered token parsing of the source stream, a second run of the ISO 32000-2 Algorithm 2.A password check during decryption, and a slow inner AES-128-CBC loop inside the R6 hardened hash. On Delphi targets the R6 fix alone took a document load from about 66 ms to 16 ms

None of these showed up on the small test files most suites use. An unsigned, unencrypted sample of one or two kilobytes opens in a few hundredths of a second whatever the parser does. The trouble starts with the documents customers actually send: a contract with an embedded CMS signature, around 35 KB, protected with a revision 6 password. That combination switches on every expensive code path at once, and each of them had its own reason for being slower than it needed to be. This article goes through them in the order they were found, together with the cache-key mistakes that came up when signature verification results started being memoised

Why does a 34 KB signed PDF take 170 ms to load?

A 34 KB signed PDF took 170 ms to load because the parser was reading a bare THandleStream directly, and every token byte turned into its own ReadFile system call. A PDF tokenizer advances one byte at a time whilst it looks for delimiters, so an unbuffered handle stream converts a cheap in-memory scan into thousands of kernel transitions

The details explain why the bug survived. THotPDF.LoadFromFile has wrapped its file source in an internal read buffer for a long time, and the AutoDecrypt probe had already been fixed for the same reason after it spent 1.4 s re-querying the size of a 56 MB file. Signed and encrypted documents take a different route, though. To make sure lazy object reads and signature inspection always see the same immutable revision, even after the caller frees its own stream, THotPDF.LoadFromStream takes a snapshot of the source and parses it a second time. That second parse, along with the reuse load used for authenticated revisions, read the snapshot handle with no buffer in front of it. Both now go through the same internal buffered reader that the file path uses

That buffer is internal, so you never create it yourself. What you control is the stream you hand to LoadFromStream, and HotPDF reads that stream as-is during the first structural pass. If the data is already in memory, pass a TMemoryStream. If it is on disk, call LoadFromFile, or on Delphi 10.1 Berlin and later give the loader a TBufferedFileStream rather than a raw TFileStream

uses
  System.SysUtils, System.Classes, HPDFDoc;

function LoadEncryptedFromBytes(const FileName: string;
  const Password: AnsiString): THotPDF;
var
  Mem: TMemoryStream;
begin
  Result := THotPDF.Create(nil);
  Mem := TMemoryStream.Create;
  try
    Mem.LoadFromFile(FileName);   // one bulk read instead of per-token reads
    Mem.Position := 0;
    // LoadFromStream returns the page count, 0 when the document did not load
    if Result.LoadFromStream(Mem, Password) < 1 then
      raise Exception.Create('Could not load ' + FileName);
  except
    Result.Free;
    Mem.Free;
    raise;
  end;
  Mem.Free;
end;

Freeing the memory stream right after the load is safe for signed and encrypted input, because the component already holds its own snapshot of the bytes. Keep the original file or buffer anyway if you plan to verify signatures afterwards, since verification hashes the raw /ByteRange segments of the file rather than the decrypted object model

Measuring load and verification as separate phases

Load time and signature verification time should be measured separately, because they fail in different ways and are fixed in different places. The load phase covers parsing, the password check and decryption. The verification phase rehashes the signed byte ranges and checks the CMS signature. If you fold them into one number, a regression in either one hides inside the other

The RTL TStopwatch from System.Diagnostics is enough for this. The harness below loads with the password overload of THotPDF.LoadFromFile and then verifies the first signature through the TStream overload of VerifyLoadedSignature. The stream overload is the right choice for an encrypted file, because the password load path does not keep a source file name for the parameterless overloads to reopen

uses
  System.SysUtils, System.Classes, System.Diagnostics,
  HPDFDoc, HPDFSignVerify;

procedure TimeSignedEncryptedLoad(const FileName: string;
  const Password: AnsiString);
var
  Pdf: THotPDF;
  Src: TFileStream;
  Info: THPDFSignatureInfo;
  Status: THPDFSignatureVerifyStatus;
  Watch: TStopwatch;
  Pages: Integer;
begin
  Pdf := THotPDF.Create(nil);
  try
    Watch := TStopwatch.StartNew;
    Pages := Pdf.LoadFromFile(FileName, Password);
    Writeln(Format('Load + decrypt: %d ms, %d pages, encrypted=%s',
      [Watch.ElapsedMilliseconds, Pages, BoolToStr(Pdf.IsEncrypted, True)]));
    if (Pages < 1) or (Pdf.GetLoadedSignatureFieldCount = 0) then
      Exit;

    Src := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
    try
      Watch := TStopwatch.StartNew;
      Status := Pdf.VerifyLoadedSignature(0, Src, Info);
      Writeln(Format('Verify signature 0: %d ms, status %d, signer %s',
        [Watch.ElapsedMilliseconds, Ord(Status), string(Info.SignerName)]));
    finally
      Src.Free;
    end;
  finally
    Pdf.Free;
  end;
end;

Run it several times and ignore the first run. The first call pays one-time setup costs, including building AES tables and running their self-test, and disk caching skews the first read as well. What you are looking for is the steady-state number for a real customer file, measured against an unsigned, unencrypted copy of the same document

Why was the AES-256 password derived twice?

HotPDF derived the AES-256 file key twice because two stages of the same load each ran the full password check on their own. Before the structural parse, RecoverReloadKeysFromStreamRanges pre-scans the raw bytes so that encrypted object stream containers can be decrypted whilst filters are being decoded. That pre-scan already performs the ISO 32000-2 §7.6.4.3.3 Algorithm 2.A checks, validating the password against the user and owner salts and recovering the file encryption key. DecryptLoadedDocument then ignored that result and did the entire derivation again

For revision 4 and older, the duplicate cost little more than an MD5 loop. For revision 6, Algorithm 2.A calls the Algorithm 2.B hardened hash, so the duplicate doubled the most expensive part of opening the file. Reusing the recovered key is simple. Reusing it safely is the harder part, because a cached key that outlives its own password check turns into an authentication bypass

The fix limits the reuse with a scope flag that is true only between the pre-scan and the decrypt call inside the password overloads of LoadFromFile and LoadFromStream, and is cleared in a finally block. The flag is set only when the pre-scan actually validated the password. A direct call to DecryptLoadedDocument after a plain load always runs the full derivation, because that password was never pre-scanned. Without that restriction, a stale reload key from an earlier successful open could quietly decrypt the document for a later caller who supplied the wrong password, and that is a far worse bug than a slow load

Where does the R6 hardened hash spend its time?

The R6 hardened hash spends almost all of its time in AES-128-CBC encryption. ISO 32000-2 §7.6.4.3.4 Algorithm 2.B repeats a round at least 64 times. Each round builds a buffer of 64 copies of the password, the current hash and, on the owner side, the /U string, encrypts that buffer with AES-128-CBC using the first 16 bytes of the hash as the key and the next 16 as the IV, and picks SHA-256, SHA-384 or SHA-512 for the next hash from the first 16 bytes of the ciphertext read as a number modulo 3. The hashes are short. The buffers being encrypted are several kilobytes per round

HotPDF's implementation of that loop, PDF20HashDanceR6, used to encrypt the buffer one block at a time through a byte-oriented reference AES. It now first tries a native no-padding CBC routine, which on Delphi targets reaches the linked C AES backend. With that change a full document load dropped from about 66 ms to 16 ms. If the native routine is unavailable or returns a length that does not match, the code falls back to the historical block loop, and that loop can in turn use a table-driven AES-128 block whose T-tables are generated from the existing S-box. The cipher is the same in every case, so the derived key is byte-identical regardless of which path computed it

This article is about speed. Getting the Algorithm 2.B termination rule right is a separate correctness problem, and an off-by-one there rejects correct passwords only some of the time. The configuration side of R5 versus R6 is covered in the guide to AES-256 PDF encryption setup with HotPDF

Four ways a fast AES block can lie to you

A table-driven AES block can produce wrong ciphertext and still pass its own self-test. Every one of the four traps below came up whilst building the fast path, and two of them got past the self-test as it was first designed

• Self-test re-entrancy. The readiness routine sets its initialized flag before running the self-test, so a re-entrant call made during the test takes the reference path. The first version of the test therefore compared the reference implementation with itself. The fix is an unguarded raw fast block that the self-test calls directly, so the table path is the code that actually gets tested
• Swapped table layouts. Exchanging the byte order of two of the four round tables (Te1 and Te3) produces wrong ciphertext that the re-entrant self-test had no way to notice. What caught it was a standalone probe program that printed the FIPS-197 Appendix C.1 ciphertext 69c4e0d86a7b0430d8cdb78070b4c55a for key 000102…0f and plaintext 00112233…ff. The self-test now requires both that vector and byte equality with the reference block before it trusts the fast path
• FPC range and overflow checks. Under Free Pascal with -Cr and -Co, Delphi-style explicit Byte() conversions and byte-width shift arithmetic raise range or overflow exceptions. Table indices and the GF(2^8) doubling are now computed at native integer width and masked with and $FF
• FPC inliner miscompilation. Small big-endian load and store helpers that take PByte parameters were compiled into access violations when marked inline under FPC. Removing inline fixed them, and the cost of the call is negligible next to the table lookups

The same experience is behind the final wiring. Routing every AES-128 block in the library through the fast tables caused a deterministic FPC access violation during one publication-candidate load, so that change was reverted. The shipped design is conservative: native CBC first, an explicit fast-table branch inside the R6 loop, and the reference block everywhere else. The general lesson is that new cryptographic code that must build under several compilers needs an independent known-answer probe run on each target compiler. A DLL-level test failure cannot tell you whether you have a wrong table or a crash. For more on the Free Pascal side, see the notes on HotPDF Deflate and AES backends under Free Pascal

How do you cache signature verification without caching the wrong answer?

A signature verification cache is only safe if its key covers every input that can change the verdict, including inputs that live outside the signed byte ranges. HotPDF's headless document layer memoises CMS verification results by hashing the CMS bytes, the bytes of the signature dictionary in the /Contents gap, the contents of the covered ranges, a policy fingerprint, the verification budgets, the total source size and a one-minute time bucket. Three of those inputs were added only after a bug showed they were missing

Trusted certificate files are keyed by content, not by path. The operation layer writes the same PEM material to a fresh temporary file name for every command, so a path-based key never matched and the cache never hit. The total source size had to be part of the key because the reported UnsignedTrailingBytes depends on it: a base revision and a later incremental update can share exactly the same covered content whilst having different trailing lengths, and a cache keyed only on the ranges would hand the base revision's verdict to the updated file. The background on what those trailing bytes mean is in the HotPDF guide to verifying PDF digital signatures in Delphi

The third bug was a Pascal language trap, not a design mistake. Hash routines take an untyped const buffer, and passing a dynamic array variable to an untyped parameter passes the address of the array reference, not the array data

function Checksum(const Data; Count: Integer): Cardinal;
var
  P: PByte;
  I: Integer;
begin
  Result := 0;
  P := @Data;
  for I := 0 to Count - 1 do
  begin
    Result := Result + P^;
    Inc(P);
  end;
end;

procedure DigestCMS(const CMS: TBytes);
begin
  // Wrong: reads the array reference and whatever follows it on the stack.
  // A 16 KB read from there runs off the stack page and raises an access
  // violation, and Delphi and FPC fail at different points.
  Writeln(Checksum(CMS, Length(CMS)));

  // Right: pass the first element, guarded for the empty array
  if Length(CMS) > 0 then
    Writeln(Checksum(CMS[0], Length(CMS)));
end;

The cache lock is SyncObjs.TCriticalSection rather than the Windows InitializeCriticalSection API, because the same unit also builds under FPC on Linux, where that API does not exist

What stays slow on purpose

The R6 key derivation itself is never cached across loads, and that is deliberate. External bytes and an explicitly supplied password always go through the real authentication path, so every password load of an R6 document pays for at least one Algorithm 2.B hash, and both a user-side and an owner-side hash when the owner password is tried. The scoped reuse described above removes the duplicate inside a single load and nothing beyond that

• The 66 ms to 16 ms figure applies to Delphi targets with the native AES backend linked in. FPC builds take the table-driven block, and FPC code generation on Windows ran 2 to 4 times slower than Delphi under the same assertions
• The verification cache binds to a one-minute time bucket, so repeated checks of the same bytes within a short window are cheap, whilst a cached verdict never outlives the minute in which it was computed
• A merged-revision authority cache was implemented, debugged and then removed. It did not close the FPC gap and made Delphi security callbacks measurably slower, which is a good reminder to measure before keeping a cache
• Signature verification needs the original encrypted file bytes, so keep the source available. Faster decryption does nothing to remove that requirement

FAQ

Does reusing the pre-scanned key weaken password checking?

No. HotPDF reuses the recovered file key only inside the same password-overload call that validated the password with the full Algorithm 2.A checks, and the scope flag is cleared in a finally block. A standalone DecryptLoadedDocument call always performs the complete derivation, so a key left over from an earlier open cannot be borrowed with a wrong password

Why not cache the R6 hash result between loads?

Caching the hash between loads would mean accepting a password without checking it against the file being opened. HotPDF treats every external byte source and every explicit password as untrusted input that has to authenticate on its own, so the measurable speedup came from making Algorithm 2.B faster rather than from skipping it

Should I pass a TFileStream to LoadFromStream?

Prefer LoadFromFile for files on disk, because it already buffers internally. When you need the stream overload, pass a TMemoryStream or a TBufferedFileStream, since HotPDF reads the stream you supply directly during the first structural pass and a raw handle stream costs a system call for each small read

Password-protected loading, AES-256 R6 encryption, CMS signature verification and the rest of the loaded-document API ship together in one native VCL library for Delphi and C++Builder. Feature details and supported IDE versions are on the HotPDF Component product page