PDF Certificate Encryption in Delphi: RSA-OAEP and ECDH

HotPDF encrypts a PDF for specific certificate holders through the ISO 32000 public-key security handler: EnablePubKeyEncryption takes a 20-byte random seed, and each recipient gets its own CMS envelope, built by AddPubKeyRecipientCertificate for RSA keys (RSA-OAEP key transport) or AddPubKeyAgreementRecipientWithSecret for elliptic-curve keys (ECDH on P-256, P-384, P-521, X25519 or X448). Nobody shares a password; whoever holds a matching private key opens the file

The use case is always some version of the same story. A quarterly audit pack goes to three external reviewers, legal wants each of them to read it, only one of them may print it, and nobody wants a password sitting in an email thread next to the attachment. Password encryption cannot express that. Certificate encryption can, because every recipient unlocks the document with a key they already hold, and every recipient can carry a different permission set inside their own envelope

How does certificate-based PDF encryption differ from a password?

A public-key encrypted PDF derives its file key from a random seed plus the exact bytes of every recipient envelope, not from anything a person types. The handler is described in ISO 32000-1 §7.6.4 (§7.6.5 in ISO 32000-2), and the envelopes are CMS EnvelopedData structures as defined in RFC 5652. HotPDF writes /Filter /Adobe.PubSec with /SubFilter /adbe.pkcs7.s5; for AES-256 that means /V 5 and a /DefaultCryptFilter entry under /CF with /CFM /AESV3, and the /Recipients array lives inside that crypt filter. Each envelope encrypts 24 bytes: the 20-byte seed followed by that recipient's 32-bit permission word. The /P value in the encryption dictionary is only a placeholder, because the real permissions travel inside each envelope. At load time a reader unwraps one envelope, recovers the seed, and hashes the seed together with every envelope in /Recipients order (SHA-256 for AES-256, SHA-1 for the older ciphers) to rebuild the file key. If you are still deciding between this model and ordinary passwords, the AES-256 password encryption and permission flags guide covers the other side of that trade-off

Writing RSA recipients with EnablePubKeyEncryption

For RSA certificates, call EnablePubKeyEncryption with aes256, then call AddPubKeyRecipientCertificate once per DER-encoded certificate before BeginDoc. The helper builds an RSAES-OAEP envelope in-process with THPDFRSAOAEPHash values for the OAEP digest and the MGF1 digest (rohSHA256, rohSHA384 or rohSHA512), and it encrypts the envelope content with AES-256-CBC

uses
  System.SysUtils, System.IOUtils, HPDFDoc, HPDFCrypt, HPDFRSA;

procedure WriteAuditPack(const OutFile: string);
var
  Pdf: THotPDF;
  Seed: AnsiString;
begin
  SetLength(Seed, 20);                      // exactly 20 bytes, even for AES-256
  AESGenerateRandomBytes(@Seed[1], Length(Seed));
  Pdf := THotPDF.Create(nil);
  try
    Pdf.AutoLaunch := False;
    Pdf.FileName := OutFile;
    Pdf.EnablePubKeyEncryption(Seed, aes256, True);   // default key type is aes128
    // Reviewer A may print; reviewer B may only read and extract
    Pdf.AddPubKeyRecipientCertificate(TFile.ReadAllBytes('reviewer-a.cer'),
      [prPrint, prPrint12bit, prExtractContent], rohSHA256, rohSHA256);
    Pdf.AddPubKeyRecipientCertificate(TFile.ReadAllBytes('reviewer-b.cer'),
      [prExtractContent]);
    Pdf.BeginDoc;
    Pdf.CurrentPage.SetFont('Arial', [], 11);
    Pdf.CurrentPage.TextOut(72, 720, 0, 'Q3 audit pack');
    Pdf.EndDoc;
  finally
    Pdf.Free;
  end;
end;

Three details in that listing are load-bearing. First, the seed length is fixed at 20 bytes for every key type, AES-256 included; EnablePubKeyEncryption raises on any other length. Second, EnablePubKeyEncryption defaults to aes128, and both certificate helpers refuse to run unless the key type is aes256, so forgetting the second argument gets you the exception "certificate envelopes require aes256". The legacy ciphers (k40, k128, aes128) still work, but only through AddPubKeyRecipient with an envelope you built elsewhere. Third, AES-256 public-key encryption is a PDF 2.0 feature, so HotPDF raises the document version to 2.0 automatically. With StrictVersionLock set on a lower version, EnablePubKeyEncryption returns without enabling anything, and the failure only appears on the next line as "call EnablePubKeyEncryption first". Switching encryption during an incremental update raises EInvalidOpException straight away

Adding ECDH recipients: P-256, P-384, P-521, X25519 and X448

For elliptic-curve certificates, AddPubKeyAgreementRecipientWithSecret writes a CMS key-agreement recipient (KeyAgreeRecipientInfo, the KARI structure from RFC 5753, with the X25519 and X448 profile from RFC 8418) and computes the ECDH shared secret in-process. You choose the curve with a THPDFPubKeyAgreementScheme value: pkasECDHP256, pkasECDHP384, pkasECDHP521, pkasX25519 or pkasX448. The scheme has to match the key in the certificate, or the call raises "Certificate key does not match the requested agreement scheme". Under the hood, each envelope gets a fresh random 32-byte UKM, a key-encryption key derived with the stdDH KDF (SHA-256 for P-256 and X25519, SHA-384 for P-384, SHA-512 for P-521 and X448), and an AES-256 key wrap as defined in RFC 3394. The shared secret itself comes from pure Pascal curve code, with no platform crypto provider involved; the pure Pascal NIST curve arithmetic article explains how that layer was built and verified. For the Montgomery curves the whole ephemeral key pair can be generated locally:

uses
  System.SysUtils, System.IOUtils, HPDFDoc, HPDFCrypt, HPDFPubSec,
  HPDFKeyAgreement;

procedure AddLegalRecipient(Pdf: THotPDF);
var
  Scalar, OriginatorPublic: TBytes;
begin
  // Fresh ephemeral scalar per envelope; clamping happens inside the ladder
  SetLength(Scalar, 32);
  AESGenerateRandomBytes(@Scalar[0], Length(Scalar));
  try
    OriginatorPublic := HPDFX25519PublicFromScalar(Scalar);
    Pdf.AddPubKeyAgreementRecipientWithSecret(
      TFile.ReadAllBytes('legal-x25519.cer'),
      [prPrint, prExtractContent], pkasX25519,
      OriginatorPublic, Scalar,
      []);   // OwnPublicPoint: only meaningful for the NIST curves
  finally
    HPDFSecureClearBytes(Scalar);
  end;
end;

The NIST curves need more from the caller. HotPDF ships public-key helpers only for X25519 and X448 (HPDFX25519PublicFromScalar, HPDFX448PublicFromScalar), so for P-256, P-384 and P-521 you generate the ephemeral key pair with your own tooling and pass a big-endian scalar of exactly the field size (32, 48 or 66 bytes) plus the matching uncompressed 0x04||X||Y point as OriginatorPublicKey. HotPDF validates the recipient point against the curve equation, but it cannot check that your originator public key actually belongs to your scalar. Mismatched halves still produce a perfectly well-formed envelope that no recipient can open, which is why a round-trip load belongs in your test suite, not just a file-size check

Why does the order of /Recipients matter?

The order of /Recipients matters because the file key is a digest over the seed and every envelope in array order, so writer and reader must hash the same bytes in the same sequence. HotPDF keeps envelopes in the order you add them and writes them unchanged, which means you can add recipients in any order you like, but nothing downstream may reorder, re-encode or "clean up" that array. Most of the real bugs in this area were a variation on that theme, where two sides hashed slightly different bytes:

• Storing dynamic arrays in a TList via Add keeps only a raw pointer while the reference count stays with the local variable. The next SetLength frees the buffer and may reuse it, so every slot ended up aliasing the last envelope and multi-recipient files derived the wrong key. The fix is to store an owned copy with List.Add(Pointer(System.Copy(Bytes)))
• Envelope unwrapping parses the DER in place, and the key-recovery pass originally hashed those same live arrays. The reader now snapshots pristine copies of every envelope before any unwrap touches them, and the digest runs over the snapshots
• Binary DER passed through a Unicode TStringList gets bytes at or above $80 re-encoded by the code page, so HotPDF stores envelopes as hex text internally
• Encrypted and binary strings must be written as hex strings. A literal string is subject to end-of-line normalisation, where CR, LF and CRLF all become a single LF (ISO 32000-1 §7.3.4.2), and that silently rewrites ciphertext. HotPDF emits every /Recipients entry as a hex string and exempts it from string encryption, since every reader needs the envelopes before it holds any key
• The first byte of a DER BIT STRING counts unused bits and must be zero for byte-aligned keys. Leaving it uninitialised after SetLength wrote whatever was on the stack, and a strict unwrapper rejected the originator key, so a file could occasionally fail to open with the very key it was written for
• When the same key still cannot decrypt, compare layer by layer: the file key, then the ciphertext prefix (the IV), then the object key, then the plaintext. The bug lives right after the first layer that disagrees

How do you open a certificate-encrypted PDF with a private key?

To open a certificate-encrypted PDF, register the private key material before calling LoadFromFile, because HotPDF recovers the file key during the structural pass. Assign an RSA or EC key parsed with HPDFParsePFX to PubSecKeyMaterial, add further RSA keys with AddPubSecKeyMaterial, and register raw ECDH scalars with AddPubSecAgreementKeyMaterial(CurveOID, PrivateScalar, OwnPublicPoint), using the HPDFOIDX25519, HPDFOIDX448, HPDFOIDECP256, HPDFOIDECP384 or HPDFOIDECP521 constants. The NIST curves require the recipient's own uncompressed public point; the Montgomery curves ignore it

uses
  System.SysUtils, System.IOUtils, HPDFDoc, HPDFPFX, HPDFKeyAgreement;

procedure OpenAuditPack(const LegalScalar: TBytes);
var
  Reader: THotPDF;
begin
  Reader := THotPDF.Create(nil);
  try
    Reader.AutoLaunch := False;
    Reader.PubSecKeyMaterial :=
      HPDFParsePFX(TFile.ReadAllBytes('reviewer-a.pfx'), 'pfx-password');
    Reader.AddPubSecAgreementKeyMaterial(HPDFOIDX25519, LegalScalar, nil);
    // Optional: pick the envelope directly instead of trying them all
    Reader.PubSecRecipientQuery :=
      function(Context: Pointer; RecipientCount: Integer): Integer
      begin
        Result := -1;   // -1 = try every envelope in order
      end;
    Reader.LoadFromFile('audit-pack.pdf', '');
    Writeln('Pages: ', Reader.GetLoadedPageCount);
  finally
    Reader.Free;
  end;
end;

Without a callback, HotPDF tries every envelope against every registered key: the primary key first, then each additional RSA key, then the EC material. PubSecRecipientQuery receives the envelope count and returns a zero-based index or -1, and an index outside the array raises an exception rather than being clamped. Note that AddPubSecKeyMaterial accepts only RSA material (it insists on a modulus and a private exponent), so EC keys belong in PubSecKeyMaterial or AddPubSecAgreementKeyMaterial. When no key unwraps any envelope, the recovery step returns without a file key instead of raising, so verify that the content you expect actually decrypted rather than trusting that the load call returned

What HotPDF does not guarantee

HotPDF guarantees that its own writer and reader agree byte for byte, and it builds envelopes that follow the CMS structures cited above. It does not guarantee that every PDF viewer opens every combination. Support for RSA-OAEP key transport and for X25519 or X448 recipients varies between readers and versions, and we have not published compatibility results for those combinations. If a document must open in a specific viewer, encrypt a test file for a test certificate of the same key type and open it there before you commit to a scheme. Permissions carried in the envelope remain policy that conforming software honours, exactly as they do under password encryption. Seed quality is your responsibility too: AESGenerateRandomBytes is there for that job, and HotPDF wipes its copy of the seed once the file key has been derived. If you also need a string, stream or attachment to use a different crypt filter, the crypt filter policy guide for StmF, StrF and EFF shows which filter names the public-key handler accepts

Certificate encryption, RSA-OAEP and ECDH recipient envelopes, and private-key loading all ship in the HotPDF Delphi PDF component, alongside password encryption, digital signatures and the rest of the ISO 32000 toolset for Delphi and C++Builder