PDF Certificate Encryption in Delphi: RSA-OAEP and ECDH

A certificate-encrypted PDF carries a single /P value in its encryption dictionary no matter who the file was written for; in the files built later in this article it is -64. Three recipients with three different permission sets still share that one number. Code that answers "may this user print?" from /P is reading a placeholder, because under the public-key security handler the real permission word travels inside each recipient's encrypted envelope and only appears once that recipient's private key has opened it

That design is the reason certificate encryption is worth the trouble. A quarterly audit pack goes to three external reviewers; legal wants all of them to read it, only one may print it, and nobody wants a password sitting in the email thread next to the attachment. A password can't express that. Certificates can: each recipient unlocks the file with a key they already hold, and each envelope carries its own permission word

How does certificate-based PDF encryption differ from a password?

A public-key encrypted PDF derives its file key from a random seed plus the exact bytes of every recipient envelope, and nothing a person types enters the calculation. The handler is specified in ISO 32000-1 §7.6.4 (§7.6.5 in ISO 32000-2), and the envelopes are CMS EnvelopedData structures from RFC 5652. HotPDF writes /Filter /Adobe.PubSec with /SubFilter /adbe.pkcs7.s5. For AES-256 that means /V 5, a /DefaultCryptFilter entry under /CF with /CFM /AESV3, and the /Recipients array placed inside that crypt filter instead of the top-level dictionary

Each envelope encrypts 24 bytes: the 20-byte seed followed by that recipient's 32-bit permission word. At load time a reader unwraps one envelope, recovers the seed, and hashes the seed together with every envelope in /Recipients order (SHA-256 for AES-256, SHA-1 for the older ciphers) to rebuild the file key. Reviewer B never learns what reviewer A may do, yet B's reader still hashes A's envelope bytes, so the whole array has to survive byte for byte. If you are still choosing between this model and ordinary passwords, the AES-256 password encryption and permission flags guide covers the other side of that trade-off

Writing RSA recipients with EnablePubKeyEncryption

For RSA certificates the sequence is short: EnablePubKeyEncryption with aes256, then one AddPubKeyRecipientCertificate call per DER-encoded certificate, all before BeginDoc. The helper builds an RSAES-OAEP envelope in-process. Its last two parameters pick the OAEP digest and the MGF1 digest from THPDFRSAOAEPHash (rohSHA256, rohSHA384 or rohSHA512), both defaulting to rohSHA256, and the envelope content is encrypted with AES-256-CBC

uses
  System.SysUtils, System.IOUtils, HPDFDoc, HPDFCrypt, HPDFRSA;

procedure WriteAuditPack(const OutFile: string);
var
  Pdf: THotPDF;
  Seed: AnsiString;
begin
  SetLength(Seed, 20);                      // exactly 20 bytes, even for AES-256
  AESGenerateRandomBytes(@Seed[1], Length(Seed));
  Pdf := THotPDF.Create(nil);
  try
    Pdf.AutoLaunch := False;
    Pdf.FileName := OutFile;
    Pdf.EnablePubKeyEncryption(Seed, aes256, True);   // default key type is aes128
    // Reviewer A may print; reviewer B may only read and extract
    Pdf.AddPubKeyRecipientCertificate(TFile.ReadAllBytes('reviewer-a.cer'),
      [prPrint, prPrint12bit, prExtractContent], rohSHA256, rohSHA256);
    Pdf.AddPubKeyRecipientCertificate(TFile.ReadAllBytes('reviewer-b.cer'),
      [prExtractContent]);
    Pdf.BeginDoc;
    Pdf.CurrentPage.SetFont('Arial', [], 11);
    Pdf.CurrentPage.TextOut(72, 720, 0, 'Q3 audit pack');
    Pdf.EndDoc;
  finally
    Pdf.Free;
  end;
end;

Three details in that listing carry weight. The seed is 20 bytes for every key type, AES-256 included, and EnablePubKeyEncryption raises on any other length. The key type defaults to aes128 while both certificate helpers insist on aes256, so dropping the second argument makes the first AddPubKeyRecipientCertificate call raise "certificate envelopes require aes256". The legacy ciphers (k40, k128, aes128) remain available through AddPubKeyRecipient, which takes a complete DER envelope built by other tooling

The third detail is the version. AES-256 under the public-key handler is a PDF 2.0 feature, and HotPDF raises the document version to 2.0 on its own. With StrictVersionLock set on a lower version that bump is not allowed: EnablePubKeyEncryption returns without switching anything on, and the error surfaces one line later when the recipient helper reports "call EnablePubKeyEncryption first". That message sends people hunting for a call-order mistake. Check the version lock instead. Changing encryption during an incremental update is refused outright with EInvalidOpException

Adding ECDH recipients on P-256, P-384, P-521, X25519 and X448

Elliptic-curve certificates go through AddPubKeyAgreementRecipientWithSecret, which writes a CMS key-agreement recipient (KeyAgreeRecipientInfo from RFC 5753, with the X25519 and X448 profile from RFC 8418) and computes the ECDH shared secret in-process. A THPDFPubKeyAgreementScheme value names the curve: pkasECDHP256, pkasECDHP384, pkasECDHP521, pkasX25519 or pkasX448. The scheme has to match the certificate's key algorithm, curve and key length, or the call raises "Certificate key does not match the requested agreement scheme"

Every envelope gets a fresh random 32-byte UKM, a key-encryption key from the stdDH KDF (SHA-256 for P-256 and X25519, SHA-384 for P-384, SHA-512 for P-521 and X448), and an AES-256 key wrap as defined in RFC 3394. The curve arithmetic is plain Pascal with no platform crypto provider underneath; the pure Pascal NIST curve arithmetic article walks through how that layer is put together. On the Montgomery curves the whole ephemeral key pair can be generated locally:

uses
  System.SysUtils, System.IOUtils, HPDFDoc, HPDFCrypt, HPDFPubSec,
  HPDFKeyAgreement;

procedure AddLegalRecipient(Pdf: THotPDF);
var
  Scalar, OriginatorPublic: TBytes;
begin
  // Fresh ephemeral scalar per envelope; clamping happens inside the ladder
  SetLength(Scalar, 32);
  AESGenerateRandomBytes(@Scalar[0], Length(Scalar));
  try
    OriginatorPublic := HPDFX25519PublicFromScalar(Scalar);
    Pdf.AddPubKeyAgreementRecipientWithSecret(
      TFile.ReadAllBytes('legal-x25519.cer'),
      [prPrint, prExtractContent], pkasX25519,
      OriginatorPublic, Scalar,
      []);   // OwnPublicPoint: only meaningful for the NIST curves
  finally
    HPDFSecureClearBytes(Scalar);
  end;
end;

The NIST curves ask more of the caller. HotPDF derives originator public keys only for X25519 and X448 (HPDFX25519PublicFromScalar, HPDFX448PublicFromScalar), so for P-256, P-384 and P-521 the ephemeral pair comes from your own tooling. Pass a big-endian scalar of exactly the field size (32, 48 or 66 bytes) and the matching uncompressed 0x04||X||Y point (65, 97 or 133 bytes) as OriginatorPublicKey, and hand the same point to OwnPublicPoint, the parameter reserved for the NIST schemes. Wrong lengths raise immediately. The recipient point taken from the certificate is checked against the curve equation before any multiplication runs

What no library can check is whether your originator public key belongs to your scalar. Mismatched halves still produce a well-formed envelope. The writer derives its key from scalar times recipient point, the recipient derives theirs from their private key times the wrong originator point, and the RFC 3394 integrity check on the recipient's side is the first thing to notice, by which time the file has shipped. A round-trip load against a test key on the same curve belongs in your build; a successful EndDoc and a plausible file size prove nothing here

That pairing step is why the curve choice matters in practice. If you control certificate issuance, X25519 is the easier curve to get right from Delphi: one 32-byte random scalar, one helper call, no point to keep in sync by hand. The NIST curves earn their place when the recipients' certificates come from a PKI that only issues P-256 or P-384, and in that case the round-trip load stops being optional

Why does the order of /Recipients matter?

The order of /Recipients matters because the file key is a digest over the seed followed by every envelope in array order, so writer and reader must hash identical bytes in an identical sequence. HotPDF keeps envelopes in the order you add them and writes them unchanged, which leaves you free to pick any order. What happens to the array afterwards is the risky part. Post-processing tools, signing pipelines and home-grown envelope builders are where it breaks, and the failures share one shape — two sides hashing slightly different bytes:

• Delphi's untyped TList stores a raw pointer. Collecting envelopes with List.Add(Pointer(Bytes)) leaves the reference count with the local variable, so the next SetLength on that variable frees or reuses the buffer and every slot can end up aliasing the last envelope. TList<TBytes> or a plain TArray<TBytes> keeps the references counted
• DER parsers that work in place modify the buffer they parse. Hash a pristine copy of each envelope, taken before any unwrap attempt touches it
• A Unicode TStringList is no place for binary DER: bytes at or above $80 get re-encoded by the code page on the way in. Keep envelopes in TBytes, or hex-encode them first
• Ciphertext belongs in hex strings. Inside a literal string, CR, LF and CRLF all read back as a single LF (ISO 32000-1 §7.3.4.2), so an envelope that happens to contain $0D $0A loses a byte. HotPDF writes every /Recipients entry as a hex string and leaves it out of string encryption, since a reader needs the envelopes before it holds any key
• The first content byte of a DER BIT STRING counts unused bits and must be $00 for byte-aligned keys. Hand-written DER that drops this byte and puts the 0x04||X||Y point straight into the content shifts the whole key by one byte, and a strict parser rejects the originator key outright
• When a key that should work still cannot decrypt, compare layer by layer, in this order: file key, the IV at the front of the ciphertext, per-object key, plaintext. The fault sits right after the first layer that disagrees

Loading a certificate-encrypted PDF with private keys

Private keys have to be registered before LoadFromFile runs, because HotPDF recovers the file key during the structural pass. RSA identities parsed with HPDFParsePFX go into PubSecKeyMaterial, and further RSA keys into AddPubSecKeyMaterial, which rejects material without a modulus and a private exponent. Both of those key slots are matched on the RSA modulus and private exponent, so EC keys, PFX-based ones included, take a third route: AddPubSecAgreementKeyMaterial(CurveOID, PrivateScalar, OwnPublicPoint), with one of the HPDFOIDX25519, HPDFOIDX448, HPDFOIDECP256, HPDFOIDECP384 or HPDFOIDECP521 constants. The NIST curves require the recipient's own uncompressed public point there; the Montgomery curves ignore it

An EC identity that arrives as a PFX file fits the same call. HPDFParsePFX reports it with KeyAlgorithm = pfxkaECDSA (X25519 and X448 identities included) and fills CurveOID, the private scalar in PrivateKey and the certificate point in PublicKey, which map straight onto the three parameters

uses
  System.SysUtils, System.IOUtils, HPDFDoc, HPDFPFX, HPDFKeyAgreement;

procedure OpenAuditPack(const LegalScalar: TBytes);
var
  Reader: THotPDF;
  EcKey: THPDFPFXKeyMaterial;
begin
  Reader := THotPDF.Create(nil);
  try
    Reader.AutoLaunch := False;
    // RSA identity becomes the primary key
    Reader.PubSecKeyMaterial :=
      HPDFParsePFX(TFile.ReadAllBytes('reviewer-a.pfx'), 'pfx-password');
    // EC identity from a PFX: curve, scalar and point go to the agreement list
    EcKey := HPDFParsePFX(TFile.ReadAllBytes('reviewer-c-p256.pfx'), 'pfx-password');
    if EcKey.KeyAlgorithm = pfxkaECDSA then
      Reader.AddPubSecAgreementKeyMaterial(EcKey.CurveOID,
        EcKey.PrivateKey, EcKey.PublicKey);
    // Raw X25519 scalar held by the legal team; no point needed
    Reader.AddPubSecAgreementKeyMaterial(HPDFOIDX25519, LegalScalar, nil);
    Reader.LoadFromFile('audit-pack.pdf', '');
    Writeln('Pages: ', Reader.GetLoadedPageCount);
  finally
    Reader.Free;
  end;
end;

Left alone, HotPDF tries every envelope against every registered key: the primary RSA key, then each additional RSA key, then the EC material. PubSecRecipientQuery narrows that to a single envelope. The callback receives the envelope count and must return a zero-based index inside the array; any other value, -1 included, raises during the load. To try every envelope, leave the property unassigned. Under Delphi the type is a reference to a function, so an anonymous method works, while under FPC it is a plain function pointer whose Context argument comes from PubSecRecipientQueryContext

One behavior deserves a test of its own. When no registered key unwraps any envelope, key recovery ends without a file key and does not raise for that reason. Assert on content you know the document holds, such as a text string or a metadata value; a LoadFromFile call that returns normally only shows the file structure parsed

What should you check in the target viewer?

HotPDF's writer and reader agree byte for byte, and the envelopes follow the CMS structures cited above. Third-party viewers are a different question: support for RSA-OAEP key transport, and even more so for X25519 or X448 recipients, varies between readers and between versions of the same reader. If the document must open in a particular viewer, encrypt a test file for a test certificate of the same key type and open it there before committing to a scheme. When the viewer is outside your control, its support list picks the scheme, and that outranks the X25519 convenience argument made earlier

Envelope permissions remain policy that conforming software honors, exactly as they do under password encryption. Seed quality is on you: AESGenerateRandomBytes exists for that job, and HotPDF wipes its own copy of the seed once the file key has been derived. If a string, stream or attachment needs a different crypt filter, the crypt filter policy guide for StmF, StrF and EFF shows how to assign one; the public-key handler accepts DefaultCryptFilter or Identity

Per-recipient envelopes for RSA and elliptic-curve certificates, along with the key loading on the way back in, are part of the HotPDF PDF component for Delphi and C++Builder, next to password encryption and digital signatures