Technical Article

HotPDF Cert Store PDF Signing: CNG vs CAPI Byte Order

HotPDF signs a PDF against a certificate already sitting in the Windows Certificate Store by handing the digest to Windows itself, and Windows completes that request through one of two private-key backends: CNG, which returns the RSA signature big-endian, or the legacy CryptoAPI CSP, which returns it little-endian. Get the two confused and the CMS signature HotPDF embeds is byte-reversed for whichever backend actually answered, so a conforming validator reports the signature as invalid even though the document bytes were never touched

Two unrelated problems hide behind that one sentence, and HotPDF's system-certificate signer has to solve both before it signs anything at all. The byte-order mismatch is silent: the signing call still returns True, the PDF still opens, and the failure only shows up when a viewer walks the CMS structure and rejects it. The second problem is loud and specific to C++Builder: half a dozen crypt32 functions refuse to link, because the import library RAD Studio ships does not export them. Neither problem exists if you only ever sign with a PFX file, which is why it tends to catch developers moving from PFX-based one-call signing to a certificate an IT department already installed in the user's profile

Selecting a certificate from the store

HotPDF exposes this path as HPDFSignPDFStreamWithSystemCertificate and HPDFSignPDFFileWithSystemCertificate, both driven by a THPDFCertificateStoreSelector record: Location (cslCurrentUser or cslLocalMachine), StoreName ('MY', the personal store, by default), a SHA-1 Thumbprint, and an AllowUI flag. The thumbprint is normalized internally, so hyphens or spaces copied straight out of the Certificate Manager UI are stripped before the comparison runs

var
  Selector: THPDFCertificateStoreSelector;
  Options: THPDFCMSSignOptions;
begin
  Selector := THPDFCertificateStoreSelector.Default;  // cslCurrentUser, store 'MY'
  Selector.Thumbprint := 'A1B2C3D4E5F6A7B8C9D0E1F2A3B4C5D6E7F8A9B0';
  Selector.AllowUI := False;

  Options := HPDFCMSDefaultOptions(palBaseline_B_B);
  if not HPDFSignPDFFileWithSystemCertificate('invoice.pdf',
    'invoice-signed.pdf', Selector, Options) then
    raise Exception.Create('Certificate-store signing failed');
end;

AllowUI = False matters more than it looks, because it maps directly to CRYPT_ACQUIRE_SILENT_FLAG, and Windows honors that literally: if the matched certificate's private key lives on a smart card or token that needs a PIN prompt Windows has not already cached, CryptAcquireCertificatePrivateKey fails rather than popping a dialog from what might be a service process. That failure is loud, an EHPDFCMSError you see immediately, but it is easy to misread as "certificate not found" when the real cause is a token sitting there waiting for a PIN nobody is going to type

Why do CNG and CAPI disagree about byte order?

Which backend answers is not a guess: CryptAcquireCertificatePrivateKey reports it directly through a KeySpec out-parameter, and that single value is what HotPDF's signer branches on. A CNG Key Storage Provider key comes back with KeySpec set to the sentinel CERT_NCRYPT_KEY_SPEC ($FFFFFFFF); anything else is a traditional CryptoAPI CSP key. Most personal certificates issued or imported on a current Windows install resolve to CNG even though a legacy CSP shim still exists for compatibility, which is why HotPDF requests CRYPT_ACQUIRE_ALLOW_NCRYPT_KEY_FLAG together with CRYPT_ACQUIRE_PREFER_NCRYPT_KEY_FLAG before it looks at which value came back

The two backends do not just call different functions, NCryptSignHash against a CNG key, CryptSignHashA against a CSP key; they hand back the raw RSA signature in opposite byte order. CNG's output already matches what PKCS#1 expects: a big-endian octet string, most significant byte first, exactly what RFC 8017's I2OSP conversion produces and what a CMS SignerInfo (RFC 5652) needs in its signature field under ISO 32000-1 §12.8.3. CryptoAPI's CryptSignHash, by contrast, returns the signature little-endian, a documented quirk that goes back to how classic CSPs represented big numbers internally. Skip the reversal on the CAPI path and every byte in the signature sits in the wrong place; the RSA math is still correct, but the octet string a verifier reads is not the one PKCS#1 defines

// CryptSignHashA returns the RSA signature least-significant byte first;
// CMS/PKCS#7 (ISO 32000-1 Section 12.8.3) needs it most-significant byte first.
for I := 0 to (Length(Signature) div 2) - 1 do
begin
  Temp := Signature[I];
  Signature[I] := Signature[High(Signature) - I];
  Signature[High(Signature) - I] := Temp;
end;

What about a custom signer callback?

Anyone bypassing HotPDF's built-in cert-store signer inherits the same byte-order rule. HPDFCMSSignPDFStreamWithExternalSigner takes a THPDFCMSSignDigestCallback, a closure of type reference to function(const SignedAttributesSHA256: TBytes): TBytes, for signing through an HSM, a smart-card middleware stack, or anything else that is not a certificate the Windows store can hand you a key handle for. Whatever backend sits behind that callback, the bytes it returns have to land in big-endian order before HotPDF folds them into the CMS structure

Signer :=
  function(const SignedAttributesSHA256: TBytes): TBytes
  begin
    if UsesCngKeyStorageProvider then
      Result := SignWithMyCngKey(SignedAttributesSHA256)       // already big-endian
    else
      Result := ReverseBytes(SignWithMyLegacyToken(SignedAttributesSHA256));
  end;
HPDFCMSSignPDFStreamWithExternalSigner(InputStream, OutputStream,
  CertificateDER, Signer, Options);

Worth being explicit about a boundary here: HotPDF's two built-in signing paths, CNG through NCryptSignHash with PKCS#1 padding and CAPI through CryptSignHashA, both target RSA keys signing a 32-byte SHA-256 digest. Neither negotiates an ECDSA signature format. A certificate whose private key is EC-based needs a signer you write yourself against HPDFCMSSignPDFStreamWithExternalSigner, encoding the ECDSA signature the way CMS expects rather than assuming a fixed-length RSA byte string, so do not expect the built-in cert-store signer to do the right thing for a token provisioned with an EC certificate

Why does C++Builder fail to link CertOpenStore?

Because RAD Studio's default C++Builder import library, import32.lib, does not export CertOpenStore, or five of its neighbors: CertEnumCertificatesInStore, CertGetCertificateContextProperty, CertFreeCertificateContext, CertCloseStore, and CryptAcquireCertificatePrivateKey. Delphi builds never see this, because dcc32/dcc64 resolve a static external 'crypt32.dll' import straight into the PE import table. C++Builder is different: the Delphi compiler emits an OMF .obj for the package build, ilink32 links it, and at that point the same external declaration is just an unresolved symbol waiting for an import library on the command line. Pointing the linker at the Windows SDK's psdk directory, where the full crypt32.lib does export all six symbols, does not fix it either: ilink32 only links the import libraries actually named on its command line, import32.lib cp32mt.lib by default, and adding a search path does not make it pull in anything extra from that path. Running tdump against import32.lib confirms the gap directly, zero hits for CertOpenStore, against six clean hits in the SDK's crypt32.lib

HotPDF resolves this the same way it already handles certificate enumeration elsewhere in the library: instead of asking the linker for these symbols, it loads them at runtime. An internal THPDFCryptoProcs record carries a crypt32.dll handle, an advapi32.dll handle, and eleven function-pointer fields; LoadCryptoProcs loads both DLLs and resolves every entry point with GetProcAddress exactly once, at the start of HPDFSignPDFStreamWithSystemCertificate, raising EHPDFCMSError immediately if anything is missing rather than failing later with an access violation deep inside the signing flow

type
  TCertOpenStoreFn = function(lpszStoreProvider: Pointer; dwEncodingType: DWORD;
    hCryptProv: NativeUInt; dwFlags: DWORD; pvPara: Pointer): HCERTSTORE; stdcall;
var
  Crypt32Handle: HMODULE;
  CertOpenStore: TCertOpenStoreFn;
begin
  Crypt32Handle := LoadLibrary('crypt32.dll');
  if Crypt32Handle = 0 then
    raise Exception.Create('crypt32.dll could not be loaded');
  @CertOpenStore := GetProcAddress(Crypt32Handle, 'CertOpenStore');
  // ... use CertOpenStore, then FreeLibrary(Crypt32Handle) when signing returns
end;

Loading happens once per call rather than lazily inside each helper, because the closure that picks between CNG and CAPI captures the loaded function table by value and has to stay alive for the whole signing flow, including the callback into HPDFCMSSignPDFStreamWithExternalSigner; both DLL handles are freed in the outermost finally block once signing finishes or raises. None of this touches the public surface: HPDFSignPDFStreamWithSystemCertificate, HPDFSignPDFFileWithSystemCertificate, and THPDFCertificateStoreSelector keep the exact signatures they had before, so picking up the fix is a rebuild for existing callers, not a code change

What this does not cover

Getting the byte order and the C++Builder link right produces a CMS SignerInfo a validator can parse and a signature it can check arithmetically; it says nothing about whether that validator should trust the certificate behind it, since chain building, revocation checking, and timestamp policy are separate concerns layered on top through the CMS options, not something byte-order correctness buys for free. Two housekeeping details matter as much as the cryptography: the PCCERT_CONTEXT returned by certificate lookup must be freed with CertFreeCertificateContext before the store closes, and an acquired CNG or CSP key handle, when the API reports the caller owns it, must be released through the matching backend's own call, never the other one's. If the svValid result you get back after all this turns out narrower than you expected, the article on verifying PDF signatures lays out exactly what that flag does and does not promise. Because the certificate stays in Windows' custody the whole time here, cert-store signing sidesteps an entire attack surface: there is no PKCS#12 file to parse and no ASN.1 to walk yourself, which is the problem HotPDF's PKCS#12 and ASN.1 hardening addresses for the PFX-file signing path instead

Cert-store signing, PFX signing, and external-signer callbacks are three doors into the same CMS/PKCS#7 pipeline inside the HotPDF PDF component for Delphi and C++Builder, and picking the right one mostly comes down to who is allowed to hold the private key: your process, a PFX file, or Windows itself