Unable to export private key in order to use a more capable algorithm

Question

As part of our CI process we have a number of unit tests running. As part of our testing suite, we generate a self signed certificate and use that to sign a mail. This works on my Windows 10 dev machine and our Windows 2016 build server. We recently switched to Windows 2019 domain joined build servers and now our tests start failing when trying to sign messages with the error:

Unable to export private key in order to use a more capable algorithm

We are using 2018R4.

Code to generate the test certificate:

public static X509Certificate2 CreateTestCertificate()
{
    // Generate asymmetric key pair.
    var rsa = RSA.Create(2048);

    // Set up the certificate properties.
    var request = new CertificateRequest("CN=doctor@who.com, O=WHO, OU=LAB, L=SEA, S=xxx, C=US, E=doctor@who.com", rsa, HashAlgorithmName.SHA512, RSASignaturePadding.Pss);

    request.CertificateExtensions.Add(new X509BasicConstraintsExtension(false, false, 0, false));

    // Set up the subject key identifier.
    request.CertificateExtensions.Add(new X509SubjectKeyIdentifierExtension(request.PublicKey, false));

    // Set up the enhanced key usage field.
    request.CertificateExtensions.Add(new X509EnhancedKeyUsageExtension(new OidCollection
        {
            new Oid("1.3.6.1.5.5.7.3.2"),    // Client Authentication
            new Oid("1.3.6.1.5.5.7.3.4")     // Secure Email
        },
        true));

    // Set up the SAN.
    var sanBuilder = new SubjectAlternativeNameBuilder();
    sanBuilder.AddEmailAddress("doctor@who.com");
    request.CertificateExtensions.Add(sanBuilder.Build());

    // Set up validity dates.
    var notBefore = new DateTimeOffset(DateTime.UtcNow.AddDays(-1));
    var notAfter = new DateTimeOffset(DateTime.UtcNow.AddYears(1));

    // Generate the certificate.
    var certificate = request.CreateSelfSigned(notBefore, notAfter);
    certificate.FriendlyName = "doctor@who.com";

    return new X509Certificate2(certificate.Export(X509ContentType.Pfx, "SuperSecret"), "SuperSecret", X509KeyStorageFlags.MachineKeySet);
}

Answer 1

This error indicates that the certificate's private key has been loaded into a CryptoAPI certificate store, probably due to the way it was generated. CryptoAPI is a legacy Windows API, and unlike the newer CNG API, it does not support certain operations such as PSS signatures. Rebex code tries to work around this problem by exporting importing the private key into a CNG key store, but that only works if the private key is exportable, which it apparently is not in your scenario.

To work around the issue, ensure the private key is loaded into a CNG key store by specifying AlwaysCng or PreferCng option when loading the PFX for use with Rebex Secure Mail:

CertificateChain chain = CertificateChain.LoadPfx(
    path,
    password,
    KeySetOptions.AlwaysCng | KeySetOptions.MachineKeySet);

(I also specified KeySetOptions.MachineKeySet - feel free to remove that if it doesn't match your scenario.)

Comments

Thanks. Your explanation did the trick. Based on that, it was as simple as marking the key as exportable. So the last line of my code became

return new X509Certificate2(certificate.Export(X509ContentType.Pfx, "SuperSecret"), "SuperSecret", X509KeyStorageFlags.Exportable | X509KeyStorageFlags.MachineKeySet);

---

Yes, that also does the trick by actually making the workaround possible, but it involves actually exporting/importing the private key each time a signature is to be created, which is sub-optimal.

The following solution might be better - once you generate the certificate and key and save the PFX file, load it into Certificate class once with AlwaysCng option, and then save it again, overwriting the original file:

Certificate cert = CertificateChain.LoadPfx(
    path,
    password,
    KeySetOptions.AlwaysCng);
cert.Save(path, CertificateFormat.Pfx, password);

This should result in a PFX that loads into CNG key store by default, eliminating the need for the workaround.