RevocationCheckFailed (only on deployment clients; not getting the error on dev machine)

Question

My users are unable to send mail, but I'm not having the problem on my development PC. So I enabled debug-level logging. They are getting a RevocationCheckFailed but I am not. We are going against the same server, same port, using Smtp 2.0.5715.0, so what could be causing the different result? What is different? I assume the DLLs have been copied locally, or they could not even be connecting, right?

09 13:46:45.923 DEBUG Smtp(1)[1] Info: Connection succeeded.
2016-06-09 13:46:45.925 DEBUG Smtp(1)[1] Info: Upgrading connection to TLS/SSL.
2016-06-09 13:46:45.939 INFO Smtp(1)[1] TLS: State StateChange:Negotiating
2016-06-09 13:46:45.939 DEBUG Smtp(1)[1] TLS: HandshakeMessage:ClientHello was sent.
2016-06-09 13:46:46.001 DEBUG Smtp(1)[1] TLS: HandshakeMessage:ServerHello was received.
2016-06-09 13:46:46.013 DEBUG Smtp(1)[1] TLS: HandshakeMessage:Certificate was received.
2016-06-09 13:46:46.014 DEBUG Smtp(1)[1] TLS: HandshakeMessage:ServerKeyExchange was received.
2016-06-09 13:46:46.014 DEBUG Smtp(1)[1] TLS: HandshakeMessage:ServerHelloDone was received.
2016-06-09 13:46:46.017 DEBUG Smtp(1)[1] TLS: Verifying server 
// I've sanitized the servername here
certificate ('CN=xxxx.xxxxxxxxx.com, OU=EssentialSSL, OU=Domain Control Validated').
09 13:46:46.071 INFO Smtp(1)[1] TLS: Certificate verification status: UnknownRev (0)
2016-06-09 13:46:46.071 DEBUG Smtp(1)[1] TLS: Certificate verification result: RevocationCheckFailed
2016-06-09 13:46:46.073 DEBUG Smtp(1)[1] TLS: Error while processing TLS packet: Rebex.Net.TlsException: Unable to perform revocation check of the server certificate.
   at Rebex.Net.AIB.CE(String A, CertificateChain B)
   at Rebex.Net.AIB.EE(Byte[] A, Int32 B, Int32 C, LHB D)
   at Rebex.Net.AIB.KC(Byte[] A, Int32 B, Int32 C)
   at Rebex.Net.ZHB.CD(Byte[] A, Int32 B, Int32 C)
   at Rebex.Net.ZHB.HD()
2016-06-09 13:46:46.074 INFO Smtp(1)[1] TLS: Alert Alert:Alert was sent.
2016-06-09 13:46:46.074 INFO Smtp(1)[1] TLS: State StateChange:Closed
2016-06-09 13:46:46.075 DEBUG Smtp(1)[1] TLS: Closing TLS socket.

Answer 1

Yes, you appear to have all the required DLLs. The RevocationCheckFailed error indicates that the operating system was unable to retrieve a certificate revocation list (CRL) from the server certificate's issuer and perform a check to determine whether the server certificate has been revoked.

To retrieve the CRL, the OS has to be able to connect to the Internet (usually using the HTTP protocol), and if this is not possible, revocation check would fail. Perhaps the CRL is not accessible from your development PC?

There is a built-in utility in Windows called certutil which you can to run to see what's wrong. First, download the server certificate using the following code:

Smtp client = new Smtp();
client.Settings.SslAcceptAllCertificates = true;
client.Connect("server", SslMode.Implicit);
client.TlsSocket.ServerCertificate[0].Save("cert.der", CertificateFormat.Base64Der);

Then, tun the certutil command line utility on the downloaded cert.der file:

certutil -verify -urlfetch cert.der

Please check its output to see whats wrong, and send us a copy (or post it here) if possible.

Comments

Thanks. I'm out of the office at the moment but will try tomorrow. But in the meantime, a clarification. My dev PC is working fine; it is the user's PCs that have the problem. They are connecting to the internet without problem, based on the fact that the initial connection to the mail server succeeds, according to the Rebex log; the failure occurs when upgrading the connection to TLS/SSL.  

What does this line in the log mean? 09 13:46:46.071 INFO Smtp(1)[1] TLS: Certificate verification status: UnknownRev (0)

Does UnknownRev mean they were unable to confirm revocation status, but the reason for their inability is unknown?

Do users need any folder permissions  in order to perform this CRL check? Do they need access to any MSFT cryptology libraries?

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

 Is Rebex handling the certificate in memory as a stream?

I have Admin privileges, and that might explain why I can do it and they cannot.

---

Also, since I continue to have NO such problems on my development PC, we can assume the certificate is good.  What specific permissions does a Windows user require in order to perform this certificate check?

---

The line in the log indicates UnknownRev error, which corresponds to Windows CryptoAPI error status CERT_TRUST_REVOCATION_STATUS_UNKNOWN (see https://msdn.microsoft.com/en-us/library/windows/desktop/aa377590(v=vs.85).aspx). It doesn't specify the exact cause of the failure - that's what the certutil utility is for. Running it at the client's machine (using the same account running the Rebex code) should provide enough information to make it possible to tell what is going on.

---

Also, to answer your other questions and suggestions:
- Yes, we can assume the certificate is good.
- On Windows, we validate certificates through .NET's X509Certificate2/X509Store objects, which uses Windows CryptoAPI.
- The user was able to invoke the validation, which indicates that he has access to Windows CryptoAPI.
- No special permissions should be required to perform revocation checks, and administrator privileges are definitely not needed for this.
- To learn more about Windows CRL infrastructure, check out https://technet.microsoft.com/en-us/library/ee619754(v=ws.10).aspx
- C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys is for private keys, not CRLs.
 (check the previous article for details in CRL storage)
- Check out http://blogs.interfacett.com/how-to-examine-any-certificate-revocation-list-in-windows-with-certutil for information on using certutil utility the display revocation info.

However, before trying to delve deep into this, please just try running the certutil utility (as described above). It will tell you where exactly is the problem.

---

I have the certutil results from a user where there is the error, and the certutil results from my PC where everything is working.  The two files are quite different.  The users are running AVAST. I am not.   It looks to me as if the users' machines are examining the Avast proxy server and not the Rackspace mail server.   Have I reached the correct conclusion?
USER MACHINE CERTUTIL RESULTS:

Issuer:
    CN=avast! Web/Mail Shield Root
    O=avast! Web/Mail Shield
    OU=generated by avast! antivirus for SSL/TLS scanning
  Name Hash(sha1): 8e9e99e357ad05143a3859a76308cb50ae23601f
  Name Hash(md5): b0fcd17867f2ae2649407da6c4af2cd9
Subject:
    CN=secure.emailsrvr.com
    OU=EssentialSSL
    OU=Domain Control Validated
  Name Hash(sha1): 1f2910944103298e78aa6b2f73541720ed126d4f
  Name Hash(md5): 7d4907b4600e3af0f1be6adc471b1343
Cert Serial Number: 75a4c970b3575d4a8d75a717f2f4ca94

<snip>

My dev PC certutil results:

Issuer:
    CN=COMODO RSA Domain Validation Secure Server CA
    O=COMODO CA Limited
    L=Salford
    S=Greater Manchester
    C=GB
Subject:
    CN=secure.emailsrvr.com
    OU=EssentialSSL
    OU=Domain Control Validated
Cert Serial Number: 60f1f5454be9f164b8a7f4ff0b0eceb1

<snip>

---

Thanks! You reached the correct conclusion - the certificate at the user machine is different. Apparently, Avast uses an equivalent of man-in-the-middle attack to make it possible to intercept the encrypted traffic. For this to work, it has to replace the original certificates with ones signed by its own certification authority and act like a proxy. I assume they use the same process for SMTP/SSL as they do for HTTPS: https://blog.avast.com/2015/05/25/explaining-avasts-https-scanning-feature/

However, this still does not entirely  explain the "revocation check failed" error. Was certutil actually able to validate the certificate re-signed by Avast, or did it fail as well?

But one thing is certain - when they re-signed, the certificate, they had to replace or remove the CRL URLs (which are embedded in the certificate) as well. Can you please open the certificate (just double-click the .cer or .der file), locate the "CRL Distribution Points" fields and compare them with the original certificate?

---

I have asked the user on whose machine the testing was done to email me the .der file.  I will post the info when I receive it from her.  Thanks for the help.

---

I dont' see any "CRL Distribution Points" fields so-named, but the issuer is
CN = avast! Web/Mail Shield Root
O = avast! Web/Mail Shield
OU = generated by avast! antivirus for SSL/TLS scanning

and the issuer of the "good" one is

Issuer:
    CN=COMODO RSA Domain Validation Secure Server CA
    O=COMODO CA Limited
    L=Salford
    S=Greater Manchester
    C=GB
Subject:
    CN=secure.emailsrvr.com
    OU=EssentialSSL

---

Thanks! Could you please post the rest of `certutil` output as well, or at least confirm that it reports an equivalent of "Revocation status unknown"? I assume it would look like this - http://security.stackexchange.com/questions/116147/tinyca-missing-crl-information-in-root-and-leaf-certificates - but I would like to make sure.

---

In any case, please try using the custom certificate validator described in my new answer to work around the issue:
http://forum.rebex.net/6248/revocationcheckfailed-deployment-clients-getting-machine?show=6281#a6281

Answer 2

Thanks to your help, we have reached the following conclusion regarding the cause of the error:

1. The user getting revocation check failure uses Avast's TLS/SSL scanning feature (they seem to use the same process for SMTP/SSL as they do for HTTPS).
2. For this to work, Avast has to replace the original certificates with ones signed by its own certification authority and act like a proxy. This means that the user gets a different certificate instead of the proper one. It's issued and signed by "avast! Web/Mail Shield Root" CA.
3. Unfortunately, this "avastified" certificate does not contain "CRL distribution point" (CDP) extension.
4. Default certificate validator in Rebex Secure Mail asks the Windows certificate infrastructure to perform certificate validation that includes the revocation check.
5. Windows can't perform the revocation check because there is no CDP extension on the certificate. Due to this, the client gets "revocation check failed" error.

Solution:

Trying to convince users to disable Avast for non-HTTPS TLS/SSL is most likely not practical, and convincing Avast to simulate CRL distribution points might take a lot of effort as well.

Instead, please try the following custom certificate validator that disables CRL check for certificate chains that lack CDP extension:

public static void CustomCertificateValidator(object sender, SslCertificateValidationEventArgs e)
{
    // determine whether any certificate in the chain contains the CDP extension
    bool hasCrl = false;
    foreach (var cert in e.CertificateChain)
    {
        if (e.Certificate.GetCrlDistributionPoints() != null)
        {
            hasCrl = true;
        }
    }

    // skip CRL check if no CDP extensions found
    ValidationOptions options = ValidationOptions.None;
    if (!hasCrl)
    {
        options |= ValidationOptions.SkipRevocationCheck;
    }

    // validate the certificate
    ValidationResult result = e.CertificateChain.Validate(e.ServerName, options);

    // accept or reject it according to the result
    if (result.Valid)
    {
        e.Accept();
    }
    else
    {
        e.Reject(result.Status);
    }
}

To utilize this, register the event handler with Smtp object (applies to Imap, Pop3, Ews and Ftp as well) prior to calling the Connect method:

var smtp = new Smtp();
smtp.ValidatingCertificate += CustomCertificateValidator;
...