A fingerprint-based custom certificate validation equivalent to the code above is in fact substantially faster than the default validation, which involves multiple asymmetric key operations and possibly even downloading of certificate revocation lists. Unless you are doing something much more complicated in the certificate validation event handler, this is unlikely to slow down your download process, and the slow down is most likely caused by something else.
In any case, solving the "Server certificate was rejected by the verifier because it has expired" problem by switching to a custom fingerprint-based certificate validator is strongly discouraged - it will stop working as soon as someone fixes the issue at the server by renewing the certificate (which is actually the proper solution to this problem).