Thank you for the log file. It showed that the problem is: Diffie-Hellman CSP not available or doesn't support this key size, and managed Diffie-Hellman forbidden in FIPS-only mode.
You can either ignore FIPS mode and force using non-FIPS implementations by setting:
Rebex.Security.Cryptography.CryptoHelper.UseFipsAlgorithmsOnly = false;
Or you can try to use different key exchange algorithm to find one, which can be run in FIPS mode on your server.
It can be set like this:
From verbose log you can analyze received server's
SSH_MSG_KEXINIT packet to see key exchange algorithms your server supports.