In FIPS-only mode, only FIPS-validated cryptographic algorithms provided by Windows are used. The same applies when Rebex
UseFipsAlgorithmsOnly option is set to true (this is the default value for Windows running in FIPS mode).
In your scenario, the server most likely refuses AES/CBC ciphers, does not support AES/GCM ciphers, and requires the client to use AES/CTR ciphers. However, Windows does not offer any FIPS-compliant AES/CTR implementation, so this cipher is not available in FIPS-only mode of Rebex components.
There are several possible solutions:
Configure the server to allow AES/CBC ciphers, or upgrade to a server that supports AES/GCM ciphers.
Configure the client app to allow AES/CTR ciphers by disable FIPS-only mode for Rebex SFTP:
Rebex.Security.Cryptography.CryptoHelper.UseFipsAlgorithmsOnly = false;