Hello, thanks for your detailed description!
This is actually an old issue of "disagreement between an SSH client and SSH server about the correct way to pad RSA signatures".
Some background: An RSA signature created with a private key of N bytes will usually be N bytes long. In few cases, it might be slightly shorter. In other cases, its most significant bit might be set and your server seem to require a padding of one additional zero octet at the beginning of the signature when this occurs. Because some SSH clients (including Rebex SFTP and PuTTY) send the signature unpadded, the server reports an error.
Unfortunately, SSH Authentication RFC does not specify whether the signatures are supposed to be sent padded or unpadded. However, we believe that this implies that both forms should actually work.
A similar issue has been recently discovered in libssh, but it was present in OpenSSH until 2002 as well:
And there is another explanation here:
We can add a workaround for this, but it would have to be only enabled when working with servers that reject unpadded signatures (because different servers may reject padded signatures). Which SFTP server do you use? Would it be possible to send a communication log showing its welcome message?