sFTP file downloads hang for large files

Question

We're running into some issues downloading large files (about 4GB) with our Rebex sFTP implementation (.Net Core). We tried updating to the latest packages but we are still able to reproduce the issue:

Rebex.Common" Version="5.0.7501"
Rebex.Elliptic.Ed25519" Version="1.2.1"
Rebex.FileServer" Version="5.0.7501"
Rebex.FileSystem" Version="5.0.7501"

It seems like some sFTP clients are unable to fully download the file while others seem to be fine downloading large files from the sFTP. The clients themselves see the issue in different ways but it essentially results in the download being paused indefinitely. Depending on the client's timeout settings, it may result in an error or just appear to hang after downloading about 1-2GB of data.

Running in diagnostic/debug mode, it looks as though the issue lines up with when the server is "Performing algorithm negotiation and key exchange". On occasion this is fast and we see "Key exchange finished" pretty soon afterwards. However, there are times when the key exchange never completes and the file stops downloading.

Works
Remote SSH version: SSH-2.0-WinSCP_release_5.17.6
Performing key exchange using curve25519-sha256@libssh.org with ssh-rsa

Fails Most of the time
Remote SSH version: SSH-2.0-JSCH-0.1.54
Performing key exchange using ecdh-sha2-nistp256 with ssh-rsa

Fails Intermittently
Remote SSH version: SSH-2.0-Renci.SshNet.SshClient.0.0.1
Performing key exchange using diffie-hellman-group-exchange-sha256 with ssh-rsa

Most of the time there is no indication of failure and things just hang indefinitely. I have seen one instance with the JSCH client where it did record timeout in the logs "Key exchange timed out after 139 seconds of inactivity".

It looks like I can get around it by disabling the byte transfer based session renegotiation. Setting this to -1 seems to disable the back and forth when downloading larger files.

/// <summary>
/// Gets or sets maximum number of bytes transferred during a session. When this value is reached, a session renegotiation occurs.
/// </summary>
/// <remarks>
/// Setting the value to 0 or -1 disables session renegotiation based on bytes transferred.
/// Minimum allowed number of bytes is 16 MB. Recommended value is 1 GB.
/// </remarks>
public long MaxSessionTransferredBytes

Is there a reason why I shouldn't disable this? The SSH-2 protocol specification recommends a limit of at most 1 gigabyte which seems to be the default value so I'm hesitant to disable it completely.

Is it better to force a specific key re-negotiation scheme instead? The curve25519-sha256 seems to work but I can't see a way to choose specific algorithms. What's the recommended approach for this issue?

Answer 1

SSH renegotiation is a common cause of issues. In addition of it being prone to bugs, there also seems to be a disagreement on whether the SSH specification allows SSH data packets to be transmitted while a renegotiation is in progress. In most common contemporary SSH clients including of OpenSSH, WinSCP, Filezilla and PuTTY's psftp.exe, this is allowed, while in very old versions of OpenSSH and in SSH.NET (SSH-2.0-Renci.SshNet.SshClient.0.0.1) it is not.

JSCH seems to be a special case, because even though it does allow those packets, there seem to be a bug in their implementation that occasionally leads to hung connection (we have investigated this last year and it appears to occur when the client-side SSH window size happens to be lower than 32KB at the time of reexchange), which is consistent with your results.

However, this also means that the key exchange algorithm in use is not the cause, so forcing a specific scheme is not going to help. For example, the latest beta of SSH.NET (Renci) supports curve25519-sha256@libssh.org, but still fails during renegotiation with the following error:

Unhandled exception. Renci.SshNet.Common.SshException: Message type 94 is not valid in the current context.
   at Renci.SshNet.Sftp.SftpFileReader.Read()
   at Renci.SshNet.SftpClient.InternalDownloadFile(String path, Stream output, SftpDownloadAsyncResult asyncResult, Action`1 downloadCallback)
   at Renci.SshNet.SftpClient.DownloadFile(String path, Stream output, Action`1 downloadCallback)

(Message type 94 is SSH_MSG_CHANNEL_DATA, an SSH data packet.)

We already have a workaround for older SSH clients that disables data transfer during SSH negotiation. We'll try enabling this by default for JSCH and Renci as well, which would hopefully resolve the problem. I'll mail you a link to a hotfix shortly.

Disabling renegotiation by setting MaxSessionTransferredBytes could slightly weaken integrity (and to a lesser extent confidentiality) of the SSH session, although only a bit unless you are going to transfer tens of gigabytes over a single session.

Comments

Thanks for the quick reply. I'm going to spend some time testing out the hotfix build today and let you know how it goes.

---

This build fixed the problem for me. Do you think this build be officially published to Nuget?

---

Hello, thanks for giving this a try! There has been a release few days ago, so I’m afraid it will take at least several weeks before the next one is out. However, we can provide NuGet packages that can be used via a custom or offline repository in the meantime – just let us know if you are interested.

---

Thanks! We'll reference the dlls provided for now until the next offical release is ready. Appreciate the support!

---

I appear to be having this same issue with the Buru SFTP (2.0.0 and 2.1.0)  server while running backups from a Cisco Server which uses "SSH-2.0-JSCH-0.1.54".  Will this fix make it's way to Buru also?  Issue did not exist with 1.x series.

---

The fix already is in Buru which means there are probably more issues with JSCH. Since 1.x we increased the buffer sizes which may have exacerbated the issue. May I reach you using the email provided in registration so we can try a custom build which disables renegotiation (as mentioned in the original answer)?

---

Please do reach out.  Resolving this would be most welcome.