SFTP Server upload file slow

Question

Hello,
I built a sftp server using Rebex.FileServer_v7.0.8865. The code is like this:

var _server = new FileServer();
_server.Bind(port, FileServerProtocol.Sftp);
_server.Keys.Add(GeneratePrivateKey());
_server.Authentication += (sender, e) => Server_Authentication(sender, e).Wait();
_server.Start();

The sftp server is deployed on ServerA which OS is CentOS7. Then I upload a file from ServerB using the sftp command.
Default the RTT between ServerA and ServerB is less then 1ms, the upload speed is 11M/s.
When I set the RTT to 100ms(the command is like below), the upload speed is 2.5M/s.

$ tc qdisc add dev ens192 root netem delay 100ms

It seems that with the increase of RTT, the upload speed drops sharply.
Is there any way to improve this situation?

Answer 1

SFTP protocol is very susceptible no network latency due to its design. An SFTP file transfer consists of series of requests/responses, where the client requests the server to read or write short blocks of data, usually less than 64 KB each (the size can be customized using UploadBufferSize and DownloadBufferSize properties in FileServer.Settings).

This means that for transferring a large file, the SFTP client has to issue thousands of read or write requests. The only reason why performance of SFTP does not suffer badly on high-latency connections due to high RTT is the fact that most SFTP clients use pipelining - they send many requests in advance, without waiting for response to each of them. The queue length varies fro client to client (some clients are known to queue more than 100 requests in advance).

Additionally, SFTP runs over SSH, which supports multiple channels withing the same SSH session (which runs over a single TCP connection), where each channel has its own independent send/receive buffers and its own flow control. This applies even if only a single channel is used, as with most SFTP connections. It also implies that simply tweaking TCP send/receive buffer sizes might not sufficient in order to achieve higher throughput.

Rebex File Server library provides some settings to address this. It can't force SFTP clients to use a queue length of particular size, but using larger buffers might mitigate some of the effects of increased RTT. Try increasing TCP buffer sizes for SSH session TCP sockets to see whether it makes any noticeable difference:

fileServer.Settings.ReceiveBufferSize = 64 * 1024 * 1024;
fileServer.Settings.SendBufferSize = 64 * 1024 * 1024;

There is also an experimental feature that makes it possible to increase the buffer size of SFTP protocol’s SSH channel. For example, to increases it to 4 MB, call this:

CryptoHelper.SetOption(fileServer.Settings, "SftpChannelBufferSize", 4 * 1024 * 1024);

(Where "fileServer" is an instance of Rebex.Net.Servers.FileServer and "CryptoHelper" is a static Rebex helper class from Rebex.Common assembly’s Rebex.Security.Cryptography namespace.)

Comments

I set the SftpChannelBufferSize to 4M, set the SendBufferSize and ReceiveBufferSize to 64M, and then using the same client to upload files to Rebex.FileServer and OpenSSH respectively(the two sftp services deployed on the same server), under different RTT, the upload speed is as follows:

RTT        Rebex.FileServer7.0.8865    OpenSSH7.4
1ms    11M/s                    11M/s
10ms    11M/s                    11M/s
20ms    9.2M/s                    11M/s
30ms    6.5M/s                    11M/s
50ms    4.1M/s                    11M/s
100ms    2.1M/s                    11M/s
150ms    1.4M/s                    11M/s
200ms    1.0M/s                    9.5M/s

---

I have found the answer. The ReceiveBuffer of Socket is limited by CentOS.
Thanks for your answer.

---

Thanks or the update! I was planning to respond with some more in-depth info, but we were quite busy last week preparing the next release. In summary:

- We have not extensively measured performance of Rebex libraries on non-Windows platforms yet. The performance seems to be good enough in most scenarios, but there is obviously room for improvement.

- We are well aware of several bottlenecks in our current SSH core, and we are working on eliminating them for Rebex File Server 8.0. We'll extensively benchmark the next version on Linux as well. SFTP/SSH performance is a top priority for version 8.0, which should become available in Q4 2024.