Ask a Question

Problems Reaching Good SFTP Download Speed

0 votes
by (120 points)
edited by

I am comparing SFTP download speeds I can achieve between same two endpoints using REBEX vs. using Putty's PSCP. I finagled with most Sftp.Client settings and no matter what I tried, I couldn't get the speed over ~2000 kB/s. On the other hand, PSCP gives me ~7700 kB/s.

Any suggestions what I can try to speed up REBEX would be greatly appreciated!

PSCP Command
PSCP.EXE -l remote_user -pw password -v remote_host:/remote_path/remote_file.dat E:\local_path\local_file.dat

Output
Connecting to xxx.xxx.xxx.xxx port 22
Server version: SSH-2.0-6.4.1.19 SSH Tectia Server
Using SSH protocol version 2
We claim version: SSH-2.0-PuTTY_Release_0.62
Doing Diffie-Hellman group exchange
Doing Diffie-Hellman key exchange with hash SHA-256
Host key fingerprint is:
ssh-rsa 1536 70:94:8f:c3:0b:b7:e8:41:9d:2c:41:ee:fb:5d:8d:50
Initialised AES-256 SDCTR client->server encryption
Initialised HMAC-SHA1 client->server MAC algorithm
Initialised AES-256 SDCTR server->client encryption
Initialised HMAC-SHA1 server->client MAC algorithm
Using username "remote_user".
Using SSPI from SECUR32.DLL
Attempting GSSAPI authentication
GSSAPI authentication request refused
Attempting keyboard-interactive authentication
Access granted
Opened channel for session
Started a shell/command

...
local_file | 893440 kB | 7769.0 kB/s | ETA: 00:02:58 | 39%
local_file | 903616 kB | 7789.8 kB/s | ETA: 00:02:56 | 39%
local_file | 914016 kB | 7812.1 kB/s | ETA: 00:02:54 | 40%
...

REBEX PowerShell Script

Add-Type -Path "Rebex.Common.dll"
Add-Type -Path "Rebex.Networking.dll"
Add-Type -Path "Rebex.Sftp.dll"

$client = New-Object Rebex.Net.Sftp

# set log writer
$log = New-Object Rebex.FileLogWriter -ArgumentList "RebexTransfer.log"
$client.LogWriter = $log
$client.LogWriter.Level = [Rebex.LogLevel]::Debug

# transfer progress event handler
$client_SpeedMonitoring = { param($sender, [Rebex.Net.SftpTransferProgressChangedEventArgs] $e) Write-Host "Current speed: $($e.BytesPerSecond/1KB) KB/s" }
$client.add_TransferProgressChanged($client_SpeedMonitoring)

# connect
$client.Connect("remote_host", 22)
$client.Login("remote_user", "password")

# download
$client.GetFile("/remote_path/remote_file.dat", "E:\local_path\local_file.dat")

Output
...
Current speed: 2138.7314453125 KB/s
Current speed: 2144.439453125 KB/s
Current speed: 2133.2646484375 KB/s
Current speed: 2123.486328125 KB/s
Current speed: 2114.9306640625 KB/s
...

Log
2017-12-12 23:50:37.913 INFO Sftp(19)[12] Info: Connecting to remote_host:22 using Sftp.
2017-12-12 23:50:37.913 INFO Sftp(19)[12] Info: Assembly: Rebex.Sftp 2017 R6.1 for .NET 4.0-4.7 (Trial)
2017-12-12 23:50:37.913 INFO Sftp(19)[12] Info: Platform: Windows 6.2.9200 64-bit; CLR: 4.0.30319.42000
2017-12-12 23:50:37.913 DEBUG Sftp(19)[12] Info: Culture: en; Windows-1252
2017-12-12 23:50:37.913 DEBUG ProxySocket(10)[12] Proxy: Resolving 'remote_host'.
2017-12-12 23:50:37.991 DEBUG ProxySocket(10)[12] Proxy: Connecting to none proxy at xxx.xxx.xxx.xxx:22.
2017-12-12 23:50:38.069 DEBUG Sftp(19)[12] SSH: Server is 'SSH-2.0-6.4.1.19 SSH Tectia Server'.
2017-12-12 23:50:38.069 INFO Sftp(19)[12] SSH: Negotiation started.
2017-12-12 23:50:38.116 DEBUG Sftp(19)[12] SSH: Group exchange.
2017-12-12 23:50:38.209 DEBUG Sftp(19)[12] SSH: Negotiating key.
2017-12-12 23:50:38.240 DEBUG Sftp(19)[12] SSH: Received 2048-bit Diffie-Hellman prime (minimum allowed size is 1024 bits).
2017-12-12 23:50:38.334 DEBUG Sftp(19)[12] SSH: Validating signature.
2017-12-12 23:50:38.350 DEBUG Sftp(19)[12] SSH: Received 1536-bit RSA server key (minimum allowed size is 1024 bits).
2017-12-12 23:50:38.350 INFO Sftp(19)[12] SSH: Negotiation finished.
2017-12-12 23:50:38.350 INFO Sftp(19)[12] Info: Server: SSH-2.0-6.4.1.19 SSH Tectia Server
2017-12-12 23:50:38.350 INFO Sftp(19)[12] Info: Fingerprint (MD5): 70:94:8f:c3:0b:b7:e8:41:9d:2c:41:ee:fb:5d:8d:50
2017-12-12 23:50:38.350 INFO Sftp(19)[12] Info: Fingerprint (SHA-256): Ghh6y7IaDCAr7a+Gm0uQdVILRTOkfdnVhKt+MrzKCZk
2017-12-12 23:50:38.350 INFO Sftp(19)[12] Info: Cipher info: SSH 2.0, diffie-hellman-group-exchange-sha256, ssh-rsa-sha256@ssh.com, aes256-ctr/aes256-ctr, hmac-sha1/hmac-sha1
2017-12-12 23:50:38.428 DEBUG Sftp(19)[12] SSH: Allowed authentication methods for 'remote_user': gssapi-with-mic, password, publickey, keyboard-interactive.
2017-12-12 23:50:38.428 DEBUG Sftp(19)[12] SSH: Trying password authentication for 'remote_user'.
2017-12-12 23:50:38.474 DEBUG Sftp(19)[12] SSH: Authentication successful.
2017-12-12 23:50:38.506 DEBUG Sftp(19)[12] SSH: Requesting subsystem 'sftp'.
2017-12-12 23:50:38.568 INFO Sftp(19)[12] Command: SSH_FXP_INIT (4)
2017-12-12 23:50:38.599 INFO Sftp(19)[12] Response: SSH_FXP_VERSION (3, 1 extension)
2017-12-12 23:50:38.599 INFO Sftp(19)[12] Info: Using SFTP v3.
2017-12-12 23:50:38.599 INFO Sftp(19)[12] Command: SSH_FXP_REALPATH (1, '.')
2017-12-12 23:50:38.646 INFO Sftp(19)[12] Response: SSH_FXP_NAME (1, 1 item)
2017-12-12 23:50:38.646 INFO Sftp(19)[12] Info: Home directory is '/ftp/remote_user'.
2017-12-12 23:50:47.523 INFO Sftp(19)[12] Command: SSH_FXP_OPEN (2, '/remote_path/remote_file.dat', 1)
2017-12-12 23:50:47.569 INFO Sftp(19)[12] Response: SSH_FXP_HANDLE (2, 0x1C000E501C000D60)
2017-12-12 23:50:47.569 INFO Sftp(19)[12] Command: SSH_FXP_FSTAT (3, 0x1C000E501C000D60)
2017-12-12 23:50:47.601 INFO Sftp(19)[12] Response: SSH_FXP_ATTRS (3)
2017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (4, 0x1C000E501C000D60, 0, 28672)
2017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (5, 0x1C000E501C000D60, 28672, 28672)
017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (6, 0x1C000E501C000D60, 57344, 28672)
2017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (7, 0x1C000E501C000D60, 86016, 28672)
2017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (8, 0x1C000E501C000D60, 114688, 28672)
2017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (9, 0x1C000E501C000D60, 143360, 28672)
2017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (10, 0x1C000E501C000D60, 172032, 28672)
2017-12-12 23:50:47.616 DEBUG Sftp(19)[12] Command: SSH_FXP_READ (11, 0x1C000E501C000D60, 200704, 28672)
...

Applies to: Rebex SFTP

1 Answer

0 votes
by (144k points)

Hello, these are some suggestions and comments on this:

  • PuTTY's pscp uses the SCP protocol, while our Sftp object uses the SFTP protocol. It would be useful to measure speed of PuTTY's psftp and Rebex Scp object to see how Sftp compares to psftp or how Scp compares to pscp. SCP might actually be faster than SFTP due to the very simple nature of the SCP protocol.

  • The aes256-ctr cipher is partly implemented as managed .NET code and is almost certainly a lot slower than PuTTY's implementation. Soon, we plan to add support for aes256-gcm@openssh.com which should be very fast on Windows. This might improve the speed a bit.

  • Using a LogWriter might negatively affect transfer speed as well, particularly at the Debug level.

  • Have you tried setting Sftp.Settings.DownloadQueueLength to very high value such as 128 or 256? In some scenarios, this has a noticeable effect on download speed.

  • We are currently working on a new version of our SSH core that should be faster and more scalable that the current one. This will be available within next year and should improve transfer speed as well.

...