Sftp client after 80 file downloads distributed over one week "exhausted"

Question

Hello,

I do the following (quite vanilla I think)

Sftp ftp = new Sftp();
ftp.Timeout = 30000;
ftp.Settings.UseLargeBuffers = true;
ftp.Settings.SshParameters.PreferredHostKeyAlgorithm = SshHostKeyAlgorithm.ECDsaNistP521;
ftp.Connect(server, Sftp.DefaultPort);
ftp.Login(username, password);
ftp.Download( "tenfiles*.csv" , templocalfolder , TraversalMode.MatchFilesShallow);
ftp.Disconnect();

With rebex sftp class I am uploading a 1.5Gbyte file every morning and I am downloading roundabout 10 to 15 files every evening (filesizes 400 to 800kbyte).

Version of rebex class is 5.0.70270.0 from Nov 2019.

It is running on Windows Server 2016 with every security updates in place.

This was running very smooth and healthy in the last 18 month.
With only one reboot per month (when I had to install the security updates).

Since November security update (I think this is the cause) this works for 6 days.

After 6 days I get the following error:

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
Rebex.Net.SftpException: The operation was not completed within the specified time limit. ---> Rebex.Net.SshException: The operation was not completed within the specified time limit.
at Rebex.Net.SshSession.yfpe[a,b](ioix2 aff, Int32 afg, ioiy afh, b afi, a afj, a afk) at Rebex.Net.SshSession.yfpd[a,b](ioix2 afd, b afe)
at Rebex.Net.SshChannel.bnkb[a,b](ioix`2 wr, b ws)
at Rebex.Net.SshChannel.Receive(Byte[] buffer, Int32 offset, Int32 count)
at bsdk.ocjc(Byte[] m, Int32 n, Int32 o)
at bsdz.lctu(Int32 arq)
at bsdz.lctw(rhth& art)
at bsdz.lctt(UInt32 aro, Boolean arp)
at bsdz.lcsv(UInt32 aon, bsdj aoo)
at Rebex.Net.Sftp.wnav(bsdj sb, String sc, Stream sd, Int64 se, Int64 sf, rhua sg)
--- End of inner exception stack trace ---
at Rebex.Net.Sftp.wnav(bsdj sb, String sc, Stream sd, Int64 se, Int64 sf, rhua sg)
at Rebex.Net.Sftp.wnat(FileMode ro, bsdj rp, String rq, String rr, Int64 rs, Int64 rt, Int64 ru, rhua rv)
at bsdu.rllx(FileMode amw, String amx, String amy, Int64 amz, Int64 ana)
at bsdu.rdgr(rhtu bev, rhtu bew, String bex, String bey, Boolean bez)
at rhtx.rdfn(String ece, String ecf, rhtu ecg, rhtu ech, rhtu eci, rhtu ecj, Boolean eck, Boolean ecl, Boolean ecm)
at rhtx.rdfm(rhtv ecd)
at rhtx.rdfj()
at rhtx.rdfe(TransferAction ebo, rhtt ebp, String ebq, TransferMethod ebr, MoveMode ebs, LinkProcessingMode ebt, ActionOnExistingFiles ebu, rhtu ebv)
at Rebex.Net.Sftp.wnbt(TransferAction ut, bsdj uu, bsdt uv, String uw, TransferMethod ux, MoveMode uy, LinkProcessingMode uz, ActionOnExistingFiles va)
at Rebex.Net.Sftp.Download(String remotePath, String localDirectoryPath, TraversalMode traversalMode, TransferMethod transferMethod, ActionOnExistingFiles existingFileMode)
at MyStuff.ReportingHelper.GrabReportingFromServer()
at MyStuff.ReportingHelper.ProcessReporting(DateTime pdt, Int32 ptransactionnumber)

************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3890.0 built by: NET472REL1LAST_B

CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll

System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3835.0 built by: NET472REL1LAST_B

CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GACMSIL/System.Windows.Forms/v4.04.0.0.0__b77a5c561934e089/System.Windows.Forms.dll

System
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3650.0 built by: NET472REL1LAST_B

CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GACMSIL/System/v4.04.0.0.0__b77a5c561934e089/System.dll

System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1

CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GACMSIL/System.Drawing/v4.04.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll

System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3630.0 built by: NET472REL1LAST_B

CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GACMSIL/System.Configuration/v4.04.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll

System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1

CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GACMSIL/System.Xml/v4.04.0.0.0__b77a5c561934e089/System.Xml.dll

Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.7.3062.0 built by: NET472REL1

CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GACMSIL/Accessibility/v4.04.0.0.0__b03f5f7f11d50a3a/Accessibility.dll

When I start my exe after this error and only want to download 10 files it is very slow.
After a reboot of this machine I have normal speeds for one week.
After one week of operation it starts again with a very slow download speed (until it didn't work because of timeouts).

I can buy a new license of rebex - this is not a problem.

I want back the rock solid operation from two month ago (please, this makes me gray hairs).

Best regards,
Thomas

Answer 1

Could you please clarify the versions of Rebex SFTP you have been using? There is no v5.0.70270.0 from Nov 2019, so I guess you might be using v5.0.7027 from March 2019. Is that correct?

Anyway, this is the first time we encountered an issue that seems to have been triggered by Windows Server 2016 November security update. It's quite likely the update might have introduced an issue that is beyond our control, and the observation that issues persist even after the application has been restarted also points in that direction.

But before we try to find out more, please be aware that v5.0.7027 is rather old, and that we addressed several SSH-related deadlock issues in later releases: 5.0.7290, 5.0.7320 and 5.0.7357. Some of these have been present for years, but only started manifesting due to addition of faster ciphers or OS tweaks. Therefore, it would be useful to try upgrading to a more recent release. Your support contract ended in August 2020, which means you can still download v5.0.7501 from July 2020, where all these issues have been fixed. Could you please give this version a try and let us know whether it behaves any differently?

Comments

Thank you for the fast response.

Sorry for this typo.
Yes it was v5.0.7027

I upgraded my source to v5.0.7501

I let it run for one week and look if it is better now.

I let you know.

Dec 14th is Windows patch day perhaps with this patch day it is getting better.

Do you think Garbage Collection after uploading and downloading could help?

---

Thanks, please let us know when you get any results. This looks more like a networking issue than GC issue. But calling GC.Collect after uploading/downloading most is unlikely to make things worse, so you might as well go ahead to give it a try.

Also, if things go wrong again, it might be useful to create a debug log (see https://www.rebex.net/kb/logging/ for details) - this would hopefully make it possible to learn more details about the issue.

---

Further details

I migrated to .net framework 4.8
and rebex v5.0.7501

My dev-machine:
windows 11 pro fully patched on a dsl-connection
4 seconds for downloading of 20 files with 500kb each file

My prod-machine:
win server 2016 fully patched on a 1gbit connection directly in a datacenter
19 seconds for downloading of 20 files with 500kb each file

This is on the ftp-client side with the sftp-server with low load.


When the sftp-server has a lot of load:

My dev-machine:
15 seconds for downloading of 20 files with 500kb each file

My prod-machine:
!200! seconds for downloading of 20 files with 500kb each file
(and partly with timeouts - filezilla has the same problems on the prod machine - filezilla and rebex on the dev-machine at the same time is nice and fast)

With this sftp server I am forced to use eliptic curve.
When I connect to a different sftp server with more relaxed encryption algo there is no performance difference. (so the LAN driver / raw TCP/IP-Stack is not damaged)


I have the following crypto recommendation(upfront):

ftp.Settings.SshParameters.PreferredHostKeyAlgorithm = SshHostKeyAlgorithm.ECDsaNistP521;

After handshaking this is it:

INFO Sftp(1)[1] Info: Cipher info: SSH 2.0, ecdh-sha2-nistp256, ssh-dss, aes256-ctr/aes256-ctr, hmac-sha2-256/hmac-sha2-256


This is the list of supported encryption algos sent from the sftp server

ecdh-sha2-nistp256,  ecdh-sha2-nistp384,  ecdh-sha2-nistp521 , diffie-hellman-group-exchange-sha256 ,
diffie-hellman-group-exchange-sha1 , diffie-hellman-group14-sha1  ,  diffie-hellman-group1-sha1,
rsa1024-sha1

ssh-rsa  ,  ssh-dss

aes256-ctr ,  aes192-ctr , aes128-ctr ,  aes256-cbc, aes192-cbc  ,  aes128-cbc ,  blowfish-ctr,
blowfish-cbc  , cast128-cbc  ,  arcfour256  ,  arcfour128  ,  3des-ctr  ,  3des-cbc
 
aes256-ctr ,  aes192-ctr  ,  aes128-ctr  ,  aes256-cbc ,  aes192-cbc , aes128-cbc , blowfish-ctr ,
blowfish-cbc  , cast128-cbc ,  arcfour256 , arcfour128 , 3des-ctr,  3des-cbc

jhmac-sha2-256  , hmac-sha2-512 ,  hmac-sha1 ,  hmac-sha1-96 , hmac-md5 ,  hmac-md5-96,
hmac-ripemd160 , umac-64@openssh.com
 

jhmac-sha2-256 , hmac-sha2-512 , hmac-sha1 , hmac-sha1-96 , hmac-md5 , hmac-md5-96,
hmac-ripemd160 , umac-64@openssh.com


perhaps if i enforce every single algo to a middle class encryption algo by hand?

---

This is actually quite unexpected - host key algorithms (such as ssh-rsa or ssh-dsa) and key exchange algorithms (such as ecdh-sha2-nistp256 or diffie-hellman-group14-sha1) are only used during the SSH protocol's negotiation phase, which only occurs after connecting to the server (before authentication), and then it occurs again only after lot of data (such as 2GB) has been transferred or after the session has been active for many hours. Are you sure changing these ciphers indeed makes a difference?

On the other hand, symmetric encryption algorithms (such as aes256-cbc or aes256-ctr) and MAC ciphers (such as hmac-sha2-256 or hmac-sha1) are used all the time.

Also, ciphers using CTR mode (such as "aes256-ctr") are among the slowest in Rebex SFTP, because their implementation is partially in managed code. We made it a bit faster in Rebex SFTP R5.2 (https://www.rebex.net/sftp.net/history.aspx#R5.2), so it might be useful to try that version (or one of the more recent ones). Trying to force "aes256-cbc" for comparison might be useful as well.

---

This sftp server is made for a security aware environment and the operator of this sftp server is prioritizing security over performance.


I enforced this:
ftp.Settings.SshParameters.PreferredHostKeyAlgorithm = SshHostKeyAlgorithm.ECDsaNistP256;
ftp.Settings.SshParameters.EncryptionAlgorithms = SshEncryptionAlgorithm.AES;
ftp.Settings.SshParameters.EncryptionModes = SshEncryptionMode.CBC;


And as a result now I get this:
INFO Sftp(1)[1] Info: Cipher info: SSH 2.0, ecdh-sha2-nistp256, ssh-dss, aes256-cbc/aes256-cbc, hmac-sha2-256/hmac-sha2-256

So now I have aes256-cbc

And now I get the files downloaded in 8 seconds.

Your class is really versatile with all this.
Thank you.