SSH: Rebex.Net.SshException: The connection was closed by the server.

Question

Hello,
we've been using the rebex library to connect to various sftp servers (customers) and this has been working without problems for years. Last week we started getting intermittent errors for some of them. We opened support tickets with the owners of the servers but so far no luck with finding out what the cause of issue is. We are able to connect/upload files to the affected servers manually using other sftp clients.

I'm including a partial verbose log below for one of the servers that fails (sensitive PHI/customer information removed). Full log appears to be too long to paste here.

The C# code we use to connect comes from the Rebex's example pages (wrapped in try/catch etc):

var sftp = new Rebex.Net.Sftp();
sftp.LogWriter = new Rebex.FileLogWriter(LogFilename, Rebex.LogLevel.Verbose);
sftp.Connect(ServerAddress, ServerPort);

Any help or additional insight will be appreciated

last section of the log:

2019-05-13 13:58:28.384 INFO Sftp(1)[10] Info: Current directory is 'HOME:/'.
2019-05-13 13:58:28.399 VERBOSE Sftp(1)[10] SSH: Sending packet SSHMSGCHANNELDATA (52 bytes).
0000 |5E-00-00-00-00-00-00-00 2B-00-00-00-27-03-00-00| ^.......+...'...
0010 |00-02-00-00-00-15-2F-53 56-43-43-54-65-6C-65-76| ....../xxxxxxxxx
0020 |6F-78-2E-74-78-74-2E-74 65-73-74-00-00-00-1A-00| xxxxxxxxxxx.....
0030 |00-00-00-05 | ....
2019-05-13 13:58:28.399 INFO Sftp(1)[10] Command: SSHFXPOPEN (2, '/xxxxxxxxxxxxxxxxxxxx', 26)
2019-05-13 13:58:29.040 VERBOSE Sftp(1)[10] SSH: Received packet SSHMSGCHANNELDATA (26 bytes).
0000 |5E-00-00-00-00-00-00-00 11-00-00-00-0D-66-00-00| ^............f..
0010 |00-02-00-00-00-04-00-00 00-00 | ..........
2019-05-13 13:58:29.040 INFO Sftp(1)[10] Response: SSHFXPHANDLE (2, 0x0)
2019-05-13 13:58:29.040 VERBOSE Sftp(1)[10] SSH: Sending packet SSHMSGCHANNEL_DATA (28710 bytes).
0000 |5E-00-00-00-00-00-00-07 39-00-00-07-35-06-00-00| ^.......9...5...
0010 |00-04-00-00-00-04-00-00 00-00-00-00-00-00-00-00| ................
0020 |70-00-00-00-07-1C-20-00 53-00-70-00-65-00-63-00| p..... .S.p.e.c.
0030 |69-00-66-00-69-00-65-00 64-00-22-00-2C-00-22-00| i.f.i.e.d.".,.".
0040 |48-00-22-00-2C-00-22-00 22-00-2C-00-22-00-30-00| H.".,.".".,.".0.
.
.
.
0710 |22-00-22-00-2C-00-22-00 22-00-2C-00-22-00-22-00| ".".,.".".,.".".
0720 |2C-00-22-00-22-00-2C-00 22-00-22-00-2C-00-22-00| ,.".".,.".".,.".
0730 |22-00-2C-00-22-00-22-00 2C-00-22-00-22-00-0D-00| ".,.".".,."."...
0740 |0A-00 | .. | ..
2019-05-13 13:58:48.119 ERROR Sftp(1)[10] SSH: Rebex.Net.SshException: The connection was closed by the server. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at Rebex.Net.ProxySocket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at Rebex.Net.SshSession.TR(KU A, Boolean B)
--- End of inner exception stack trace ---
at Rebex.Net.SshSession.TR(KU A, Boolean B)
at Rebex.Net.SshChannel.XB(KU A)
at Rebex.Net.SshChannel.Send(Byte[] buffer, Int32 offset, Int32 count)
2019-05-13 13:58:48.119 ERROR Sftp(1)[10] Info: Rebex.Net.SftpException: The server has closed the connection.
at Rebex.Net.GEB.JC(Int32 A)
at Rebex.Net.GEB.IC(GDB& A)
at Rebex.Net.GEB.IC(UInt32 A, Boolean B)
at Rebex.Net.Sftp.YT(DFB A, NDB B, String C, Stream D, Int64 E, Int64 H, KCB I)

Comments

Which version of Rebex SFTP do you use? Would it be possible to create a log (using LogLevel.Debug instead of LogLevel.Verbose) that shows the whole session? (Remove any sensitive data if needed, but please don't remove information about any packets. Alternatively, mail the logs to support@rebex.net)

Answer 1

(This response is based on a debug log we received via email.)

Thanks for the new log! Originally, I thought that parts of the log might be missing, but now it looks like the server disconnects the client soon after it receives the first data packet for the file. As you also report that this is occurring quite consistently based on file data, it looks like this is a server-side bug (the server violates the SFTP protocol – it should have reported any errors using appropriate SFTP error responses). To confirm this, try uploading a file (observed to cause errors) to the server using a third-party SFTP client (such as PuTTY’s psftp command line utility ) – if it fails as well, it would confirm the issue is server-side.

Unfortunately, if this is confirmed to be a server-side issue, there is very little an SFTP client can do about it – all it can really do is to only send files in a form that the server is willing to accept.

Additionally, the version of Rebex SFTP you use is more than four years old. It might be useful to determine whether the issue still occurs with the latest version.