performance issue with sftp direct stream access (GetUploadStream)

Question

I am working with rebex sftp (.NET 3.5) component for various types of sftp servers.

I have significant performance issue with direct write to the sftp stream.

follows my code snippet (elapsed time is for 16MB file and time is reset on each output)

Thank you.

using (Sftp client = new Sftp())
{
...
FileInfo localFile = ... ;

// [time... 00:00:00]

client.PutFile(localFile.FullName, remoteFile);

// [test #1, duration ... 00:00:07.35]

using (var src = localFile.OpenRead())   {
    using (var dst = client.GetUploadStream(remoteFile)) {
       var buffer = new byte[0x7000];
       while (true) {
       int read = src.Read(buffer, 0, buffer.Length);
       if (read <= 0) break;            
       dst.Write(buffer, 0, read);
     }

// [test #2, duration ... 00:00:45.26]

     src.Position = 0;
  client.PutFile(src, remoteFile);

// [test #3, duration ... 00:00:07.03]

}

Answer 1

The GetUploadStream method, unlike PutFile, was not optimised for speed.

When you call upload-stream's Write method, Rebex SFTP sends a single SSH_FXP_WRITE command to the server, waits for response, and only then it returns. On the other hand, PutFile initially sends multiple SSH_FXP_WRITE commands without waiting for server reply and when each reply is received, subsequent SSH_FXP_WRITE is sent, keeping the link saturated.

If you need a writable upload stream with PutFile-like speed, use BeginPutFile method with our experimental Pipe class:

            src.Position = 0;
            using (var pipe = new Rebex.IO.Pipe())
            {
                var dst = pipe.Input;
                var ar = client.BeginPutFile(pipe.Output, remoteFile, null, null);
                var buffer = new byte[0x7000];
                while (true)
                {
                    int read = src.Read(buffer, 0, buffer.Length);
                    if (read <= 0) break;
                    dst.Write(buffer, 0, read);
                }
                dst.Close();
                long n = client.EndPutFile(ar);
            }

            // [test #4, duration ... 00:00:??.??]

Please let me know whether this works!

Comments

Hi, we were experiencing the same performance issues with the Sftp.GetUploadStream() function. When we implemented the Pipe.cs class, the performance issues went away. Is the Pipe.cs file linked in the answer still experimental, or has it been vetted and safe to use if it's working in our implementation?

---

The Pipe.cs case has actually became a part of one of our forthcoming products and we have not found any problems in it so far. Therefore, it can be considered safe.

---

Hi, I found this answer looking for solutions to the same issue. We ran into a problem with the Pipe class, in which the last few writes don't make it across. It seems that when calling Pipe.Input.Close() it closes both input and output without waiting for the transfer to finish. We do call Flush() on the Input, but none of the Flush methods are implemented. Is there a newer version of the Pipe class? We have the File Transfer Pack, and I couldn't find the Pipe class defined anywhere in there.

---

The Pipe class is not part of our API.

You can implement Flush on the Input stream to wait until all data are read from Pipe. But, I think it is not necessary. The issue is probably somewhere else.

I suggest you to do this test:
  Count number of bytes written into Pipe and read from Pipe (inside Pipe.Read and Pipe.Write methods).
  Make the values public like this:
    public long TotalWritten { get { return _totalWritten; } }
    public long TotalRead { get { return _totalRead; } }

  After the line "long n = client.EndPutFile(ar);" compare values n, TotalWritten, TotalRead. They should be all equal. If not, please let us know the result.

I have done this test for couple of files in lengths from 1KB to 2GB. Always with success.

---

Can you also please share the info about server you are using and the version of Rebex components?

For example, Titan server has a bug, which causes invalid data upload when using 64KB blocks.
We added workaround for this in version 2017 R4 (see https://www.rebex.net/sftp.net/history.aspx#2017R4)

---

Thanks for the info. You were right, the problem was not in the Pipe class. The issue was that I was using
Sftp.PutFileAsync(pipe.Output, path).ContinueWith(t => pipe.Output.Close())
Which, of course, prematurely closes the stream that the sftp client is reading from. Thanks again!

Answer 2

Thank you for the response. I can change solution to the Put/Get File calls.

I had no idea that there is so huge performance difference in the usage. It would be helpful to mention it on your help pages: GetDownloadStream, GetFile, GetUploadStream, PutFile ...