SFTP, LargeBuffers, and Large Object Heap

Question

Hello, I am using the sftp client with the large buffers option enabled. My application is a service that is heavily threaded and moves large files around (up too 20gb). I have had issues in the past with large object heap fragmentation with my old sftp client.

My question is, are the internal buffers that are used for transfer allocated up front when the client is created and then reused? Or are there new buffers created for each file transfer? I am using both Get/PutFile and GetStream.

If latter is true and new buffers are created for each transfer of a file, what is the default buffer size used for transferring with the large buffers option and without?

Thanks!

Answer 1

When UseLargeBuffers option is enabled, the TCP socket's receive buffer length is set to 4MB and send buffer length is set to 256KB. Without this option, default values are used instead. In addition to this, a 2MB receive buffer is allocated for an SSH channel with UseLargeBuffers (128KB without the option).

Regardless the option, an internal buffer of 32KB is allocated for SFTP session and 32KB/50KB buffers are allocated for the underlying SSH session.

PutFile - an internal buffer of 28KB is allocated for each transfer.

GetFile - no additional per/transfer buffer.

GetStream / Write - no additional per/transfer buffer.

GetStream / Read - a receive buffer (same size as number of received bytes) is allocated for each read call.

Advice: UseLargeBuffer only offers noticable advantages in some network configurations. In your case, disabling it might be more useful than keeping it. Also, when using the GetStream method, read/write operations are noticably slower than when using GetFile/PutFile methods - check out this Q&A for details.

Comments

Cool, thanks for the answer. Have you guys ever thought about with the useLargeBuffers option off, only using buffers less than 85kb so they will not be put on the large object heap?

---

Not really - no one asked until now. However, with a bit of additional code, you can try this with the current release - only one of the buffers is larger than 85KB (without UseLargeBuffers option) and it is possible to make it smaller:

// establish SSH session
SshSession session = new SshSession();
session.Connect("server01");
session.Authenticate("username", "password");

// open SFTP channel
SshChannel channel = session.OpenChannel(SshChannelType.Session, 0x10000);
channel.RequestSubsystem("sftp");

// bind
Sftp sftp = new Sftp();
sftp.Bind(channel.ToSocket());

// start using the sftp object now
sftp.Download("test.zip", @"c:\temp");
...