OutOfMemoryException in Rebex.Sftp.dll

Question

I'm trying to download a file of ~20 Mb from a SFTP Server and I'm getting this error after ~1 minute from starting.

Exception of type 'System.OutOfMemoryException' was thrown
at Rebex.Net.Sftp.QCW(CRQ U, String W, Stream R, Int64 I, Int64 Q, UWQ C)
at Rebex.Net.Sftp.VQW(String U, Stream W, Int64 R, Int64 I, UWQ Q)
at Rebex.Net.Sftp.GetFile(String remotePath, Stream outputStream)
at ENP.Services.RebexSftpService.GetFile(String filePath) in D:\Projects\ENP\Source\src\Integration\ENP.Services.RebexSftpService.cs:line 53

The code where I have implemented this:

public class RebexSftpService : ISftpService
{
    private readonly ILogger _logger;
    private bool _disposed;
    private Sftp _client;
    private string _host;
    private string _username;
    private string _password;

    public RebexSftpService(ILogger logger)
    {
        _logger = logger;
    }

    public Sftp SftpClient
    {
        get
        {
            if (_client == null)
            {
                _client = GetSftpClient();
            }

            if (!_client.IsConnected)
            {
                _client.Connect(_host);
                _client.Login(_username, _password);
            }

            return _client;
        }
    }

    public SftpResult GetFile(string filePath)
    {
        var result = new SftpResult { Errors = new List<string>() };

        try
        {
            using (var memStream = new MemoryStream())
            {
                SftpClient.GetFile(filePath, memStream);

                result.File = memStream.ToArray();
            }
        }
        catch (Exception ex)
        {
            _logger.Log(Level.Critical, ex.Message, ex);
            result.Errors.Add(ex.Message);
        }

        return result;
    }

    private Sftp GetSftpClient()
    {
        _host = ConfigurationManager.AppSettings["SftpHost"];
        _username = ConfigurationManager.AppSettings["SftpUserName"];
        _password = ConfigurationManager.AppSettings["SftpPassword"];

        var client = new Sftp
        {
            Settings = new SftpSettings
            {
                DisableProgressPercentage = true,
                DisablePathNormalization = true,
                DisablePutFileZeroOffsetTruncate = true,
                DisableRealPathWorkaround = true,
                DownloadBufferSize = 48,
                RaiseEventsFromCurrentThread = false,
                WaitForServerWelcomeMessage = false,
                UseLargeBuffers = true,
                TryPasswordFirst = true,
                DisableFxpStatWorkaround = true,
                EnableSignaturePadding = false
            }
        };

        client.Connect(_host);
        client.Login(_username, _password);

        return client;
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    protected virtual void Dispose(bool disposing)
    {
        if (!_disposed)
        {
            if (disposing)
            {
                if (_client != null)
                {
                    if (_client.IsConnected)
                    {
                        _client.Disconnect();
                    }

                    _client.Dispose();
                }
            }

            // Indicate that the instance has been disposed
            _disposed = true;
        }
    }

I'm disabling all those SftpSettings because the download is faster than not setting any setting. (e.g. no settings = 52 seconds download time. These settings, 22 seconds download time over a random file). But it still takes double time compared to WinSCP, which downloads the same file in 12 seconds.

Thanks!

Answer 1

This looks like something very strange is going on somewhere.

• Rebex SFTP doesn't actually take a lot of memory. Even with UseLargeBuffers option enabled, it's total memory usage should remain well below 20 MB the whole time. What's your machine's total amount of RAM? And how much memory does your application consume before the download is started? If you are consistently getting OutOfMemoryException errors from a simple application that doesn't contain much more that the code above, and you have ruled out other causes of the exception (such as accumulation of multiple result.File arrays), could you share it with us? If we were able to reproduce the issue, we would be quickly able to tell what is going on.

• MemoryStream is not suitable for large datasets. It uses continuous blocks of memory as its backing store, which leads to memory fragmentation and eventually OutOfMemoryExceptions. There are many replacement objects available such as this one (includes a more thorough description of the issue and its cause) or Microsoft's one.

• DownloadBufferSize setting specifies the size in bytes, not kilobytes. Setting it to 48 means that the actual buffer size used is only 4 KB (the minimum allowed). Please try 48 * 1024 instead.

• DownloadQueueLength setting affects the speed a lot on slow connections. Please try this (but only enable compression if the files you are transferring are actually compressible):

      UseLargeBuffers = true,
      UploadBufferSize = 32 * 1024,
      UploadQueueLength = 10,
      DownloadBufferSize = 48 * 1024,
      DownloadQueueLength = 10,
      SshParameters.Compression = true,
  

• DisablePathNormalization, DisablePutFileZeroOffsetTruncate, DisableRealPathWorkaround, TryPasswordFirst, WaitForServerWelcomeMessage and DisableFxpStatWorkaround settings don't affect transfer speed at all. Changing these values is not recommended unless you are trying to work around a specific issue (see the description of those settings).