How to remove memory out exception while fetching large volume of files via SFTP?

Question

I am fetching around 50000 files from a directory and downloading to another folder via sftp.

While fetching 10000 to 30000 files no such issue seen. But with 50000 files getting exception "ListException of type 'System.OutOfMemoryException' was thrown"

List ListSFTPFile = new List();
SftpItemCollection List = listSftp.GetList(GetSFTPServerPath("stfiles"));

                        foreach (SftpItem item in List)
                      {
                          objSFTPFile = new SFTPFile();
                          objSFTPFile.FileType = Enum.GetName(typeof(InwardMessageType), t);
                          objSFTPFile.Extension = Enum.GetName(typeof(CBExtensionTest), r);
                          objSFTPFile.Status = SFTPStatus.L.ToString();
                          objSFTPFile.FileName = item.Name.Trim();
                          objSFTPFile.FileSize = item.Size;
                          objSFTPFile.Mode = SFTPMode.D.ToString();
                          objSFTPFile.BankCode = Common.BANKCODE;
                          ListSFTPFile.Add(objSFTPFile);
                          Common.WriteLog(MethodInfo.GetCurrentMethod().Name + " FileType: " + objSFTPFile.FileType + " FileName: " + objSFTPFile.FileName, ServiceThread.Listing.ToString());

                      }

Answer 1

The Sftp.GetList() method keeps all the information about all items process in memory, which seems to be over your memory limits.

To solve this issue, handle items on-the-fly one-by-one. It can be done like this:

// register listing handler
client.ListItemReceived += (s, e) =>
{
    // process current item
    SftpItem item = e.Item;
    // ... e.g. Console.WriteLine(item.Name);

    // skip adding to resulting collection
    e.Ignore();
};

// perform listing
var list = client.GetList();

// the resulting collection contains no items
// due to e.Ignore() call in listing handler
Console.WriteLine(list.Count);

However, please note that by calling ListSFTPFile.Add(objSFTPFile); you also keep all the information about processed items in memory. This can also cause System.OutOfMemoryException.

Comments

. what i understood is to take files on the fly and there by free too much memory load to avoid memory exception

the code is not clear. can you explain it better

---

Your code in on-the-fly processing can look like this:

    List ListSFTPFile = new List();

    // process items on-the-fly
    listSftp.ListItemReceived += (s, e) =>
    {
        SftpItem item = e.Item;
        objSFTPFile = new SFTPFile();
        objSFTPFile.FileType = Enum.GetName(typeof(InwardMessageType), t);
        objSFTPFile.Extension = Enum.GetName(typeof(CBExtensionTest), r);
        objSFTPFile.Status = SFTPStatus.L.ToString();
        objSFTPFile.FileName = item.Name.Trim();
        objSFTPFile.FileSize = item.Size;
        objSFTPFile.Mode = SFTPMode.D.ToString();
        objSFTPFile.BankCode = Common.BANKCODE;
        ListSFTPFile.Add(objSFTPFile);
        Common.WriteLog(MethodInfo.GetCurrentMethod().Name + " FileType: " + objSFTPFile.FileType + " FileName: " + objSFTPFile.FileName, ServiceThread.Listing.ToString());

        // skip adding item to resulting collection
        e.Ignore();
    };

    // clear resulting collection first
    ListSFTPFile.Clear();

    // issue directory listing
    listSftp.GetList(GetSFTPServerPath("stfiles"));

---

I am getting list time out issue when trying files in the range  of 80k or so,
i have followed below code level:
_listSftp.ListItemReceived += (s, e) =>
                    {
                        SftpItem item = e.Item;
                        objSFTPFile = new SFTPFile();
                        objSFTPFile.FileType = "DDS300";
                        objSFTPFile.Extension = "DDR";
                        objSFTPFile.Status = SFTPStatus.L.ToString();
                        objSFTPFile.FileName = item.Name.Trim();
                        objSFTPFile.FileSize = item.Size;
                        objSFTPFile.Mode = SFTPMode.D.ToString();
                        objSFTPFile.BankCode = Common.BANKCODE;
                        ListSFTPFile.Add(objSFTPFile);
                        Common.WriteLog(MethodInfo.GetCurrentMethod().Name + " FileType: " + objSFTPFile.FileType + " FileName: " + objSFTPFile.FileName, ServiceThread.Listing.ToString());
                        // skip adding item to resulting collection
                        e.Ignore();
                    };

                    // clear resulting collection first
                    ListSFTPFile.Clear();

                    // issue directory listing
                    SftpItemCollection List = _listSftp.GetList(GetSFTPServerPath("st"));

and then saved to Table in sql where command time out also is given. (600 or so)

---

Please, send communication log to support@rebex.net for analysis. It can be produced like this:

    // assign logger
    _listSftp.LogWriter = new Rebex.FileLogWriter("sftp.log", Rebex.LogLevel.Debug);

    // issue directory listing
    SftpItemCollection List = _listSftp.GetList(GetSFTPServerPath("st"));