How to remove memory out exception while fetching large volume of files via SFTP?

0 votes
asked Feb 27 by alexander.junior (380 points)
edited Feb 27 by alexander.junior

I am fetching around 50000 files from a directory and downloading to another folder via sftp.

While fetching 10000 to 30000 files no such issue seen. But with 50000 files getting exception "ListException of type 'System.OutOfMemoryException' was thrown"

List ListSFTPFile = new List();
SftpItemCollection List = listSftp.GetList(GetSFTPServerPath("stfiles"));

                        foreach (SftpItem item in List)
                      {
                          objSFTPFile = new SFTPFile();
                          objSFTPFile.FileType = Enum.GetName(typeof(InwardMessageType), t);
                          objSFTPFile.Extension = Enum.GetName(typeof(CBExtensionTest), r);
                          objSFTPFile.Status = SFTPStatus.L.ToString();
                          objSFTPFile.FileName = item.Name.Trim();
                          objSFTPFile.FileSize = item.Size;
                          objSFTPFile.Mode = SFTPMode.D.ToString();
                          objSFTPFile.BankCode = Common.BANKCODE;
                          ListSFTPFile.Add(objSFTPFile);
                          Common.WriteLog(MethodInfo.GetCurrentMethod().Name + " FileType: " + objSFTPFile.FileType + " FileName: " + objSFTPFile.FileName, ServiceThread.Listing.ToString());

                      }
Applies to: Rebex SFTP

1 Answer

0 votes
answered Feb 27 by Lukas Matyska (43,090 points)

The Sftp.GetList() method keeps all the information about all items process in memory, which seems to be over your memory limits.

To solve this issue, handle items on-the-fly one-by-one. It can be done like this:

// register listing handler
client.ListItemReceived += (s, e) =>
{
    // process current item
    SftpItem item = e.Item;
    // ... e.g. Console.WriteLine(item.Name);

    // skip adding to resulting collection
    e.Ignore();
};

// perform listing
var list = client.GetList();

// the resulting collection contains no items
// due to e.Ignore() call in listing handler
Console.WriteLine(list.Count);

However, please note that by calling ListSFTPFile.Add(objSFTPFile); you also keep all the information about processed items in memory. This can also cause System.OutOfMemoryException.

commented Feb 27 by alexander.junior (380 points)
reshown Mar 1 by alexander.junior
. what i understood is to take files on the fly and there by free too much memory load to avoid memory exception

the code is not clear. can you explain it better
commented Feb 27 by Lukas Matyska (43,090 points)
Your code in on-the-fly processing can look like this:

    List ListSFTPFile = new List();

    // process items on-the-fly
    listSftp.ListItemReceived += (s, e) =>
    {
        SftpItem item = e.Item;
        objSFTPFile = new SFTPFile();
        objSFTPFile.FileType = Enum.GetName(typeof(InwardMessageType), t);
        objSFTPFile.Extension = Enum.GetName(typeof(CBExtensionTest), r);
        objSFTPFile.Status = SFTPStatus.L.ToString();
        objSFTPFile.FileName = item.Name.Trim();
        objSFTPFile.FileSize = item.Size;
        objSFTPFile.Mode = SFTPMode.D.ToString();
        objSFTPFile.BankCode = Common.BANKCODE;
        ListSFTPFile.Add(objSFTPFile);
        Common.WriteLog(MethodInfo.GetCurrentMethod().Name + " FileType: " + objSFTPFile.FileType + " FileName: " + objSFTPFile.FileName, ServiceThread.Listing.ToString());

        // skip adding item to resulting collection
        e.Ignore();
    };

    // clear resulting collection first
    ListSFTPFile.Clear();

    // issue directory listing
    listSftp.GetList(GetSFTPServerPath("stfiles"));
commented Feb 28 by alexander.junior (380 points)
I am getting list time out issue when trying files in the range  of 80k or so,
i have followed below code level:
_listSftp.ListItemReceived += (s, e) =>
                    {
                        SftpItem item = e.Item;
                        objSFTPFile = new SFTPFile();
                        objSFTPFile.FileType = "DDS300";
                        objSFTPFile.Extension = "DDR";
                        objSFTPFile.Status = SFTPStatus.L.ToString();
                        objSFTPFile.FileName = item.Name.Trim();
                        objSFTPFile.FileSize = item.Size;
                        objSFTPFile.Mode = SFTPMode.D.ToString();
                        objSFTPFile.BankCode = Common.BANKCODE;
                        ListSFTPFile.Add(objSFTPFile);
                        Common.WriteLog(MethodInfo.GetCurrentMethod().Name + " FileType: " + objSFTPFile.FileType + " FileName: " + objSFTPFile.FileName, ServiceThread.Listing.ToString());
                        // skip adding item to resulting collection
                        e.Ignore();
                    };

                    // clear resulting collection first
                    ListSFTPFile.Clear();

                    // issue directory listing
                    SftpItemCollection List = _listSftp.GetList(GetSFTPServerPath("st"));

and then saved to Table in sql where command time out also is given. (600 or so)
commented Feb 28 by Lukas Matyska (43,090 points)
Please, send communication log to support@rebex.net for analysis. It can be produced like this:

    // assign logger
    _listSftp.LogWriter = new Rebex.FileLogWriter("sftp.log", Rebex.LogLevel.Debug);

    // issue directory listing
    SftpItemCollection List = _listSftp.GetList(GetSFTPServerPath("st"));
...