How to increase time out for sftp while fetching large volume of files?

Question

I want to increase time out for sftp while fetching large volume of files. I am getting time out while fetching large data (say 80000 records or so).

Fetching done as below:
SftpItemCollection list = dSftp.GetList(strServerPath + "//" + "*." + extension);

Answer 1

The timeout should arise only in case the client didn't receive any data from the server for specified time. Directory listing is typically performed using multiple SSH_FXP_READDIR requests. Even for very large directories the timeout should not arise.

It seems that the server got stucked for some reason or it tries to send all listed items at once.

Can you please create communication log and send it to support@rebex.net for analysis. Based on the finding, we can provide you a solution.

The log can be created like this:

dSftp.LogWriter = new Rebex.FileLogWriter(@"C:\data\sftp.log", Rebex.LogLevel.Info);
SftpItemCollection list = dSftp.GetList(strServerPath + "//" + "*." + extension);

Comments

We tried to move  40000 files around or so from sftp server. Only 15271 files were downloaded. Since then nothing happened. No exception seen.

Set the sftp log writer and got below info:
2018-02-14 18:04:05.664 Opening log file.
2018-02-14 18:04:05.664 Using FileLogWriter version 2.5.6426.0.
2018-02-14 18:08:47.287 Opening log file.
2018-02-14 18:08:47.287 Using FileLogWriter version 2.5.6426.0.
2018-02-14 19:42:56.072 Opening log file.
2018-02-14 19:42:56.072 Using FileLogWriter version 2.5.6426.0.
2018-02-15 09:39:46.941 Opening log file.
2018-02-15 09:39:46.942 Using FileLogWriter version 2.5.6426.0.
2018-02-15 09:46:35.807 Opening log file.
2018-02-15 09:46:35.807 Using FileLogWriter version 2.5.6426.0.

---

Hi Rebex Team,.
We tried to  download  40000 files or so from the sftp server but only 15271 files were  downloaded. Log is writing but no exception seen either.

Set the communication long of SFTP as mentioned: Below is the info got

2018-02-14 18:04:05.664 Opening log file.
2018-02-14 18:04:05.664 Using FileLogWriter version 2.5.6426.0.
2018-02-14 18:08:47.287 Opening log file.
2018-02-14 18:08:47.287 Using FileLogWriter version 2.5.6426.0.
2018-02-14 19:42:56.072 Opening log file.
2018-02-14 19:42:56.072 Using FileLogWriter version 2.5.6426.0.
2018-02-15 09:39:46.941 Opening log file.
2018-02-15 09:39:46.942 Using FileLogWriter version 2.5.6426.0.
2018-02-15 09:46:35.807 Opening log file.
2018-02-15 09:46:35.807 Using FileLogWriter version 2.5.6426.0.

dt = fnGetFileList(ref isSFTPConnected);

private DataTable fnGetFileList(ref bool isSFTPConnected)
        {
            DataTable dt = new DataTable();
            string sftpLog = ConfigurationSettings.AppSettings["sftpLog"].ToString();//Added for rebex sftp file load test
            try
            {
               

                dSftp.LogWriter = new Rebex.FileLogWriter(sftpLog, Rebex.LogLevel.Info);
                dt.Columns.Add("FileType", typeof(string));
                dt.Columns.Add("FileName", typeof(string));
                dt.Columns.Add("Size", typeof(string));
                dt.Columns.Add("ModifiedDate", typeof(string));
                string[] sSFTPHost = ConfigurationSettings.AppSettings["SFTPHost"].Split('|');
               
                using (var client = new Sftp())
                {
                    try
                    {
                        // connect and log in
                        client.Connect(sSFTPHost[0], Convert.ToInt32(sSFTPHost[1]));
                        client.Login(sSFTPHost[2], sSFTPHost[3]);
                        /****Added by ABHIJITH  on 20171112 : Added Against CMG 2017 CHANGES***/
                        isSFTPConnected = true;
                    }
                    catch
                    {
                        isSFTPConnected = false;
                    }
             
                   
                    fnFileList("FTSFIMPath", "FTR", ref dt, client);
                    
                 }
             
            }
            catch (Exception ex)
            {
                dSftp.LogWriter = new Rebex.FileLogWriter(sftpLog, Rebex.LogLevel.Info);//Added for rebex sftp file load test
                Common.WriteFTSLog(MethodInfo.GetCurrentMethod().Name + ": " + ex.Message);
            }
          

            return dt;
        }

---

The log doesn't contain any information. It is because you assign the logger to `dSftp` instance, but then you use `client` instance.

Please associate the logger with instance, which do the work and send us produced log to support@rebex.net.