How to know if all files were successfully transferred in SFTP simultaneous upload

Question

Hi,
I used single Sftp to upload files, i just execute Sftp.Upload() several times to upload multiple files.
I would like to closed the connection upon completion of all uploads, how should I do this?

Answer 1

With the Task-based asynchronous API, just call the async variants of your method as many times as you wish (in my example, I only have two tasks and I used GetFileAsync instead of UploadAsync method, but the same approach can be used universally), and then save the tasks into a field like this:

Sftp sftp = new Sftp();
sftp.Connect(server);
sftp.Login(username, password);

Task[] tasks = new Task[2];
tasks[0] = sftp.GetFileAsync("file.txt", @"c:\file.txt");
tasks[1] = sftp.GetFileAsync("file2.txt", @"c:\file2.txt");

Then use the Task.WaitAll method and then disconnect. However, the WaitAll method is "blocking" and it stops the execution of the program until all tasks are finished:

Task.WaitAll(tasks);
sftp.Disconnect();

Or you can take advantage of the Task.Factory.ContinueWhenAll which can be a better choice, cause it is not blocking, the disconnect will be executed once all tasks are finished.

var continuation = Task.Factory.ContinueWhenAll(
    tasks,
    (e) =>
    {
            sftp.Disconnect();
            Console.WriteLine("Disconnected.");
    });

    // continuation.Wait();

Comments

great solution!, though its a shame its only for .NET framework 4.0 and up.

---

In .NET 2.0 and .NET 3.5 there is the ManualResetEvent class. In combination with BeginGetFile/EndGetFile async operations of Rebex SFTP and the System.Threading.WaitHandle.WaitAll method it is possible to achieve the same purpose as with the Task.WaitAll method. Try this example:

        WaitHandle[] handles = new WaitHandle[2];
        Sftp _sftp = new Sftp();

        string server = "server";
        string username = "user";
        string password = "password";

        void ManualResetEventsExample()
        {
            // initialize
            handles[0] = new ManualResetEvent(false);
            handles[1] = new ManualResetEvent(false);

            // connect to server
            _sftp.Connect(server);
            _sftp.Login(username, password);

            // begin getfile operations
            _sftp.BeginGetFile("file.txt", @"c:\temp\file.txt", callback1, null);
            _sftp.BeginGetFile("file2.txt", @"c:\temp\file2.txt", callback2, null);

            // wait for all transfers to end - this is a blocking method
            System.Threading.WaitHandle.WaitAll(handles);

            Console.WriteLine("disconnecting...");
            _sftp.Disconnect();
        }

        void callback1(IAsyncResult ar)
        {
            _sftp.EndGetFile(ar);
            ((ManualResetEvent)handles[0]).Set();

            Console.WriteLine("file 1 downloaded.");
        }

        void callback2(IAsyncResult ar)
        {
            _sftp.EndGetFile(ar);
            ((ManualResetEvent)handles[1]).Set();

            Console.WriteLine("file 2 downloaded.");
        }

---

Hi Thomas,
I got that working, but I encountered problem using this approached.
Does ProblemDetected fire during async operation? I can't seem to fire this event.

---

Hi,

in my code example, I used the [single-file][1] GetFile method which does not fire the event (the [ProblemDetected][2] event is only fired in methods meant for [multi-file transfers][3] - namely sftp.Download, sftp.Upload, sftp.Delete and sftp.GetItems methods.

So if you change the BeginGetFile and EndGetFile in my above example to the Download method and then implement a ProblemDetected handler, the event should be working for you even with the async variants:

    _sftp.BeginDownload("file.txt", @"c:\temp\", 0, 0, 0, callback1, null);
    _sftp.BeginDownload("file2.txt", @"c:\temp\", 0, 0, 0, callback2, null);

    static void _sftp_ProblemDetected(object sender, SftpProblemDetectedEventArgs e)
        {
            if (e.Action == Rebex.IO.TransferAction.Downloading && e.ProblemType == Rebex.IO.TransferProblemType.FileExists)
                e.Overwrite();
        }

  [1]: https://www.rebex.net/sftp.net/features/single-file-operations.aspx
  [2]: https://www.rebex.net/doc/api/Rebex.Net.Sftp.ProblemDetected.html
  [3]: https://www.rebex.net/sftp.net/features/multiple-files-operations.aspx

---

I'm now using sftp.UploadAsync, is ProblemDetected fire for this?

---

Yes! It is fired when the problem is detected.

For instance this snippet fires the event for me (but only if the "file.txt" or "file2.txt" is already present in the current remote directory of the server that I am connecting to):

            Sftp sftp = new Sftp();
            sftp.ProblemDetected += sftp_ProblemDetected;
            sftp.Connect(server);
            sftp.Login(username, password);

            Task[] tasks = new Task[2];

            // upload to the current directory - if the file is already there, the ProblemDetected event will be fired
            tasks[0] = sftp.UploadAsync(@"c:\temp\file.txt", ".", 0, 0, 0);
            tasks[1] = sftp.UploadAsync(@"c:\temp\file2.txt", ".", 0, 0, 0);

            var continuation = Task.Factory.ContinueWhenAll(
                tasks,
                (e) =>
                {
                    sftp.Disconnect();
                    Console.WriteLine("Disconnected.");
                });

            continuation.Wait();
        }

        static void sftp_ProblemDetected(object sender, SftpProblemDetectedEventArgs e)
        {
            Console.WriteLine("ProblemDetected event fired.");
        }

---

thanks! all is working now :-)

Answer 2

Hi,

the Sftp.Upload method is meant to be used for uploading multiple files with just one call of the method. You can specify more files by using wildcards, for an example see this blogpost or SFTP multi file operations code snippets.

However, the Upload/Download method itself does not upload the files simultaneously (in parallel), but rather it takes the files one by one and uploads them serially. So once your Upload method finishes, you can be sure that all the files matching the search criteria have been transfered and close the connection then:

Sftp sftp = new Sftp();
sftp.Connect("server");
sftp.Login("user", "password");

// upload bunch of files
sftp.Upload("...", "...");
// upload another bunch of files
sftp.Upload("...", "...");

// uploads have been successfully finished
sftp.Disconnect();

And if you only need to upload/download one file with one method call, then the PutFile/GetFile methods are suitable for you.

Comments

Hi Tomas, i call sftp.Upload via separate thread, am I doing it wrong?
Also, thanks for the PutFile suggestion, i will try it out

---

Hi maynero,

Generally said, you do not have to call the Sftp.Upload method in separate thread (unless you need it for a special purpose, like non-blocking UI, etc.). If you call it in a separate thread, then you have to track from the original thread whether the upload thread has finished etc.

So, if you do not have a special need, just call the Upload method from the same thread. If you need the separate thread and you have problems with it, let me know then.

---

Basically, my project is to let user select several files in their local pc and upload it to sftp server (in different remote path) simultaneously, I was able to do that by calling sftp.Upload in different separate threads however my problem is when to close the sftp connection. I need to know if all upload are completed before I close the connection.