Does BeginGetFile support simultaneous downloads?

0 votes
asked Jan 6, 2012 by Brad_Kremer (190 points)
edited Jun 13, 2012

I'm using Rebex.Net.Scp to read in groups of files. I was hoping that I could use BeginGetFile to start 10 asynchronous downloads, and then wait until all 10 have completed. I can do this, but it is the same amount of time as downloading each file serially with ReadMemory.

Is there a way to read multiple files simultaneously to speed up the transfer?

Thanks - Brad

commented Jan 6, 2012 by Lukas Matyska (60,650 points)
edited Jan 6, 2012

We are able to reproduce the issue, but have not the cause yet. We will keep you updated. Thanks for bringing it to our attention.

commented Jan 10, 2012 by Brad_Kremer (190 points)
edited Jan 10, 2012

I was able to get it working, but I get errors.
I'm trying to download a group of 10 files at a time. I created a new SCP object for each file I am downloading in the group. I have one SSH object that I Bind to each SCP. By doing this I can get it simultaneously download 10 files at a time. It is quite a bit faster, but I keep getting errors. I'm wondering if the SCP object is thread safe?

commented Jan 11, 2012 by Lukas Matyska (60,650 points)
edited Jan 11, 2012

Both Scp and SshSession should be thread safe. Unfortunately we have found some threading bugs. I am sending you current beta build at your email. Please give it a try.

1 Answer

0 votes
answered Jun 13, 2012 by Lukas Matyska (60,650 points)
edited Jun 13, 2012

We have performed speed tests against several servers and we have come to the conclusion that the performance depends on the server itself. With the same code, we got totally different results for different servers. Simply some servers can use a session at full and some cannot. You should do your own testing to determine which method is the fastest against your server. The offered methods are following:

  1. Make it whole synchronously (use one Scp object and call GetFile for each file)
  2. Use one Scp for all files (uses one SshSession), call BeginGetFile for each file
  3. Use one Scp for each file over one shared SshSession, call BeginGetFile on each Scp object
  4. Use one Scp for each file every with its own SshSession, call BeginGetFile on each Scp object
...