0 votes
by (160 points)
edited by

I am looking for the pros and cons of reading each individual file as a stream vs. pulling all the files to a local disk and reading them from there.

In my situation I need to:

  1. Read the files from a remote server,
  2. Update a database with information from the files
  3. Save those files to another remote location,
  4. Delete them from the original SFTP server.

I know myClient.Download has the ability to perform Tasks 1,3 and 4
in one line of code but then I have to read from another hard-disk (not using SFTP).

so I am wondering of these two methods, is one more performant or reliable with multiple files? Is there anything else I should consider?

https://www.rebex.net/doc/api/Rebex.Net.Sftp.html

Applies to: Rebex SFTP

1 Answer

+1 vote
by (72.7k points)
selected by
 
Best answer

You can download files from SFTP server using 3 methods:

  1. Sftp.Download()
  2. Sftp.GetFile()
  3. Sftp.GetDownloadStream()

There should be no significant performance differences between the methods for large files (the code for transferring data is same).

The methods offer different functionality:

  1. Sftp.Download() - High level API for downloading multiple files or directories, use can use wildcards in path, you can specify Overwrite mode, etc.
  2. Sftp.GetFile() - Simple API for downloading single file (to file or stream).
  3. Sftp.GetDownloadStream() - Low level API which enables you to access remote file on stream basis. You can seek to specific location, read block by block, etc.

It depends on your needs:

  1. If you want to download multiple files in single line of code, I suggest you to use Sftp.Download().
  2. If you want to download single file (or one part of it), I suggest you to use Sftp.GetFile() method.
  3. If you want access remote file on stream basis or you want to download more than one part of it, I suggest you to use Sftp.GetDownloadStream().
by (160 points)
I appreciate the clarification. In my situation I am not dealing with a few large files but many small ones. I was actually wondering if there was any kind of magic-batch-processing that Sft.Download() might be doing to reduce server load when downloading multiple files.
by (72.7k points)
No, there is no magic-batch-processing in Sft.Download(). Performance should be comparable for single Sftp.Download() call or multiple Sftp.GetFile() calls.
...