0 votes
by (270 points)
edited

When I am downloading a file from an SFTP site, how do i know that the file is complete? i.e. the file is not being currently uploaded into the server.

I tried with the Rebex FTP Library pack (uploading a 10 MB file to the SFTP site in one instance and downloading the same file on another instance) and it seems to download the complete file.

  1. Although the lib seems to work OK, are there any situations where i would end up with a partial file?
  2. What are the options available to mitigate this issue?

Thanks.

Applies to: Rebex SFTP

2 Answers

+1 vote
by (73.6k points)
edited by

It depends on the server and timing. Some server can response with "Access denied" if the file is not completed, other server can read up to the current length of the incomplete file.

You should use some synchronization design to mitigate the issue. I can think of these:

Add the .part file extension to the file name and never download files with this extension. After finished upload rename the file as follows:

// upload the file
client.PutFile("file.txt", "/file.txt.part");
// rename it
client.Rename("/file.txt.part", "/file.txt");

Or upload the file into special incoming directory, then move it to its appropriate location as follows:

// upload the file
client.PutFile("file.txt", "/incoming/file.txt");
// move it
client.Rename("/incoming/file.txt", "/file.txt");

Or create the .lock file to indicate the file is being currently uploaded. Remove the file after upload is finished.

// upload process...
// create empty file .lock indicating the file is being transferred
client.PutFile(new MemoryStream(), "/file.txt.lock");
// upload the file
client.PutFile("file.txt", "/file.txt");
// remove the .lock file to indicate the transfer is completed
client.DeleteFile("/file.txt.lock");

// download process...
// wait until the file is complete 
// note: adding some timeout check is needed to prevent deadlock in case
//       the upload process failed and the .lock file remains on server
while (client.FileExists("/file.txt.lock"))
    Thread.Sleep(1000);
// download the file
client.GetFile("/file.txt", "file.txt");

If none of the above is possible, you can try re-download with timeout approach as described here.

by (270 points)
edited

Thanks Lukas for the quick response. Unfortunately, I don't have control over the file that is being uploaded. I am just a consumer and only have access to download the file placed in the site by the provider. The options you provided are good, but it requires the provider of the file to implement these. I am trying to see if I have some way of figuring this out from the client side. We have multiple provides and it might be harder to convince all of them to follow one (or a few) of these options. The upload/download example I mentioned above was just for testing purposes.

0 votes
by (270 points)
edited

One option i can think of is to check (every few seconds) if the file size increased and if it didn't then probably the file is complete.

by (73.6k points)
edited

Yes, this is definitely an option, but it can be "very" time expensive to be sure the upload process doesn't hang for a while. Therefore I didn't mentioned it in my answer, but it seems to be the only solution in your case.

by (270 points)
edited

Yes, I agree. It is expensive and might not be a scalable solution. The assumption here is that the simultaneous upload/download events are less frequent. If is not the case then I have to reevaluate my options.

...