Downloading a file that is being currently uploaded into the SFTP site.

Question

When I am downloading a file from an SFTP site, how do i know that the file is complete? i.e. the file is not being currently uploaded into the server.

I tried with the Rebex FTP Library pack (uploading a 10 MB file to the SFTP site in one instance and downloading the same file on another instance) and it seems to download the complete file.

1. Although the lib seems to work OK, are there any situations where i would end up with a partial file?
2. What are the options available to mitigate this issue?

Thanks.

Answer 1

// upload the file
client.PutFile("file.txt", "/file.txt.part");
// rename it
client.Rename("/file.txt.part", "/file.txt");

// upload the file
client.PutFile("file.txt", "/incoming/file.txt");
// move it
client.Rename("/incoming/file.txt", "/file.txt");

// upload process...
// create empty file .lock indicating the file is being transferred
client.PutFile(new MemoryStream(), "/file.txt.lock");
// upload the file
client.PutFile("file.txt", "/file.txt");
// remove the .lock file to indicate the transfer is completed
client.DeleteFile("/file.txt.lock");

// download process...
// wait until the file is complete 
// note: adding some timeout check is needed to prevent deadlock in case
//       the upload process failed and the .lock file remains on server
while (client.FileExists("/file.txt.lock"))
    Thread.Sleep(1000);
// download the file
client.GetFile("/file.txt", "file.txt");

Comments

Thanks Lukas for the quick response. Unfortunately, I don't have control over the file that is being uploaded. I am just a consumer and only have access to download the file placed in the site by the provider. The options you provided are good, but it requires the provider of the file to implement these. I am trying to see if I have some way of figuring this out from the client side. We have multiple provides and it might be harder to convince all of them to follow one (or a few) of these options. The upload/download example I mentioned above was just for testing purposes.

Answer 2

One option i can think of is to check (every few seconds) if the file size increased and if it didn't then probably the file is complete.

Comments

Yes, this is definitely an option, but it can be "very" time expensive to be sure the upload process doesn't hang for a while. Therefore I didn't mentioned it in my answer, but it seems to be the only solution in your case.

---

Yes, I agree. It is expensive and might not be a scalable solution. The assumption here is that the simultaneous upload/download events are less frequent. If is not the case then I have to reevaluate my options.