Downloading a file that is being currently uploaded into the SFTP site.

0 votes
asked Aug 30, 2011 by AdvStpDev (270 points)
edited Aug 30, 2011

When I am downloading a file from an SFTP site, how do i know that the file is complete? i.e. the file is not being currently uploaded into the server.

I tried with the Rebex FTP Library pack (uploading a 10 MB file to the SFTP site in one instance and downloading the same file on another instance) and it seems to download the complete file.

  1. Although the lib seems to work OK, are there any situations where i would end up with a partial file?
  2. What are the options available to mitigate this issue?

Thanks.

Applies to: Rebex SFTP

2 Answers

+1 vote
answered Aug 30, 2011 by Lukas Matyska (44,610 points)
edited Aug 30, 2011

It depends on the server and timing. Some server can response with "Access denied" if the file is not completed, other server can read up to the current length of the incomplete file.

You should use some synchronization design to mitigate the issue. I can think of these:

Create also .info file to indicate the file is being currently uploaded. Remove the file after upload is finished.

// upload process...
// crteate empty file .info indicating the file is being transferred
client.PutFile(new MemoryStream(), "file.txt.info");
// upload the file
client.PutFile("file.txt", "file.txt");
// remove the .info file to indicate the transfer is completed
client.DeleteFile("file.txt.info");

// download process...
// wait until the file is complete 
// adding some timeout check is needed to prevent deadlock
    // in case the upload process failed and the .info file remains on server
while (client.FileExists("file.txt.info"))
    Thread.Sleep(1000);
// download the file
client.GetFile("file.txt", "file.txt");

Or add the .part file extension to the file name and never download files with this extension. After finished upload rename the file as follows:

// upload the file
client.PutFile("file.txt", "file.txt.part");
// rename it
client.Rename("file.txt.part", "file.txt");

Or upload the file into special incoming directory, then move it to its appropriate location as follows:

// upload the file
client.PutFile("file.txt", "/incoming/file.txt");
// move it
client.Rename("/incoming/file.txt", "file.txt");
commented Aug 30, 2011 by AdvStpDev (270 points)
edited Aug 30, 2011

Thanks Lukas for the quick response. Unfortunately, I don't have control over the file that is being uploaded. I am just a consumer and only have access to download the file placed in the site by the provider. The options you provided are good, but it requires the provider of the file to implement these. I am trying to see if I have some way of figuring this out from the client side. We have multiple provides and it might be harder to convince all of them to follow one (or a few) of these options. The upload/download example I mentioned above was just for testing purposes.

0 votes
answered Aug 30, 2011 by AdvStpDev (270 points)
edited Aug 30, 2011

One option i can think of is to check (every few seconds) if the file size increased and if it didn't then probably the file is complete.

commented Aug 30, 2011 by Lukas Matyska (44,610 points)
edited Aug 30, 2011

Yes, this is definitely an option, but it can be "very" time expensive to be sure the upload process doesn't hang for a while. Therefore I didn't mentioned it in my answer, but it seems to be the only solution in your case.

commented Aug 30, 2011 by AdvStpDev (270 points)
edited Aug 30, 2011

Yes, I agree. It is expensive and might not be a scalable solution. The assumption here is that the simultaneous upload/download events are less frequent. If is not the case then I have to reevaluate my options.

...