When I am downloading a file from an SFTP site, how do i know that the file is complete? i.e. the file is not being currently uploaded into the server.

I tried with the Rebex FTP Library pack (uploading a 10 MB file to the SFTP site in one instance and downloading the same file on another instance) and it seems to download the complete file.

  1. Although the lib seems to work OK, are there any situations where i would end up with a partial file?
  2. What are the options available to mitigate this issue?

Thanks.

asked 30 Aug '11, 04:09

AdvStpDev's gravatar image

AdvStpDev
142
accept rate: 0%


It depends on the server and timing. Some server can response with "Access denied" if the file is not completed, other server can read up to the current length of the incomplete file.

You should use some synchronization design to mitigate the issue. I can think of these:

Create also .info file to indicate the file is being currently uploaded. Remove the file after upload is finished.

// upload process...
// crteate empty file .info indicating the file is being transferred
client.PutFile(new MemoryStream(), "file.txt.info");
// upload the file
client.PutFile("file.txt", "file.txt");
// remove the .info file to indicate the transfer is completed
client.DeleteFile("file.txt.info");

// download process...
// wait until the file is complete 
// adding some timeout check is needed to prevent deadlock
    // in case the upload process failed and the .info file remains on server
while (client.FileExists("file.txt.info"))
    Thread.Sleep(1000);
// download the file
client.GetFile("file.txt", "file.txt");

Or add the .part file extension to the file name and never download files with this extension. After finished upload rename the file as follows:

// upload the file
client.PutFile("file.txt", "file.txt.part");
// rename it
client.Rename("file.txt.part", "file.txt");

Or upload the file into special incoming directory, then move it to its appropriate location as follows:

// upload the file
client.PutFile("file.txt", "/incoming/file.txt");
// move it
client.Rename("/incoming/file.txt", "file.txt");
link

answered 30 Aug '11, 16:23

Lukas%20Matyska's gravatar image

Lukas Matyska ♦♦
90117
accept rate: 28%

Thanks Lukas for the quick response. Unfortunately, I don't have control over the file that is being uploaded. I am just a consumer and only have access to download the file placed in the site by the provider. The options you provided are good, but it requires the provider of the file to implement these. I am trying to see if I have some way of figuring this out from the client side. We have multiple provides and it might be harder to convince all of them to follow one (or a few) of these options. The upload/download example I mentioned above was just for testing purposes.

(30 Aug '11, 20:58) AdvStpDev

One option i can think of is to check (every few seconds) if the file size increased and if it didn't then probably the file is complete.

link

answered 30 Aug '11, 21:00

AdvStpDev's gravatar image

AdvStpDev
142
accept rate: 0%

Yes, this is definitely an option, but it can be "very" time expensive to be sure the upload process doesn't hang for a while. Therefore I didn't mentioned it in my answer, but it seems to be the only solution in your case.

(30 Aug '11, 22:40) Lukas Matyska ♦♦

Yes, I agree. It is expensive and might not be a scalable solution. The assumption here is that the simultaneous upload/download events are less frequent. If is not the case then I have to reevaluate my options.

(30 Aug '11, 23:44) AdvStpDev
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×141
×20

Asked: 30 Aug '11, 04:09

Seen: 375 times

Last updated: 30 Aug '11, 23:44