How to fix issue where reported size on SFTP is not the size of the downloaded file?

+1 vote
asked Apr 14, 2015 by bday-metasource (140 points)
edited Apr 22, 2015 by Lukas Pokorny

We are experiencing issues where files are not appearing to completely download using the Sftp client. We have experienced this with PGP and TIF files. We can decrycpt the PGP files and view the TIF files if we download them in FileZilla; however, when we download them with the Rebex library, the entire file is not downloading.

I'm hoping there is a setting I'm missing. Any help would be greatly appreciated.

Thanks,
~ Ben

C# CODE:

        var sftp = new Sftp
        {
            LogWriter = new ConsoleLogWriter(LogLevel.Debug)
        };
        var decryptServ = new DecyptionService();
        var TempDir = Directory.GetCurrentDirectory() + @"\temp\";

        if (!Directory.Exists(TempDir)) Directory.CreateDirectory(TempDir);

        //SFTP
        try
        {
            Console.WriteLine("- Start SFTP Process -");
            sftp.Connect(ftpServer);
            Console.WriteLine("- Connected -");
            sftp.Login(ftpUser, ftpPassword);
            Console.WriteLine("- Logged In -");

            var files = sftp.GetList(ftpTestLocation);

            foreach (var file in files)
            {
                if (Path.GetExtension(file.Name) == ".pgp")
                {
                    Console.WriteLine("\nStarting file: {0}", file.Path);
                    var fileInfo = new FileInfo(TempDir + file.Name);
                    if (fileInfo.Exists)
                    {
                        fileInfo.Delete();
                        fileInfo.Refresh();
                    }

                    // Console.WriteLine("Method: GetFile()");
                    // sftp.GetFile(file.Path, TempDir + file.Name);

                    Console.WriteLine("Method: Download()");
                    sftp.Download(file.Path, TempDir);

                    fileInfo.Refresh();

                    if (fileInfo.Length != file.Size)
                    {
                        Console.WriteLine(file.Name + " - File Sizes Don't Match.");
                        Console.WriteLine("FTP File Size: " + file.Size);
                        Console.WriteLine("Local File Size: " + fileInfo.Length);
                        Console.WriteLine();
                    }
                    else
                    {
                        Console.WriteLine("File sizes report the same on FTP and on Disk: {0}", fileInfo.Length);
                        Console.WriteLine();
                    }

                    Console.WriteLine("\n**************************************************\n");
                }
            }
        }
        catch (Exception e)
        {
            Console.WriteLine("Problem with SFTP: " + e.Message);
            Console.ReadLine();
        }
        finally
        {
            sftp.Disconnect();
        }

Here is some console output:

Starting file: /fooBar.zip.pgp
Method: Download()
2015-04-14 16:50:37.982 INFO Sftp(1)[10] Batch: Calling Download(string = '/fooBar.zip.pgp', string = 'C:\RebexSFTP_PGPTest\bin\Debug\temp\', TraversalMode = 'Recursive', TransferMethod = 'Copy', ActionOnExistingFiles = 'ThrowException').
2015-04-14 16:50:37.983 DEBUG Sftp(1)[10] Batch: Executing multi-file operation: Downloading, source = '/fooBar.zip.pgp', target = 'C:\RebexSFTP_PGPTest\bin\Debug\temp\', TransferMethod.Copy, MoveMode.All, LinkProcessingMode.FollowLinks, ActionOnExistingFiles.ThrowException.
2015-04-14 16:50:37.984 DEBUG Sftp(1)[10] Batch: Normalizing target path ('C:\RebexSFTP_PGPTest\bin\Debug\temp\').
2015-04-14 16:50:37.984 DEBUG Sftp(1)[10] Batch: Checking target path ('C:\RebexSFTP_PGPTest\bin\Debug\temp').
2015-04-14 16:50:37.985 DEBUG Sftp(1)[10] Batch: Normalizing source path ('/fooBar.zip.pgp').
2015-04-14 16:50:37.985 DEBUG Sftp(1)[10] Batch: Checking source path ('/fooBar.zip.pgp').
2015-04-14 16:50:37.986 INFO Sftp(1)[10] Command: SSH_FXP_LSTAT (18, '/fooBar.zip.pgp')
2015-04-14 16:50:37.987 INFO Sftp(1)[10] Response: SSH_FXP_ATTRS (18)
2015-04-14 16:50:37.988 DEBUG Sftp(1)[10] Batch: Multi-file operation started.
2015-04-14 16:50:37.988 DEBUG Sftp(1)[10] Batch: Transferring hierarchy started.
2015-04-14 16:50:37.988 DEBUG Sftp(1)[10] Batch: Processing file to transfer ('/fooBar.zip.pgp').
2015-04-14 16:50:37.989 DEBUG Sftp(1)[10] Batch: Checking existence of target item ('C:\RebexSFTP_PGPTest\bin\Debug\temp\fooBar.zip.pgp').
2015-04-14 16:50:37.990 DEBUG Sftp(1)[10] Batch: Transferring file ('/fooBar.zip.pgp' ---> 'C:\RebexSFTP_PGPTest\bin\Debug\temp\fooBar.zip.pgp').
2015-04-14 16:50:37.991 INFO Sftp(1)[10] Command: SSH_FXP_OPEN (19, '/fooBar.zip.pgp', 1)
2015-04-14 16:50:37.992 INFO Sftp(1)[10] Response: SSH_FXP_HANDLE (19, 0x31)
2015-04-14 16:50:37.993 DEBUG Sftp(1)[10] Command: SSH_FXP_READ (20, 0x31, 0, 28672)
2015-04-14 16:50:37.993 DEBUG Sftp(1)[10] Command: SSH_FXP_READ (21, 0x31, 28672, 28672)
2015-04-14 16:50:37.995 DEBUG Sftp(1)[10] Command: SSH_FXP_READ (22, 0x31, 57344, 28672)
2015-04-14 16:50:37.995 DEBUG Sftp(1)[10] Command: SSH_FXP_READ (23, 0x31, 86016, 28672)
2015-04-14 16:50:37.996 DEBUG Sftp(1)[10] Response: SSH_FXP_DATA (20, 28672 bytes)
2015-04-14 16:50:37.996 DEBUG Sftp(1)[10] Command: SSH_FXP_READ (24, 0x31, 114688, 28672)
2015-04-14 16:50:37.997 DEBUG Sftp(1)[10] Response: SSH_FXP_DATA (21, 28672 bytes)
2015-04-14 16:50:37.998 DEBUG Sftp(1)[10] Command: SSH_FXP_READ (25, 0x31, 143360, 28672)
2015-04-14 16:50:37.998 DEBUG Sftp(1)[10] Response: SSH_FXP_DATA (22, 28672 bytes)
2015-04-14 16:50:37.999 DEBUG Sftp(1)[10] Command: SSH_FXP_READ (26, 0x31, 172032, 28672)
2015-04-14 16:50:38.000 DEBUG Sftp(1)[10] Response: SSH_FXP_DATA (23, 0 bytes)
2015-04-14 16:50:38.000 DEBUG Sftp(1)[10] Info: Waiting for 3 outstanding read requests to arrive.
2015-04-14 16:50:38.001 DEBUG Sftp(1)[10] Response: SSH_FXP_DATA (24, 0 bytes)
2015-04-14 16:50:38.001 DEBUG Sftp(1)[10] Info: Waiting for 2 outstanding read requests to arrive.
2015-04-14 16:50:38.001 DEBUG Sftp(1)[10] Response: SSH_FXP_DATA (25, 14443 bytes)
2015-04-14 16:50:38.002 DEBUG Sftp(1)[10] Info: Waiting for 1 outstanding read requests to arrive.
2015-04-14 16:50:38.002 INFO Sftp(1)[10] Response: SSH_FXP_STATUS (26, 1, 'EOF')
2015-04-14 16:50:38.003 INFO Sftp(1)[10] Command: SSH_FXP_CLOSE (27, 0x31)
2015-04-14 16:50:38.005 INFO Sftp(1)[10] Response: SSH_FXP_STATUS (27, 0, 'Ok')
2015-04-14 16:50:38.006 DEBUG Sftp(1)[10] Batch: Multi-file operation done.
fooBar.zip.pgp - File Sizes Don't Match.
FTP File Size: 157803
Local File Size: 86016
Applies to: Rebex SFTP

1 Answer

+2 votes
answered Apr 15, 2015 by Tomas Knopp (58,580 points)
edited May 11, 2015 by Lukas Pokorny

This issue has been encountered by client using Globalscape EFT Server Enterprise 7.1.0.11. It is caused by an unfortunate combination of a six-year-old bug in Rebex SFTP and a newly introduced bug in GlobalScape 7.1.x.

Update: In Rebex SFTP 2015 R4, we fixed our bug and added a workaround for the GlobalScape bug.

Are you actually using this server? If not, please let us know the name of the server you are using.

As a workaround please try to disable the transfer queue before downloading the file like this:

Sftp sftp = new Sftp();
sftp.Settings.DisableTransferQueue = true;

// transfer files
sftp.Download(...);

And let me know whether you able to download the files succesfully with the DisableTransferQueue property enabled.

commented Apr 15, 2015 by bday-metasource (140 points)
Tomas, thank you for your response!

Yes, it would appear that the server I'm attempting to download from is Globalscape EFT Server Enterprise 7.1.0.11.  I also used the work around and it worked like a charm.

Thanks!
commented Apr 20, 2015 by kwatson (100 points)
We encountered the same issue where it was only downloading 140KB. Disabling this did work.

Has this been raised with Globalscape EFT and do they have a patch to fix this  ? Do they have an ETA ?

I see it is not a recommended approach to turn this setting off according to the documentation
commented Apr 20, 2015 by Lukas Pokorny (88,510 points)
One of our clients reported this issue to Globalscape about two weeks ago, but we are not aware of its current status.

Seeing that this is not an isolated issue, we will try contacting Globalscape as well.

You are right that turning the setting off is not recommended - it slows down transfer speeds. However, it's currently the only known workaround for this issue. We will let you know when there is a better solution!
commented Aug 19, 2015 by Lukas Pokorny (88,510 points)
In Rebex SFTP 2015 R4 (http://www.rebex.net/sftp.net/history.aspx#5700), we fixed our bug and added a workaround for the GlobalScape bug.
commented Aug 19, 2015 by kwatson (100 points)
Globalscape patched the issue and we ended up not having to make any changes
commented Aug 20, 2015 by Lukas Pokorny (88,510 points)
Thanks for letting us know! The workaround is only applied when the issue is encountered (regardless the server vendor) and does not negatively affect downloads when it's not, so upgrading to 2015 R4 would do no harm either.
...