I'm not 100% certain yet, but this seems to resemble an issue our clients encounter once in a while. Some badly behaved firewalls or routers silently close the main FTP connection while a data transfer is in progress because of inactivity - even though the main FTP connection is actually supposed to be inactive while the transfer is in progress (it uses a separate TCP connection).
Rebex FTP includes a workaround for this issue, but it doesn't work reliably with all FTP servers so it's disabled by default. To enable it, use this code:
client.Options |= FtpOptions.KeepAliveDuringTransfer; // C#
client.Options = client.Options Or FtpOptions.KeepAliveDuringTransfer ' VB.NET
This will cause a dummy NOOP command to be sent every 600 seconds (10 minutes) - you can also specify a longer interval by changing the value of Ftp object's KeepAliveDuringTransferInterval property.
Does enabling this option help?