Sftp GetList() Message with invalid length 2209797 was received.

0 votes
asked Dec 30, 2014 by ganeshspy (900 points)
edited Jan 5, 2015

When I try to get list from specific path am getting this Sftp Exception.

SftpItemCollection sftpItemCollection = _sftp.GetList(path);

ErrorDescriptionForBug: Rebex.Net.SftpException, Message with invalid length 2209797 was received. ErrorStackTrace: at SftpComponent.GetDirectoryListing(String path)

In that folder contents more than 18000 files. This method works for ftp protocol fine but not sftp.

Any suggestion.

commented Jan 5, 2015 by Lukas Matyska (52,730 points)
edited Jan 5, 2015

I mistakenly read the values of internal buffer limit and the length of the reply of your server as 2GB instead of 2MB. Sorry.

I will post new answer and delete the old one, because it changes the situation.

1 Answer

0 votes
answered Jan 5, 2015 by Lukas Matyska (52,730 points)
edited Jan 5, 2015

The exception occurred because reply of the server exceeded internal threshold of 2MB per reply.

The SFTP draft says this about packet size:

The maximum size of a packet is in practice determined by the client ... All servers SHOULD support packets of at least 34000 bytes ...

In other words: clients should use max packet size = 34000B (and "polite" server should use the same).

Unfortunately, the packet size is not determined by the client in case of listing directories. The draft in this case says:

The files in a directory can be listed using the SSH_FXP_OPENDIR and SSH_FXP_READDIR requests. Each SSH_FXP_READDIR request returns one or more file names with full file attributes for each file. The client should call SSH_FXP_READDIR repeatedly until it has found the file it is looking for or until the server responds with a SSH_FXP_STATUS message indicating an error (normally SSH_FX_EOF if there are no more files in the directory).

This says that response to directory listing can be split among various number of SSH_FXP_READDIR packets. The reason is obvious: split large directory listing into small packets (normally 34000B at max).

So, the suggested way to solve this issue is to configure your SFTP server to not send replies larger than 34000B (of course if it is possible depends on capabilities of your SFTP server).

If configuring your server or using another server is not possible for you, we can add an option to specify max packet size. But this will not solve the problem in all cases. What if the directory contains e.g. 1.000.000 files?

Other clients will probably also fail to list such directory (e.g. WinSCP have internal buffer limited to 102400B). If the client doesn't have the internal buffer size limitation it can ends with OutOfMemoryException shutting your system down.

Actually, what is your SFTP server? We can report this issue to the vendor and discussed it with him personally, because we treat this as bug of the SFTP server.

...