Does Rebex SFTP library support the SCP -f switch (SCP1 mode)?

Question

Our product uses libssh2 (https://github.com/libssh2/libssh2) to transfer files between our controller and client computers. For whatever reason, libssh2 uses scp -f in its scpreceive method, which is called by the libssh2scp_recv() method.

Does the Rebex implementation support the -f switch, which, as described here (https://docstore.mik.ua/orelly/networking_2ndEd/ssh/ch03_08.htm) places the server into a mode where it uses the scp1 protocol, rather than the newer scp2 protocol.

If it does support scp1, what might be the server switches/configuration that might enable it?

When I execute the command from a Linux command line against the Rebex server, using the -f switch, the shell hangs, waiting for something. When I don't use the -f command, it completes quickly and properly:

scp -f -i keys.ppk snausages.txt laser@169.254.203.55:/home/laser/abcd.txt

I'm using the Rebex.FileServer library (6.0.8372) under Windows 10, .NET Core 3.1

--

Could you help?

Thank you.

Answer 1

TL;DR: Rebex File Server supports SFTP and SCP. If you want Linux scp command to use the classic SCP protocol (called "SCP1" in the book you linked) instead of SFTP (called "SCP2" in the book), then "-f" is not how it's done. Use the "-O" switch instead:

scp -O -i keys.ppk snausages.txt laser@169.254.203.55:/home/laser/abcd.txt

This assumes you are using a recent version of OpenSSH. On other systems or with old OpenSSH versions, the syntax might be different, but it will never involve the "-f" switch. The purpose of that switch is entirely different, as explained below.

---

The explanation at the beginning of https://docstore.mik.ua/orelly/networking_2ndEd/ssh/ch03_08.htm is somewhat confusing. The authors of that book invented their own terms ("SCP1" and "SCP2"), even though they don't actually exist. They use "SCP1" to refer to SCP, and they even acknowledge the non-existence of "SCP2" at that same page:

We suppose you could refer to SFTP as the "scp2 protocol," but we've never heard it and don't recommend it if you want to keep your sanity.

Here I 100% agree - those invented terms are confusing, and we should not use them. There is no "SCP2", and there is no "SCP1 mode". There are two file transfer protocols commonly used over SSH, and those are SFTP and SCP. But the scp command at the client-side often supports both SFTP and SCP, which is what the linked text is trying to explain.

So what about the "SCP -f" switch? The "3.8.1. scp1 Details" chapter aims to explain the inner workings of the SCP protocol, and it explains that "-f" or "-t" switches are only used internally when the client-side scp command invokes another instance of scp at the remote server. In order to do this, it passes one of those switches to the remote scp instance, which instructs it to start in SCP-server mode. Then, the client-side scp can start communicating with the server-side scp to upload or download data.

This can also be seen in the table below "3.8.1" - these switches are not actually used when starting scp at the client-side. The client-side command is simply scp foo server:bar or scp server:bar foo. Internally, these commands actually start scp -t bar or scp -f bar at the server, but they are not supposed to passed to scp from a Linux command line.

When you run scp -f -i keys.ppk snausages.txt laser@169.254.203.55:/home/laser/abcd.txt, you are mixing server-side and client-side syntax. By passing the "-f" option, you have actually started SCP in server-side mode, so instead of connecting to an SSH server and transferring the file, the local SCP instance thinks it's a server, and starts reading SCP client's commands from the standard input of the process. But there is no SCP client there, so those commands never arrive (unless you actually type them properly into the Linux console) and the process hangs.

Please see this very informative answer for further details regarding the client-side SCP and it's options.

If you would like to learn more about the inner working of SCP, including the purpose of "-f" and "-t" switches, check How the SCP protocol works. However, please note that even though this might be interesting to know, this knowledge is not needed at all in order to use SCP to actually transfer files.

Comments

Hi Lukas,

Thank you very much for your detailed answer, which told me that my debugging tactic was incorrect.

The current problem is that our program tries to use its own library to copy files to and from various controllers. When that library tries to contact the Rebex file server to copy a file, after the SSH parts are established, the LIBSSH2 library returns the error: "Failed to recv file".

This morning I turned on the LIBSSH2 tracing ability and see that for some reason SCP returns with:

0x0dd2d620 "[libssh2] 55.361585 SCP: got 02 scp: Invalid path"

I've told the Rebex component that my root folder is "C:\ProgramData\XYX\Product", both with and without a terminal  backslash. (Have I mentioned that it's running under Windows?)  

I do this in two places in my setup code:

When I add a user, (always named "laser"):
server.Users.Add(SFTPUserName, String.Empty, viewModelEmulatorSettings.DataRootFolder, ShellType.Scp);  // Add a user.

and when authentication passes and I accept the connection:

e.Accept(new FileServerUser(e.UserName, null, viewModelEmulatorSettings.DataRootFolder));

I then use the following paths to the files when calling into the libssh2_scp_recv method.

/home/laser/<filename>
 home/laser/<filename>

\home\laser\<filename>
 home\laser\<filename>

I even tried:
<filename> (no path prepended).

These result in the following Raw arguments when Rebex calls the ShellCommand event:

"-pf '/home/laser/Signature.txt'<00>��������������������������������������������������"
"-pf  'home/laser/Signature.txt'<00>������������������������������������������������"
"-pf '\\home\\laser\\Signature.txt'<00>��������������������������������������������������"
"-pf   'home\\laser\\Signature.txt'<00>������������������������������������������������"
"-pf 'Signature.txt'<00>��������������������������"

Do you see anything wrong with what I'm doing?

---

This looks like some kind of incompatibility between LIBSSH2's and Rebex File Server's SCP protocol. We are not currently testing LIBSSH2's SCP vs Rebex SCP, so we'll give it a try to see whether we can reproduce the issue. We'll try the sample code from https://www.libssh2.org/examples/scp.html first.

---

Please see my other answer below:
https://forum.rebex.net/22091/does-rebex-sftp-library-support-the-scp-f-switch-scp1-mode?show=22106#a22106

Answer 2

OK, I gave LIBSSH2's sample code a try. My results were slightly different than yours - instead of "<00>�����", I just got a single "<00>" at the end of raw arguments.

Further investigation revealed that the "<00>" sequence comes from our SSH string sanitization code, which replaces problematic characters with ASCII code less than 0x20 with the "" sequence based on their ASCII code.

This means that there is a bug in LIBSSH2's SCP implementation. Instead of starting the server-side "scp" command with "-pf '/home/laser/Signature.txt'" argument, LIBSSH2 appends a binary zero (null) character to the end of the argument string, and sends that to the server. And in your scenario, it also appends some additional non-printable characters as well.

You can view those extra characters if you enable more-verbose-than-verbose logging in FileServer:

server.LogWriter = new FileLogWriter("server-log.txt", (LogLevel)0);

Then, you can find decrypted contents of SSH's SSH_MSG_CHANNEL_REQUEST packet in the log:

2023-01-25 13:00:16.734 VERBOSE FileServer(1)[36] SSH: Session 1: Received packet SSH_MSG_CHANNEL_REQUEST (54 bytes).
2023-01-25 13:00:16.734 LEVEL0 FileServer(1)[36] SSH: Received packet data:
 0000 |62-00-00-00-00-00-00-00 04-65-78-65-63-01-00-00| b........exec...
 0010 |00-24-73-63-70-20-2D-70 66-20-27-2F-68-6F-6D-65| .$scp -pf '/home
 0020 |2F-6C-61-73-65-72-2F-53 69-67-6E-61-74-75-72-65| /laser/Signature
 0030 |2E-74-78-74-27-00                              | .txt'.

The trailing "00" should not be there. In your scenario, there will be even more characters following after that.

This behavior goes against the recommendation in RFC 4251, which states that:

  Strings are also used to store text.  In that case, US-ASCII is
  used for internal names, and ISO-10646 UTF-8 for text that might
  be displayed to the user.  The terminating null character SHOULD
  NOT normally be stored in the string.  For example: the US-ASCII
  string "testing" is represented as 00 00 00 07 t e s t i n g. 

Apparently, the bug went unnoticed because most SSH servers are written in C/C++ or other language where strings are terminated by a binary zero character. So if the client actually sends "00 00 00 07 t e s t i n g 00 01 23 45 67 89" or something like that, the SSH server's string manipulation routines would silently ignore the erroneous "00 01 23 45 67 89" sequence at the end. However, Rebex File Server is a .NET library, and .NET strings are not null-terminated. Therefore, LIBSSH2's SCP bug causes problems with our implementation.

We will work around this LIBSSH2 issue by treating the null character as the string termination. With this workaround, your LIBSSH2-based code will most likely start working. However, please be aware that there seems to be some nasty bug in LIBSSH2 that possibly makes it leak data in some buffer to the server that should not actually be sent, which might actually pose a security issue. Please consider reporting this to LIBSSH2 maintainers so they can fix it.

Comments

Hi Lukas,

Thanks very much again for your work and answer.

I've emailed the LIBSSH2 security team referencing this forum post and someone's responded.

Would you know when a release might be available incorporating the work around you mentioned?

---

Thanks! The next release will be published next week.