SSH SFTP connection problems

Question

Hello,

I need your help/clarification on something as I'm not an expert on the matter. We have used Rebex File Transfer Pack for .NET for connecting to FTP servers without issues, but now I need to integrate with partner's SFTP server from my environment and I'm running into problems. Namely, the server differentiates between pure SSH and SFTP (and I thought that SFTP is FTP over SSH protocol and that there is no difference there) and when you connect over SSH (PuTTY's plink) you are allowed only one set of commands and when you connect over SFTP (PuTTY's psftp) you can do completely different things. The problem I'm running into is that according to Rebex logs, the connection and authentication to their server is always established as SSH, the when I try to upload a file (only allowed for SFTP connections) I get "command not allowed" exception as a response. I have tried using Rebex.Net.Ssh() and Rebex.Net.Sftp object but the behavior is the same. I'm I missing something here? Some Rebex configuration, some other object types, etc.?

Thanks for your help

Comments

Hello,

actually, SFTP is a completely different protocol than FTP. SFTP usually runs over SSH channels, but it was designed as transport independent protocol. You can dive into the details by reading our knowledge base article about the differences between SFTP and FTP.

As SFTP (usually) runs on top of SSH secure channel, I agree that connecting with an SSH client and with a SFTP client actually enables you to do different things.

If you need to do a file upload through SFTP protocol, the Rebex.Net.Sftp class is what you need. (Not the Rebex.Net.Ssh object which is used to work with the underlying SSH and will not enable you to use the Sftp protocol functionality).

Please try the following code to upload a file to an SFTP server and let us know whether it works:

Sftp sftp = new Sftp();
sftp.LogWriter = new Rebex.FileLogWriter(@"c:\temp\log.txt", Rebex.LogLevel.Debug);

sftp.Connect("server");
sftp.Login("user", "password");
sftp.PutFile(@"C:\temp\myfile.txt", "myfile.txt");
sftp.Disconnect();

If you are unable to upload the file using the above code snippet, please either post the created log of communication here or send it to support@rebex.net. The log would enable us to see what is going on and help you then. Thanks!

Answer 1

Thanks for your aditional information. The fact that your SFTP server is a mainframe and that it uses a somewhat special paths was the important but missing piece to the puzzle. I will answer your questions one by one:

1. Rebex Sftp client normalizes paths by default which would replace the doubleslash "//" with a single slash '/'). However, the normalization of paths can be disabled with the DisablePathNormalization option. So to upload a file into "//AAA.BBB.CCC" directory of your remote server, please write the following code: sftp.Settings.DisablePathNormalization = true; sftp.PutFile(@"C:\temp\myfile.custom_ext", "//AAA.BBB.CCC//myfile.custom_ext");

2. 'ls' command normally does a listing of a directory but it looks as though your SFTP server "exploits" the command for some kind of setting. Anyway, you can use our GetList method to execute 'ls /+recfm=vb,mode=text,lrecl=4004' like this:

   sftp.GetList("/+recfm=vb,mode=text,lrecl=4004");

3. Yes, it is true the connection is first established as SSH and only after that the "SFTP subsystem is invoked"...

Please let us know if you were able to upload the file after disabling the path normalization as suggested above. If not please send us a new log of communication of the failing transfer.

Comments

Dude, you are the BEST! I knew about DisablePathNormalization, but this thing with GetList was pure magic :D

Answer 2

Hello Tomas,

Thank you for your prompt response. The code you suggested is the exact same code that I have tried.

I have actually used sftp.Upload method, not sftp.PutFile

Attached is the log from that operation. I have also tried to execute some allowed commands, ls, cd, but I was only able to do that using Ssh() object because only it has the RunCommand method.

I have analysed the log with partner's tech analyst and his conclusion was the that the problem was in that I'm connecting over SSH but trying to run commands that are only allowed when you connect over SFTP. In the log, line 3, you see that is says connecting using Sftp 3.0.5298.0 but after that, on the next line it's SSH: Sending data:

I have tried using other vendor's trial library and I didn't run into this issues.

///LOG

    2014-10-06 06:02:17.987 Opening log file.
    2014-10-06 06:02:17.987 Using FileLogWriter version 2.0.5298.0.
    2014-10-06 06:02:18.018 INFO Sftp(1)[1] Info: Connecting to xxx.xxx.xxx.xxx:22 using Sftp 3.0.5298.0 (trial version).
    2014-10-06 06:02:18.096 VERBOSE Sftp(1)[1] SSH: Sending data:
     0000 |53-53-48-2D-32-2E-30-2D 52-65-62-65-78-53-53-48| SSH-2.0-RebexSSH
     0010 |5F-33-2E-30-2E-35-32-39 38-2E-30-0D-0A         | _3.0.5298.0..
    2014-10-06 06:02:29.766 VERBOSE Sftp(1)[1] SSH: Received data:
     0000 |53-53-48-2D-32-2E-30-2D 4F-70-65-6E-53-53-48-5F| SSH-2.0-OpenSSH_
     0010 |35-2E-30-0A                                    | 5.0.
    2014-10-06 06:02:29.766 DEBUG Sftp(1)[1] SSH: Server is 'SSH-2.0-OpenSSH_5.0'.
    2014-10-06 06:02:29.766 INFO Sftp(1)[1] SSH: Negotiation started.
    2014-10-06 06:02:29.797 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_KEXINIT (879 bytes).
    ///
    //lines removed
    //
    2014-10-06 06:02:29.875 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_KEXINIT (311 bytes).
    ///
    //lines removed
    //
    2014-10-06 06:02:29.891 DEBUG Sftp(1)[1] SSH: Group exchange.
    2014-10-06 06:02:29.891 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_KEX_DH_GEX_REQUEST (13 bytes).
     0000 |22-00-00-04-00-00-00-08 00-00-00-10-00         | "............
    2014-10-06 06:02:30.000 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_KEX_31 (267 bytes).
    ///
    //lines removed
    //
    2014-10-06 06:02:30.000 DEBUG Sftp(1)[1] SSH: Negotiating key.
    2014-10-06 06:02:30.156 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_KEX_DH_GEX_INIT (261 bytes).
    ///
    //lines removed
    //
    2014-10-06 06:02:36.194 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_KEX_DH_GEX_REPLY (820 bytes).
    ///
    //lines removed
    //
    2014-10-06 06:02:36.334 DEBUG Sftp(1)[1] SSH: Validating signature.
    2014-10-06 06:02:36.381 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_NEWKEYS (1 bytes).
     0000 |15                                             | .
    2014-10-06 06:02:36.381 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_NEWKEYS (1 bytes).
     0000 |15                                             | .
    2014-10-06 06:02:36.397 INFO Sftp(1)[1] SSH: Negotiation finished.
    2014-10-06 06:02:36.397 INFO Sftp(1)[1] Info: Server: SSH-2.0-OpenSSH_5.0
    2014-10-06 06:02:36.397 INFO Sftp(1)[1] Info: Fingerprint: xxxxxxx
    2014-10-06 06:02:36.397 INFO Sftp(1)[1] Info: Cipher info: SSH 2.0, DiffieHellmanGroupExchangeSHA256, RSA, aes256-cbc/aes256-cbc, hmac-sha1/hmac-sha1
    2014-10-06 06:02:36.397 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_SERVICE_REQUEST (17 bytes).
     0000 |05-00-00-00-0C-73-73-68 2D-75-73-65-72-61-75-74| .....ssh-useraut
     0010 |68                                             | h
    2014-10-06 06:02:36.412 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_SERVICE_ACCEPT (17 bytes).
     0000 |06-00-00-00-0C-73-73-68 2D-75-73-65-72-61-75-74| .....ssh-useraut
     0010 |68                                             | h
    2014-10-06 06:02:36.412 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_USERAUTH_REQUEST (39 bytes).
     0000 |32-77-77-77-08-52-44-33 35-41-94-72-57-77-00-00| 2....==username==...
     0010 |0E-73-73-68-2D-63-6F-6E 6E-65-63-74-69-6F-6E-00| .ssh-connection.
     0020 |00-00-04-6E-6F-6E-65                           | ...none
    2014-10-06 06:02:36.521 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_USERAUTH_BANNER (1834 bytes).
    ///
    //lines removed
    ///
    2014-10-06 06:02:36.521 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_USERAUTH_FAILURE (24 bytes).
     0000 |33-00-00-00-12-70-75-62 6C-69-63-6B-65-79-2C-70| 3....publickey,p
     0010 |61-73-73-77-6F-72-64-00                        | assword.
    2014-10-06 06:02:36.521 DEBUG Sftp(1)[1] SSH: Allowed authentication methods: publickey, password.
    2014-10-06 06:02:36.521 DEBUG Sftp(1)[1] SSH: Trying password authentication for '==username=='.
    2014-10-06 06:02:36.521 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_USERAUTH_REQUEST (56 bytes).
     0000 |32-77-77-77-08-52-44-33 35-41-94-72-57-77-00-00| 2....==username==...
     0010 |0E-73-73-68-2D-63-6F-6E 6E-65-63-74-69-6F-6E-00| .ssh-connection.
     0020 |00-11-11-11-11-11-11-77 6F-11-11-11-11-11-00-08| ...password.....
     0030 |11-11-88-88-99-99-96-98                       | ==password==
    2014-10-06 06:02:36.553 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_USERAUTH_SUCCESS (1 bytes).
     0000 |34                                             | 4
    2014-10-06 06:02:36.553 DEBUG Sftp(1)[1] SSH: Authentication successful.
    2014-10-06 06:02:36.553 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_CHANNEL_OPEN (24 bytes).
     0000 |5A-00-00-00-07-73-65-73 73-69-6F-6E-00-00-00-00| Z....session....
     0010 |00-02-00-00-00-00-40-00                        | ......@.
    2014-10-06 06:02:36.631 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_CHANNEL_OPEN_CONFIRMATION (17 bytes).
     0000 |5B-00-00-00-00-00-00-00 00-00-00-00-00-00-00-80| [...............
     0010 |00                                             | .
    2014-10-06 06:02:36.631 DEBUG Sftp(1)[1] SSH: Requesting subsystem 'sftp'.
    2014-10-06 06:02:36.631 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_CHANNEL_REQUEST (27 bytes).
     0000 |62-00-00-00-00-00-00-00 09-73-75-62-73-79-73-74| b........subsyst
     0010 |65-6D-01-00-00-00-04-73 66-74-70               | em.....sftp
    2014-10-06 06:02:36.646 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_CHANNEL_WINDOW_ADJUST (9 bytes).
     0000 |5D-00-00-00-00-00-20-00 00                     | ]..... ..
    2014-10-06 06:02:36.646 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_CHANNEL_SUCCESS (5 bytes).
     0000 |63-00-00-00-00                                 | c....
    2014-10-06 06:02:36.662 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_CHANNEL_DATA (18 bytes).
     0000 |5E-00-00-00-00-00-00-00 09-00-00-00-05-01-00-00| ^...............
     0010 |00-04                                          | ..
    2014-10-06 06:02:36.662 INFO Sftp(1)[1] Command: SSH_FXP_INIT (4)
    2014-10-06 06:02:39.408 VERBOSE Sftp(1)[1] SSH: Received packet SSH_MSG_CHANNEL_DATA (82 bytes).
     0000 |5E-00-00-00-00-00-00-00 49-00-00-00-45-02-00-00| ^.......I...E...
     0010 |00-03-00-00-00-18-70-6F 73-69-78-2D-72-65-6E-61| ......posix-rena
     0020 |6D-65-40-6F-70-65-6E-73 73-68-2E-63-6F-6D-00-00| me@openssh.com..
     0030 |00-01-31-00-00-00-16-63 6F-7A-2D-7A-6F-70-74-73| ..1....coz-zopts
     0040 |40-64-6F-76-65-74-61-69 6C-2E-63-6F-6D-00-00-00| @dovetail.com...
     0050 |01-30                                          | .0
    2014-10-06 06:02:39.423 INFO Sftp(1)[1] Response: SSH_FXP_VERSION (3, 2 extensions)
    2014-10-06 06:02:39.423 INFO Sftp(1)[1] Info: Using SFTP v3 on a Unix-like platform.
    2014-10-06 06:02:39.423 VERBOSE Sftp(1)[1] SSH: Sending packet SSH_MSG_CHANNEL_DATA (23 bytes).
     0000 |5E-00-00-00-00-00-00-00 0E-00-00-00-0A-10-00-00| ^...............
     0010 |00-01-00-00-00-01-2E                           | .......
    2014-10-06 06:02:39.423 INFO Sftp

Comments

Hello Milan,

thank you for the log. I think the problem is that you are trying to upload into a non-existing remote directory. The Upload method actually cannot upload directly to a non-existing directory. So before uploading with the Upload method, please make sure that the remote SFTP directory ('==to_dir==' in your case) exists and if it does not exist, first create it with

sftp.CreateDirectory method.

Please see the below code:

Sftp sftp = new Sftp();
sftp.Connect("server");
sftp.Login("user", "pass");

string remoteDir = "==to_dir==";

if (!sftp.DirectoryExists(remoteDir))
    sftp.CreateDirectory(remoteDir);

sftp.Upload(@"c:\temp\file.txt", remoteDir);
sftp.Disconnect();

Please let me know if it helped to solve the issue.

---

I do not have a permission to create dir on remote server. The server is a mainframe with virtual boxes and I can only execute ls, cd, put and get. The remote dir path looks like

"//AAA.BBB.CCC"

and that is the only location I have permission to upload a file. When using PuTTY I would execute

put C:\temp\myfile.custom_ext //AAA.BBB.CCC

Now, when we are discussing this in detail, before uploading a file, every time I establish a connection I would need to execute

ls /+recfm=vb,mode=text,lrecl=4004

How would I do that with Sftp object? SftpAttributes do not offer help with this. That's why I was trying to use Ssh.RunCommand.

Anyway, I think we are distancing ourselves from what I believe is the real problem, is it true that when I use Sftp object, the connection is first established as SSH and then after that "SFTP subsystem is invoked"? At least that is how I'm interpreting the log line

2014-10-06 06:02:36.631 DEBUG Sftp(1)[1] SSH: Requesting subsystem 'sftp'.

---

What I forgot to mention, after cross referencing my Rebex logs with logs from the server (that I don't have access to and cannot share them with you), we were able to conclude that we are trying to execute commands not allowed for SSH connections. Like I said in the first post, server differentiates over SSH and SFTP connections. When you establish a connection over SSH you cannot do a put/get commands.

---

What utility are you using for issuing the put/get commands? If it is the psftp utility then the put and get commands are only commands of the utility and they translate to the commands of the SFTP protocols, so they correspond to Rebex Sftp PutFile and GetFile commands.