Encoding.Unicode sftp issue

Question

Hi Guys:
I am concerned that rebex sftp is not able to save using the unicode format. Below shows how I create a unicode byte array which fails when sftping However similar code shows success using utf8. I am concern that we will be getting utf16 files, which is the default the .net linq xlement document uses for saving.

A) unicode example

    Dim client As New Sftp
    client.Connect(mFTPFOLDERINFO.SFTPSITE)    'hostname
    client.Login(mFTPFOLDERINFO.SFTPSITEUSER, mFTPFOLDERINFO.SFTPSITEPASSWORD)  'username, password
    client.Encoding = System.Text.Encoding.Unicode
    Try
        Dim message As String = "<?xml version='1.0' encoding='utf-16'?>" _
                               & "<CA_GT_IN>" _
                                 & "<VIN>VIN1</VIN>" _
                               & "</CA_GT_IN>"
        Dim data As Byte() = System.Text.Encoding.Unicode.GetBytes(message)
        Debug.WriteLine(System.Text.Encoding.Unicode.GetString(data))
        Dim ms1 As New System.IO.MemoryStream(data)
        client.PutFile(ms1, "message.xml")
        Return True
    Catch ex As Exception
        Debug.WriteLine(ex.Message)
        mErrMsg = ex.Message
        Return False
    End Try

The first debug statement gives: <?xml version='1.0' encoding='utf-16'?><ca_gt_in><vin>VIN1</vin></ca_gt_in> The error is No such file; 潎猠捵⁨楦敬.

---

However:
B) below works

    Dim client As New Sftp
    client.Connect(mFTPFOLDERINFO.SFTPSITE)    'hostname
    client.Login(mFTPFOLDERINFO.SFTPSITEUSER, mFTPFOLDERINFO.SFTPSITEPASSWORD)  'username, password
    client.Encoding = System.Text.UnicodeEncoding.UTF8
    Try
        Dim message As String = "<?xml version='1.0' encoding='utf-8'?>" _
                               & "<CA_GT_IN>" _
                                 & "<VIN>VIN1</VIN>" _
                               & "</CA_GT_IN>"
        Dim data As Byte() = System.Text.Encoding.UTF8.GetBytes(message)
        Debug.WriteLine(System.Text.Encoding.Unicode.GetString(data))
        Dim ms1 As New System.IO.MemoryStream(data)
        client.PutFile(ms1, "message.xml")
        Return True
    Catch ex As Exception
        Debug.WriteLine(ex.Message)
        mErrMsg = ex.Message
        Return False
    End Try

Answer 1

Hello,
there are two different encodings which doesn't relate to each other.

1. client.Encoding - specifies how to encode client commands and server responses (encoding of the communication between client and server)
2. Encoding.GetBytes(message) - specifies encoding of the message/data itself (which bytes are to be stored on the remote end)

Therefore correct usage is as follows:

Dim client As New Sftp
client.Connect(mFTPFOLDERINFO.SFTPSITE) ''hostname
client.Login(mFTPFOLDERINFO.SFTPSITEUSER, mFTPFOLDERINFO.SFTPSITEPASSWORD) ''username, password
client.Encoding = System.Text.UnicodeEncoding.UTF8 ''communication encoding     
Try
    Dim message As String = "<?xml version='1.0' encoding='utf-16'?>" _
                           & "<CA_GT_IN>" _
                             & "<VIN>VIN1</VIN>" _
                           & "</CA_GT_IN>"
    Dim data As Byte() = System.Text.Encoding.Unicode.GetBytes(message) ''data encoding
    Debug.WriteLine(System.Text.Encoding.Unicode.GetString(data))
    Dim ms1 As New System.IO.MemoryStream(data)
    client.PutFile(ms1, "message.xml")
    Return True
Catch ex As Exception
    Debug.WriteLine(ex.Message)
    mErrMsg = ex.Message
    Return False
End Try

Comments

Thanks, Do you work for Rebex and/or is there a Rebex person to talk to? Here is the issue. If you look at the xml it states the the encoding should be utf-16. Also To keep things consistent the System.Text class has 1) .UTF8Encoding 2) .UnicodeEncoding 3) .UTF32Encoding so yes I could use System.Text.UTF8Encoding which will work. But what is going on with using the memory stream to be saved as a utf-16 file. One other note we are using the Download setup package for .NET framework 2.0/3.0/3.5 and Visual Studio 2005/2008/2010 on a 64 bit operating system.

---

Actually, my apology to above code worked, Thanks

Answer 2

Hello, I am running into a similar issue w/ the latest 2012 release (trial) version. I have some files with latin characters in the name. I was ready to purchase but this is a blocking issue to that.

I am setting the encoding after the connect and login. _sftpClient.Encoding = Encoding.UTF8;

Then calling: _sftpClient.FileExists(remoteFileName) results in a file not found.

below is the sftp log.

thanks! -alp

2013-03-12 16:17:28.433 INFO Sftp(3)[38] Info: Server: SSH-2.0-OpenSSH_4.3
2013-03-12 16:17:28.433 INFO Sftp(3)[38] Info: Fingerprint: de:ae:ff:d8:3c:cb:99:27:41:78:46:52:21:f4:a3:e5
2013-03-12 16:17:28.433 INFO Sftp(3)[38] Info: Cipher info: SSH 2.0, DiffieHellmanGroupExchangeSHA1, DSS, aes256-cbc/aes256-cbc, hmac-sha1/hmac-sha1
2013-03-12 16:17:28.463 INFO Sftp(3)[38] Command: SSH_FXP_INIT (4)
2013-03-12 16:17:28.493 INFO Sftp(3)[38] Response: SSH_FXP_VERSION (3, 0 extensions)
2013-03-12 16:17:28.493 INFO Sftp(3)[38] Info: Using SFTP v3.
2013-03-12 16:17:28.493 INFO Sftp(3)[38] Command: SSH_FXP_REALPATH (1, '.')
2013-03-12 16:17:28.494 INFO Sftp(3)[38] Response: SSH_FXP_NAME (1, 1 item)
2013-03-12 16:17:28.494 INFO Sftp(3)[38] Info: Current directory is '/home/QA/contentowner'.
2013-03-12 16:17:28.494 INFO Sftp(3)[38] Command: SSH_FXP_REALPATH (2, '/home/QA/contentowner/testFiles/')
2013-03-12 16:17:28.494 INFO Sftp(3)[38] Response: SSH_FXP_NAME (2, 1 item)
2013-03-12 16:17:28.494 INFO Sftp(3)[38] Command: SSH_FXP_STAT (3, '/mnt/testFiles')
2013-03-12 16:17:28.495 INFO Sftp(3)[38] Response: SSH_FXP_ATTRS (3)
2013-03-12 16:17:28.496 INFO Sftp(3)[38] Command: SSH_FXP_STAT (4, '/mnt/testFiles/Pasión_logo')
2013-03-12 16:17:28.497 INFO Sftp(3)[38] Response: SSH_FXP_STATUS (4, 2, 'No such file')
2013-03-12 16:17:28.497 ERROR Sftp(3)[38] Info: Rebex.Net.SftpException: No such file; No such file.
    at gbMKS.PEQpz.eCVvJZ(1dRwomZ , Type )
    at gbMKS.PEQpz.1p4x0t(String , Boolean , Boolean , Boolean , String , SftpServerType , 78sj9 , Q1Py6Z& , ZD5WJ )
    at Rebex.Net.Sftp.2eZtM(String , Boolean , Boolean , Boolean , ZD5WJ )

Comments

Hello, please try GetNameList() method in the appropriate remote directory, try to identify the file with Latin characters and send to us the probably malformed name.

foreach (string item in _sftpClient.GetNameList())
{
    Console.WriteLine(item);
}

We might be able to tell you what encoding the server uses based on that. (The problem is that in Sftp version 3 no standard encoding has been defined by RFC, so the servers might use different encodings. In SFTP version 4 UTF8 is the standard).

---

Ugh, that is not cool. The getFilelist worked to get the file names:

2013-03-13 09:15:25,791 DEBUG [1363166124742] [ 5] -= File: Pasión logo.jpg
2013-03-13 09:15:25,794 DEBUG [1363166124742] [ 5] -= File: Pasión_logo.jpg

I was mistaken though, the error comes when we call sftpClient.GetFileDateTime("Pasión_logo.jpg")

So why does the GetFileList function work and the GetFileDateTime does not?

thanks! -alp

Here is the information from the servers we are using:

-bash-4.1$ uname -a
Linux hostname.net 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

-bash-4.1$ ssh -V
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010

[~]$ uname -a
Linux hostname.net 2.6.18-308.8.1.el5 #1 SMP Tue May 29 14:57:25 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
[~]$ ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

---

Thank you for your answer. Please create the communication log with the following code:

Sftp _sftpClient = new Sftp();
_sftpClient.LogWriter = new FileLogWriter(@"C:\temp\log.txt");
//Connect & Login

_sftpClient.ChangeDirectory(dir);
_sftpClient.FileExists("Pasión logo.jpg");

_sftpClient.GetFileDateTime("Pasión logo.jpg");

_sftpClient.Disconnect();

and post the log here so that we can investigate why GetFileDateTime fails in your case.

---

Cool, I had the log running here it is as well as the encoding of the strings. I used the EncodingTools project from codeproject. The paths might look funny but they do work with regular named files. The "testFiles" directory is a symlink in the path below. But we see the same behavior w/o a symlink too.

thanks!

2013-03-13 10:22:14,151 DEBUG [1363170132156] [33] -= File: PacificNissan.wmv Encoding: US-ASCII
2013-03-13 10:22:14,158 DEBUG [1363170132156] [33] -= File: Pasión logo.jpg Encoding: Western European (Windows)
2013-03-13 10:22:14,166 DEBUG [1363170132156] [33] -= File: Pasión_logo.jpg Encoding: Western European (Windows)
2013-03-13 10:22:14,186 DEBUG [1363170132156] [33] -= File: pavane.mov Encoding: US-ASCII

2013-03-13 10:22:12.532 Opening log file.
2013-03-13 10:22:12.565 INFO Sftp(1)[34] Info: Connecting to hostname.com:22 using Sftp 3.0.4700.0 (trial version).
2013-03-13 10:22:12.595 DEBUG Sftp(1)[34] SSH: Server is 'SSH-2.0-OpenSSH_4.3'.
2013-03-13 10:22:12.600 INFO Sftp(1)[34] SSH: Negotiation started.
2013-03-13 10:22:12.641 DEBUG Sftp(1)[34] SSH: Group exchange.
2013-03-13 10:22:12.645 DEBUG Sftp(1)[34] SSH: Negotiating key.
2013-03-13 10:22:12.733 DEBUG Sftp(1)[34] SSH: Validating signature.
2013-03-13 10:22:12.765 INFO Sftp(1)[34] SSH: Negotiation finished.
2013-03-13 10:22:12.765 INFO Sftp(1)[34] Info: Server: SSH-2.0-OpenSSH_4.3
2013-03-13 10:22:12.766 INFO Sftp(1)[34] Info: Fingerprint: de:ae:ff:d8:3c:cb:99:27:41:78:46:52:21:f4:a3:e5
2013-03-13 10:22:12.766 INFO Sftp(1)[34] Info: Cipher info: SSH 2.0, DiffieHellmanGroupExchangeSHA1, DSS, aes256-cbc/aes256-cbc, hmac-sha1/hmac-sha1
2013-03-13 10:22:12.787 DEBUG Sftp(1)[34] SSH: Allowed authentication methods: publickey, gssapi-with-mic, password.
2013-03-13 10:22:12.788 DEBUG Sftp(1)[34] SSH: Trying password authentication for 'dude'.
2013-03-13 10:22:12.794 DEBUG Sftp(1)[34] SSH: Authentication successful.
2013-03-13 10:22:12.804 DEBUG Sftp(1)[34] SSH: Requesting subsystem 'sftp'.
2013-03-13 10:22:12.817 INFO Sftp(1)[34] Command: SSH_FXP_INIT (4)
2013-03-13 10:22:12.859 INFO Sftp(1)[34] Response: SSH_FXP_VERSION (3, 0 extensions)
2013-03-13 10:22:12.860 INFO Sftp(1)[34] Info: Using SFTP v3.
2013-03-13 10:22:12.862 INFO Sftp(1)[34] Command: SSH_FXP_REALPATH (1, '.')
2013-03-13 10:22:12.866 INFO Sftp(1)[34] Response: SSH_FXP_NAME (1, 1 item)
2013-03-13 10:22:12.866 INFO Sftp(1)[34] Info: Current directory is '/home/QA/dude'.
2013-03-13 10:22:12.876 INFO Sftp(1)[34] Command: SSH_FXP_REALPATH (2, '/home/QA/dude/testFiles/')
2013-03-13 10:22:12.877 INFO Sftp(1)[34] Response: SSH_FXP_NAME (2, 1 item)
2013-03-13 10:22:12.879 INFO Sftp(1)[34] Command: SSH_FXP_STAT (3, '/mnt/testFiles')
2013-03-13 10:22:12.880 INFO Sftp(1)[34] Response: SSH_FXP_ATTRS (3)
2013-03-13 10:22:12.886 INFO Sftp(1)[34] Command: SSH_FXP_OPENDIR (4, '/mnt/testFiles')
2013-03-13 10:22:12.887 INFO Sftp(1)[34] Response: SSH_FXP_HANDLE (4, 0x0)
2013-03-13 10:22:12.889 INFO Sftp(1)[34] Command: SSH_FXP_READDIR (5, 0x0)
2013-03-13 10:22:12.898 INFO Sftp(1)[34] Response: SSH_FXP_NAME (5, 100 items)
2013-03-13 10:22:12.900 INFO Sftp(1)[34] Command: SSH_FXP_READDIR (6, 0x0)
2013-03-13 10:22:12.907 INFO Sftp(1)[34] Response: SSH_FXP_NAME (6, 82 items)
2013-03-13 10:22:12.908 INFO Sftp(1)[34] Command: SSH_FXP_READDIR (7, 0x0)
2013-03-13 10:22:12.909 INFO Sftp(1)[34] Response: SSH_FXP_STATUS (7, 1, 'End of file')
2013-03-13 10:22:12.910 INFO Sftp(1)[34] Command: SSH_FXP_CLOSE (8, 0x0)
2013-03-13 10:22:12.910 INFO Sftp(1)[34] Response: SSH_FXP_STATUS (8, 0, 'Success')
2013-03-13 10:22:15.017 INFO Sftp(1)[34] Command: SSH_FXP_STAT (9, '/mnt/testFiles/Pasión_logo')
2013-03-13 10:22:15.018 INFO Sftp(1)[34] Response: SSH_FXP_STATUS (9, 2, 'No such file')
2013-03-13 10:22:15.019 INFO Sftp(1)[34] Command: SSH_FXP_STAT (10, '/mnt/testFiles/Pasión_logo')
2013-03-13 10:22:15.020 INFO Sftp(1)[34] Response: SSH_FXP_STATUS (10, 2, 'No such file')
2013-03-13 10:22:15.023 ERROR Sftp(1)[34] Info: Rebex.Net.SftpException: No such file; No such file.
 at gbMKS.PEQpz.eCVvJZ(1dRwomZ , Type )
 at gbMKS.PEQpz.1p4x0t(String , Boolean , Boolean , Boolean , String , SftpServerType , 78sj9 , Q1Py6Z& , ZD5WJ )
 at Rebex.Net.Sftp.2eZtM(String , Boolean , Boolean , Boolean , ZD5WJ )

---

From the log it can be clearly seen that both FileExists and GetFileDateTime could not find 'Pasión_logo' file. But back to the original problem - we need to identify what encoding your SFTP server uses. Please run the following code and send us the verbose log it produces. We should be able to tell what encoding you have to set in order to be able to access files with special characters on your server:

        Sftp sftpClient = new Sftp();
        sftpClient.Connect("server");
        sftpClient.Login("username", "password");

        sftpClient.LogWriter = new FileLogWriter(@"C:\temp\logSftp.txt", LogLevel.Verbose);
        sftpClient.GetNameList();
        sftpClient.Disconnect();

---

The log file is pretty verbose! Pun intended. Can I mail it to you? We have license now as well, so no more trial.

thanks.

---

Yes you can mail the log to support@rebex.net.