Issues with Direct Connection to SFTP Server

Question

Hello,

I've implemented some code in a script component destination in BIDS, and the job actually runs fine until it hits a certain threshold, then fails. The code is roughly as follows:

client.Connect("xxxxxx", 12345)
client.Login("xxxxxx", "yyyyyy")
sw = New StreamWriter(client.GetUploadStream("test1.txt"))
sw.Write(Row.ZZZZ)

It tosses an error that says: "Error: 0xC0047062 at Emp Data, Script Component [4981]: System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine". It fails after about 19,700 rows written, and almost always on a date of some kind, though that could be a coincidence. The detail error is:

ScriptComponent_0f80d51f9c68465b85a87853e8d3d5a0 at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at Rebex.Net.ProxySocket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at Rebex.Net.SshSession.BXQTfX(Int32 )
at Rebex.Net.SshSession.AePCoRZ(Byte[]& ) at Rebex.Net.SshSession.cDtmAOZ(cMbfbTZ , Object[] ) at Rebex.Net.SshChannel.cDtmAOZ(Boolean , cMbfbTZ , Object[] ) at Rebex.Net.SshChannel.Receive(Byte[] buffer, Int32 offset, Int32 count)
at wWGvS.AzRMspZ.BMJzdYZ(Byte[] , Int32 , Int32 ) at wWGvS.APnbEo.BXQTfX(Int32 ) at wWGvS.APnbEo.CFLiXJ(BBdsJfZ& ) at wWGvS.APnbEo.CFLiXJ(UInt32 ) at wWGvS.BlCLrNZ.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value) at ScriptComponent_0f80d51f9c68465b85a87853e8d3d5a0.ScriptMain.Input0_ProcessInputRow(Input0Buffer Row) at ScriptComponent_0f80d51f9c68465b85a87853e8d3d5a0.UserComponent.Input0_ProcessInput(Input0Buffer Buffer) at ScriptComponent_0f80d51f9c68465b85a87853e8d3d5a0.UserComponent.ProcessInput(Int32 InputID, PipelineBuffer Buffer) at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.ProcessInput(Int32 inputID, PipelineBuffer buffer)

Any help would be greatly appreciated!

Thanks, Dean

Comments

Hello, please use Sftp object's LogWriter property to create a communication log (as described at http://www.rebex.net/kb/logging.aspx) and mail it to us for analysis. We might spot something there. In this case, it might be useful to send logs from several different failures to make it possible to tell whether there is really anything in common. Thanks!

Answer 1

I was actually able to get a way to get it to work. What I did was write the data to a stringbuilder object, convert that to bytes, then write the bytes to a memorystream. This was then used with the PutFile method. I believe it could have been due to a timeout issue on the SFTP server or something like that why the other process didn't work.

Thanks.

Comments

Thanks for letting us know! We are glad you got it working.

We would still love to know why exactly it was failing with GetUploadStream, but that proved extremely hard to analyze without being able to reproduce and observer the issue ourselves.

Answer 2

I never heard back after I sent the last round of log files. Any updates?

Thanks

Comments

Sorry - after I received no reply to my last e-mail regarding issues with upgrading to the latest release (2012 R1), I forgot to get back to you and ask whether you were able to successfully upgrade or not.

Were you able to upgrade to the latest version?

---

I was able to upgrade to the latest version, but it still fails at roughly the same place. After multiple tests, it seems to fail between rows 17,000 and 22,000. Could this be a memory issue?

I can send the new error log, but it has confidential information in it that will need to be scrubbed first, and I don't see too much from it that's different than the last log file I sent.

Thanks

---

The last log did not contain information about the actual SSH packets being received and sent (only SFTP packets were logged) due to a bug in the previous release of Rebex SFTP. That's why we asked for a new one. I don't think this is a memory issue - Rebex SFTP doesn't actually use much.

Answer 3

The thing I noticed is that it fails always on the last line of the packet from the errorlog (0420):

03F0 |30-33-31-35-37-35-7C-56 49-43-54-4F-52-7C-41-47| 
 0400 |55-53-54-49-4E-2F-47-4F 4E-5A-41-4C-45-5A-7C-7C| 
 0410 |7C-39-38-31-39-7C-7C-7C 55-50-53-20-4D-47-4D-54|
 0420 |20-55-53                                       |
2012-04-30 12:59:38.394 VERBOSE SshSession(6)[5] SSH: Sending packet SSH_MSG_DISCONNECT (27 bytes).
 0000 |01-00-00-00-02-00-00-00 0E-49-6E-74-65-72-6E-61| .........Interna
 0010 |6C-20-65-72-72-6F-72-00 00-00-00               | l error....
2012-04-30 12:59:38.425 ERROR SshSession(6)[5] SSH: System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at Rebex.Net.ProxySocket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at Rebex.Net.SshSession.1oSQZR(Int32 )
   at Rebex.Net.SshSession.JZ4YLZ(Byte[]& )
   at Rebex.Net.SshSession.1rsOcLZ(1VEj6UZ , Object[] )
   at Rebex.Net.SshChannel.1rsOcLZ(Boolean , 1VEj6UZ , Object[] )

Comments

Would it be possible to include a little bit more of the log? The error might have occurred while processing several previous packets as well, so it would be very useful to be able to see those as well. Including the send/received packets preceding the SSH_MSG_DISCONNECT packet would be useful. Thanks! (Mail the log to support@rebex.net if you don't wish to post here, and feel free to remove the actual payload of data packets.)

---

I sent the log file with I believe 4 sets of packets. I look forward to seeing the analysis. Thanks!

---

Thanks!

I have to admit that we still don’t know what is going on. According to the log, it looks like everything is fine until the connection is aborted, and the reason for this is unknown.

At this point, I would suggest experimenting a bit:

1. Instead of sw = New StreamWriter(client.GetUploadStream("test1.txt")), try using BufferedStream: sw = New StreamWriter(new BufferedStream(client.GetUploadStream("test1.txt"), 16384)) and try different buffer sizes.
2. Try using a different SFTP server to determine whether the server plays any role in this (we can set up one if needed).
3. Try uploading the same file using an equivalent code from a simple console application. Does that work, or does it cause the same error?

Hopefully, this would reveal some information which might make it possible for us to reproduce the issue.