FileServer command receive delay of 3 minutes

Question

I am trying to establish a duplex communication protocol between a FileServer and a ssh client. The client opens two Ssh sessions (in and out) to the FileServer using the NetworkSession for the incoming and outgoing messages. The communication protocol is a custom one (text based).

Everything works fine for a while. Suddenly, when the client sends a command through the out session (Scripting.SendCommand), this command reaches the FileServer (at least the ShellCommand event is fired) after 3 minutes, which, given that both processes are on the same machine, is way too much.

One more thing to note: each session, in and out, has a keep alive timer which is triggered each minute (basically it calls Scripting.KeepAlive()).

What could be the cause for such a behavior? Is is possible that SendCommand and KeepAlive interfere in a bad manner? What can I do (logging?) to further trace the issue?

Thanks,
Ioan

Answer 1

Can you provide a sample app that reproduces the issue?

KeepAlive should not interfere with SendCommand, but the 3-minute delay clearly indicates that something is wrong somewhere.

Comments

I'm afraid there are too many components involved in this scenario to be able to provide something small and valuable. I'll do my best, but I'm quite skeptical. On the other hand, is there a way to enable some sort of tracing to narrow down the problem?

---

Both server and client support logging via LogWriter property - see https://www.rebex.net/kb/logging/ for details - and the logs might be helpful in narrowing down the problem. Feel free to post them here or mail them to support@rebex.net for analysis!

---

I managed to produce a log that brings some insights. Can I attach somehow a file to this issue/comment?

---

One more thing to note: I removed the manual KeepAlive calls and I let the FileServer manage this (fileServer.Settings.KeepAlivePeriod = 30).

---

Here is what I am trying to achieve with such a design:
* Each endpoint is able to send requests and (optionally) receive responses; the pairing is the responsibility of each endpoint.

* On the client side:
  * the input Ssh session reads the lines and builds the commands together. Commands are separated by a special line and are processed individually. Responses to the commands are sent through the output Ssh session when they are available.
  * the output Ssh session sends commands without any reading, as the output of the commands are not relevant (and, anyway, nothing comes back).

* On the server side:
  * the input ServerSession awaits for commands and sends them to processing. No response is sent.
  * the output ServerSession, which is initiated by the client using a special command, sends commands with a special terminator line (as "responses" to the special command), so that the client is able to aggregate them from the received lines.

Now here is what I could tell from the logs:
* At some point, the FileServer determines a timeout after 3 minutes (180 secs) and closes the channel. It seems to me that it waits for some input that does not come and I am not sure what ca I do to influence it.

---

Unfortunately, this Q&A forum doesn't support attachments - please mail the log to support@rebex.net instead.

---

Thanks for the log!

There is a built-in 3-minutes timeout that triggers this error if it was not possible to send data to the client for 3 minutes. The reason for not being able to send data is that the SSH channel’s receive buffer has became full, which in turn indicates that the client stopped adjusting receive buffer size, and that suggests that the client stopped reading the server response. Or, simply said, the server has been trying to send data, but the client was not receiving, so eventually the client-side receive buffer became full, making it impossible for the server to send more data.

Based on the description of what you are doing, it looks like you need to call Scripting.Process() in addition to Scripting.KeepAlive() to make sure that data sent by the server is received and does not remain in the receive buffer until it eventually runs out of space.
(Or use Scripting.ReadUntil/ReadLine/ReadChar to read server output for the command started by your SendCommand call.)

---

Thanks for the answer!

I tried to use the Scripting.ReadUntil() but I have a problem with it. It actually starts reading from the second line sent by the server, ignoring the first line. I see the content of the first line in ReceivedData, but the first Read will skip it. What is the best approach?
1. Include the ReceivedData before reading the lines.
2. Send from the server a dummy line to be ignored.

---

Could you please provide some sample code that shows what you are actually doing at the client?

---

I'll send you per e-mail the core implementation, even though it is not runnable without the whole context.