FileServer command receive delay of 3 minutes

0 votes
asked Nov 27 by ioan.crisan (420 points)

I am trying to establish a duplex communication protocol between a FileServer and a ssh client. The client opens two Ssh sessions (in and out) to the FileServer using the NetworkSession for the incoming and outgoing messages. The communication protocol is a custom one (text based).

Everything works fine for a while. Suddenly, when the client sends a command through the out session (Scripting.SendCommand), this command reaches the FileServer (at least the ShellCommand event is fired) after 3 minutes, which, given that both processes are on the same machine, is way too much.

One more thing to note: each session, in and out, has a keep alive timer which is triggered each minute (basically it calls Scripting.KeepAlive()).

What could be the cause for such a behavior? Is is possible that SendCommand and KeepAlive interfere in a bad manner? What can I do (logging?) to further trace the issue?

Thanks,
Ioan

1 Answer

0 votes
answered Nov 28 by Lukas Pokorny (104,230 points)

Can you provide a sample app that reproduces the issue?

KeepAlive should not interfere with SendCommand, but the 3-minute delay clearly indicates that something is wrong somewhere.

commented Nov 28 by ioan.crisan (420 points)
I'm afraid there are too many components involved in this scenario to be able to provide something small and valuable. I'll do my best, but I'm quite skeptical. On the other hand, is there a way to enable some sort of tracing to narrow down the problem?
commented Nov 29 by Lukas Pokorny (104,230 points)
Both server and client support logging via LogWriter property - see https://www.rebex.net/kb/logging/ for details - and the logs might be helpful in narrowing down the problem. Feel free to post them here or mail them to support@rebex.net for analysis!
commented 3 days ago by ioan.crisan (420 points)
I managed to produce a log that brings some insights. Can I attach somehow a file to this issue/comment?
commented 3 days ago by ioan.crisan (420 points)
One more thing to note: I removed the manual KeepAlive calls and I let the FileServer manage this (fileServer.Settings.KeepAlivePeriod = 30).
commented 2 days ago by ioan.crisan (420 points)
Here is what I am trying to achieve with such a design:
* Each endpoint is able to send requests and (optionally) receive responses; the pairing is the responsibility of each endpoint.

* On the client side:
  * the input Ssh session reads the lines and builds the commands together. Commands are separated by a special line and are processed individually. Responses to the commands are sent through the output Ssh session when they are available.
  * the output Ssh session sends commands without any reading, as the output of the commands are not relevant (and, anyway, nothing comes back).

* On the server side:
  * the input ServerSession awaits for commands and sends them to processing. No response is sent.
  * the output ServerSession, which is initiated by the client using a special command, sends commands with a special terminator line (as "responses" to the special command), so that the client is able to aggregate them from the received lines.

Now here is what I could tell from the logs:
* At some point, the FileServer determines a timeout after 3 minutes (180 secs) and closes the channel. It seems to me that it waits for some input that does not come and I am not sure what ca I do to influence it.
commented 2 days ago by Lukas Pokorny (104,230 points)
Unfortunately, this Q&A forum doesn't support attachments - please mail the log to support@rebex.net instead.
commented 1 day ago by Lukas Pokorny (104,230 points)
Thanks for the log!

There is a built-in 3-minutes timeout that triggers this error if it was not possible to send data to the client for 3 minutes. The reason for not being able to send data is that the SSH channel’s receive buffer has became full, which in turn indicates that the client stopped adjusting receive buffer size, and that suggests that the client stopped reading the server response. Or, simply said, the server has been trying to send data, but the client was not receiving, so eventually the client-side receive buffer became full, making it impossible for the server to send more data.

Based on the description of what you are doing, it looks like you need to call Scripting.Process() in addition to Scripting.KeepAlive() to make sure that data sent by the server is received and does not remain in the receive buffer until it eventually runs out of space.
(Or use Scripting.ReadUntil/ReadLine/ReadChar to read server output for the command started by your SendCommand call.)
commented 6 hours ago by ioan.crisan (420 points)
Thanks for the answer!

I tried to use the Scripting.ReadUntil() but I have a problem with it. It actually starts reading from the second line sent by the server, ignoring the first line. I see the content of the first line in ReceivedData, but the first Read will skip it. What is the best approach?
1. Include the ReceivedData before reading the lines.
2. Send from the server a dummy line to be ignored.
...