Occasional "Unable to process command" SSH error

0 votes
asked Aug 28, 2013 by Eugene V_ (160 points)
edited Dec 9, 2013

Dear Rebex team,

We are currently using your SSH communication classes in our projects and they are really great! Unfortunately there is one thing that makes us sad - the occasional "Unable to process command" throwing on random commands sent via SSH in different times.

Our application executes complicated workflows 7/24 and extensively uses SSH communication. The said error may appear 1 or more times per week and there is no specific case for it, i.e it fails on different commands with different number of threads (even with 1 thread), etc.

Here's some technical details:

  • Version: 1.0.4546.0
  • .NET Framework: 4.0
  • Class(es): Ssh, Shell via Ssh.CreateShell();
  • Mode: Prompt;
  • Threading: Multiple (each thread creates it's own instances of Ssh and Shell classes);

I also have verbose log files, if you need them.

I hope for your help, Thank you!

commented Aug 28, 2013 by Lukas Matyska (40,360 points)
edited Aug 28, 2013

"Unable to process command" error is raised when the server replies unexpectedly to the command, so we are not able to determine where the output of the command begins. Verbose log will be very helpful in this case. Can you please send it to support@rebex.net for investigation?

commented Aug 29, 2013 by Eugene V_ (160 points)
edited Aug 29, 2013

Thank you, Lucas! Requested files were sent to support@rebex.net

BTW, we are considering buying a newest version of Rebex Total Pack.

commented Aug 30, 2013 by Eugene V_ (160 points)
edited Aug 30, 2013

Thanks for detailed explanation!

Everything is clear now, although I need to struggle more with my prompt pattern to make it as narrow as popossible.

As for initial "set prompt" command... you are right, I think we can remove it!

Lucas, if you have a patch, could you please send it to me for testing, please?

2 Answers

0 votes
answered Aug 30, 2013 by Lukas Matyska (40,360 points)
edited Aug 30, 2013

I summarize my answer from the mail for other visitors:

It was caused by coincidence of the received data and selected prompt:

prompt = regex:(.@.:s+)
command sent (due to server echo, it is also received data): set prompt="whoami@hostname: "

The server can split output data into blocks of any length. In the case of "Unable to process command" error, it was unfortunately split that way, the block ended with "`whoami`@`hostname`: " which was matched as prompt.

This can be treated as imperfection of our prompt detector. If we have detected prompt at the end of one block, we can wait let say 50ms for another block to be sure this is really the last block of output and detected prompt is the real prompt, not only part of output data. This will improve prompt detection, because if this situation arises, it is very probably, that the next block is ready to read (actually it was originally one bigger response split into two blocks in unlucky way). However, it cannot be never 100% reliable, because what if the network got stuck suddenly, and the next block doesn’t come within 50ms interval. It is all because Shell session is not suitable/designed for automatic program processing. This could be easily resolved e.g. if the server is waiting for a user input, it would send a special signal – unfortunately it wouldn’t.

Conclusion: The best practice of programming Shell processor is to use such prompt, which will never occur in the output data.

@Eugene: In your case, you are currently changing the server's prompt, so I suggest you to change it to "-BEGIN-`whoami`@`hostname`-END-: " and use prompt = regex:^(-BEGIN-[^`]*@[^`]*-END-: )$

@Eugene: I have sent the patch with 50ms delay to your email recently.

0 votes
answered Sep 3, 2013 by Eugene V_ (160 points)
edited Dec 9, 2013

We are currently testing your suggestions, Lukas. We've removed all occurrences of commands conflicting with our prompt pattern and added additional check to avoid further conflicts. This should be enough for us.

As for DLL fix... We need some time to make sure the modified DLLs are working fine before we can expose them to our clients. Our testing team will engage this.

I also sent a request to our management to prolong the support agreement with Rebex.

commented Sep 3, 2013 by Lukas Matyska (40,360 points)
edited Sep 3, 2013

Yes, removing commands conflicting with your prompt pattern will definitely fix this issue.

Of course, we have automated tests for all components, but feedback from live testers is more valuable for us. If you have any problems, don't hesitate to let us know.

Thank you for prolonging the support agreement. If you want source code of the patched version, please let me know.

commented Sep 3, 2013 by Lukas Matyska (40,360 points)
edited Sep 3, 2013

Yes, removing commands conflicting with your prompt pattern will definitely fix this issue.

Of course, we have automated tests for all components, but feedback from live testers is more valuable for us. If you have any problems, don't hesitate to let us know.

Thank you for prolonging the support agreement. If you want source code of the patched version, please let me know.

commented Sep 10, 2013 by Eugene V_ (160 points)
edited Sep 10, 2013

We tested your host fix on one of our clients and it worked brilliant!

Thank you!

commented Dec 9, 2013 by Lukas Matyska (40,360 points)
edited Dec 9, 2013

We have recently released new public version of Rebex components including changes made for you (Version 2013 R3).

You can download current version from http://www.rebex.net/protected

...