How to execute an external process using Rebex SSH Server?

Question

I'm looking to execute git or mercurial using the rebex FileServer. Any pointers. I have done this before using another library (which has problems, hence looking at alternatives), and it seems Rebex doesn't support exec in it's shell?

I see I can hook the ShellCommand event, but how do I interact with the process I spawn. For example, when running a mercurial command over ssh it does an "exec hg serve -R c:\repo --stdio" and then interacts with the instance running on the server to run the actual command (clone etc).

Looking for pointers.

Answer 1

Update: The new API required to make this work (SshConsole.GetInputStream(), SshConsole.GetOutputStream(), SshConsole.GetErrorStream() and FileServer.Settings.SshParameters.CompressionLevel) has been added to Rebex File Server 2017 R4: https://www.rebex.net/file-server/history.aspx#2017R4

---

You are right, Rebex File Server doesn't support executing external commands. Its simplified virtual shell is mostly supposed to complement SFTP and SCP protocols and all the built-in commands only work with the virtual file system.

However, what you are trying to do is a very interesting scenario and it actually turned out that adding support for it is quite simple - we just had to enhance the internals a bit and make the inner streams accessible through SshConsole, which is passed with ShellCommand event arguments.

You can give this a try as well - just sent you a link to the current beta build.

We have not sufficiently tested this yet, but I was able to clone a 300 MB repository without any issues by running hg clone ssh://demo@localhost:8022/repository_name against this proof-of-concept server app:

using System;
using System.Diagnostics;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Rebex.Net;
using Rebex.Net.Servers;

namespace Rebex.Samples
{
    public class MercurialServer
    {
        // path to mercurial
        private const string MercurialPath = @"c:\Program Files\TortoiseHg\hg.exe";

        // path to a directory that contains the repositories
        private const string RepositoryRoot = @"c:\data\repository";

        // buffer size for data proxy
        private const int BufferSize = 32 * 1024;

        // log writer
        private static readonly ILogWriter LogWriter = new ConsoleLogWriter(LogLevel.Debug);

        // user name
        private const string UserName = "demo";

        // password
        private const string Password = "password";

        // server port
        private const int ServerPort = 8022;

        // server key
        private const string ServerKey = @"-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQDIkIOdSx47C4slUBVeEeVlCySg2IyT3laqBIr0uLlKIwwkI61F
Qen57kmgFG3Cs2zmbrnnRZYKR38EbrVyLH68wC9a1l6T7evUfnOD/O1fsPM1/XB6
m2shgUZsX6kXt1AUaPv2G4uH8J6Mk+RiJcrtBFslK0Lout5rLVm9Wn1d6QIDAQAB
AoGAYoBLC2S5l6EPOQeQPu+GHG5xEkfYHsUrBfwOLKtOYOE+lL8q2WFKYrOLWEHA
OEe7m55U0gckba74bDpdBZJhuT4MqaQ29PJn8fpi6RsFvvEqsgm6f9AWiScQ/n+h
/qSUraC5V8s03qF8XWuo5/87NNUunZDaA2UvUg8Urb0p87MCQQDjww8bwYVQS0lG
cpmjtNKqeLyfXtWPzo73/sCjIGsq/0Umdlufqlhezx3jvr+H/zaopp3A2dN6AKO8
nwFRrQrHAkEA4W48KB1R/Pjo5OefdiVAa6PFkmphFK0N7qjxg0WMCnxaqgbhwZFz
qnVq8RP2Al84Wn4fwpJwpKaqk82ZF2IhzwJBAMw1r+4q7OS5G9HWHnrxPZEq/7PE
y6ZMhVNFTmL0RiIfDlkV9cCKcwFOonX4KLI+2TsNaJPoufvBZw1PY1df1zECQCNt
CmEXcnn5t8e5KpMLeZswymye8RCpvWXDAOkrNb20Gx9bI4Ei1XV1LFAkXeWzhwyZ
g241SyRk2KuPhL5q+nsCQC8oiNdcTepTCGss+gNHlTdw2U5bhgUAZacCp5WgcEQJ
YNAx781J3Bwqh7s9vOCT0ubDxbi9+8uOui0ruL8AWA8=
-----END RSA PRIVATE KEY-----
";

        public static void Main()
        {
            var server = new FileServer();

            // enable logging
            server.LogWriter = LogWriter;

            // bind SSH shell subsystem to the specified port
            server.Bind(ServerPort, FileServerProtocol.Shell);

            // use the embedded private key for the sake of simplicity
            server.Keys.Add(new SshPrivateKey(new MemoryStream(Encoding.ASCII.GetBytes(ServerKey)), "password"));

            // only one user is supported for now
            server.Users.Add(UserName, Password);

            // custom command handler
            server.ShellCommand += (sender, e) =>
            {
                switch (e.Command)
                {
                    case "hg":
                        // intercept "hg" command
                        e.Action = RunMercurial;
                        break;
                    default:
                        // reject all other commands (optional)
                        e.WriteLine("Command not supported.");
                        e.ExitCode = 127;
                        break;
                }
            };

            server.Start();

            Console.WriteLine("Server is running, press any key to stop it.");
            Console.ReadKey(true);
        }

        public static int RunMercurial(string[] args, SshConsole console)
        {
            // start Mercurial

            var info = new ProcessStartInfo(MercurialPath, PrepareArguments(args));
            info.WorkingDirectory = RepositoryRoot;
            info.CreateNoWindow = true;
            info.UseShellExecute = false;
            info.RedirectStandardInput = true;
            info.RedirectStandardOutput = true;
            info.RedirectStandardError = true;

            var process = Process.Start(info);

            // proxy stdin/stdout/stderr

            Stream consoleInput = console.GetInputStream();
            Stream consoleOutput = console.GetOutputStream();
            Stream consoleError = console.GetErrorStream() ?? Stream.Null; // error stream is only available in terminal-less mode

            Stream processInput = process.StandardInput.BaseStream;
            Stream processOutput = process.StandardOutput.BaseStream;
            Stream processError = process.StandardError.BaseStream;

            CopyAllAsync("stdin", consoleInput, processInput);
            CopyAllAsync("stderr", processError, consoleError);
            CopyAllAsync("stdout", processOutput, consoleOutput);

            // wait for the process to end
            process.WaitForExit();

            // return the exit code
            return process.ExitCode;
        }

        private static string PrepareArguments(string[] args)
        {
            // TODO: This method is supposed to convert the argument array into a string,
            // but it is not yet implemented properly - if any of the arguments contains whitespaces or quotes,
            // this produces wrong results.

            // TODO: But if the users are only supposed to have SSH access to Mercurial repositories,
            // we should only allow "-R REPO_NAME serve --stdio" or its variants anyway.

            return string.Join(" ", args);
        }

        private static Task CopyAllAsync(string name, Stream input, Stream output)
        {
            // TODO: Error handling is missing.

            return Task.Run(() =>
            {
                byte[] buffer = new byte[BufferSize];
                while (true)
                {
                    int n = input.Read(buffer, 0, buffer.Length);
                    //Console.WriteLine("{0}: {1} bytes", name, n);
                    if (n == 0)
                    {
                        output.Close();
                        break;
                    }
                    output.Write(buffer, 0, n);
                    output.Flush();
                }
            });
        }

    }
}

Comments

Wow! That works great, and in my tests with a 500MB repo, cloning took 133s vs 234s with the library we currently using (and that is native code).

Interestingly, changing the buffer size didn't make much difference to performance.

I guess my only question on this now is what do you suggest for error handling. If I abort the clone from the client side then I see this in the log :

2017-03-08 14:38:31.646 ERROR FileServer(1)[18] SSH: Session 1: Error while sending packet data: System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at Rebex.Net.UNZ.D(Byte[] V, Int32 T, Int32 Q)
   at Rebex.Net.Protocols.Ssh.OGZ.JG(UTZ V)
2017-03-08 14:38:31.653 ERROR ShellModule(1)[18] Server: Session 1: Error: Rebex.Net.Protocols.Ssh.QZZ: Error while sending packet data.
   at Rebex.Net.Protocols.Ssh.OGZ.JG(UTZ V)
   at Rebex.Net.Protocols.Ssh.STZ.KT(Int32 V, Byte[] T, Int32 Q, Int32 Z)
   at Rebex.Net.Protocols.Ssh.STZ.KT(Byte[] V, Int32 T, Int32 Q)
   at Rebex.Net.Protocols.AZZ.YI(Nullable`1 V, Stream T, Byte[] Q)
Error: Rebex.Net.Protocols.Ssh.QZZ: Error while sending packet data.
   at Rebex.Net.Protocols.Ssh.OGZ.JG(UTZ V)
   at Rebex.Net.Protocols.Ssh.STZ.KT(Int32 V, Byte[] T, Int32 Q, Int32 Z)
   at Rebex.Net.Protocols.Ssh.STZ.KT(Byte[] V, Int32 T, Int32 Q)
   at Rebex.Net.Protocols.AZZ.YI(Nullable`1 V, Stream T, Byte[] Q)

The server however does stay up and continue to function normally.

---

The buffer in CopyAllAsync is only used for passing data between the 'hg' process and File Server's internal buffers, so this is not the bottleneck - in my tests, I noticed the largest block copied was 8 KB (or 16 KB, I'm not sure any more) even when the buffer was larger.

The error indicates that the session failed and although it's ugly, it doesn't affect the rest of the server. We plan to address this in one of the next releases and log something less alarming at the "INFO" level instead.

---

I've been putting this solution to some real world testing, and it's working really well. My only concern is the cpu usage when ssh compression is used, on my machine (recent i7, m2 ssd's etc) each client uses around 7% cpu when compression is enabled, but only around 1% with compression turned off (using plink). Is this expected? I realise compression is cpu intensive however it does seem a bit excessive.

---

I am little bit confused. The CPU usage is increased on client side or server side? Please note, that we cannot utilize the client side (plink app).
If you run both server and client on the same machine, the increased 7% CPU usage can be caused by the server. Can you please prove or disprove this by running server on different machine than client?
Also please note, that compression is more CPU demanding than decompression. So, when uploading data, the more demands are put on client, for downloading more demands are put on server.
In case of Rebex components, the compression (and decompression) is 100% managed code. It will be always less efficient than native code.
However, the compression is performed with default compression level (6 of 9). I suppose that decreasing the level should decrease the CPU demand. If you want to play with this, we can expose a property to set the compression level explicitly. This will affect only compression part (in case of server = downloading process). Please let me know, whether you would like to try to set compression level explicitly.

---

The cpu usage is for the ssh server process. The ssh server process when running 4 concurrent mercurial clones with compression enabled was around 40-50% (that's not counting the cpu usage of the hg processes spawned). I realise that managed code might not be as efficient as native, but perhaps there are opportunities for optimisation. I will do more testing on other machines tomorrow to confirm if what I am seeing is realistic or a bad test.

Yes, I'd like to try setting the compression level if possible.

---

I have added the server.Settings.SshParameters.CompressionLevel, so you can utilize the compression process. However, from my measures it doesn't have big impact to CPU usage. The compression process is naturally demanding. (I have tested on random data, maybe in real scenario it would have better results).

We have no plans to optimize this. We have similar results as other managed implementations (we compared speed, but CPU usage probably corresponds to speed), so I think we reach the limit - other improvements would cost much more than possible speed gain.

I have sent you a link to current beta version to your email a minute ago.

---

SshConsole.GetInputStream(), SshConsole.GetOutputStream() and SshConsole.GetErrorStream() methods and methodsFileServer.Settings.SshParameters.CompressionLevel property have been added to Rebex File Server 2017 R4: https://www.rebex.net/file-server/history.aspx#2017R4