Implementing a proxy server behind a firewall

Question

I have the following requirements:

• A proxy server behind a firewall. Only outgoing connections are possible from this server outside the firewall.
• The proxy server has permissions to connect to any machine behind the firewall, using any protocol (ssh/telnet).
• An application outside the firewall.
• The application needs to open ssh/telnet connections to machines behind the firewall with the help of the proxy server.
• Note: the communication between the application and proxy needs to be two-way.

Could you help me figure out how can I use Rebex to achieve such a behavior? Thank you in advance.

Comments

Before we propose a solution, I need to make sure that our assumptions are correct:
- I assume that any solutions that require any configuration changes at the 'firewall' are not acceptable. Is that correct?
- On the other hand, I assume it is possible to install and run a custom .NET application (possibly based on Rebex libraries) on the 'proxy server'. Please confirm whether this is the case.
- If both of the assumptions are correct, any solution would involve setting up a different kind of proxy server (possibly an SSH server) outside the 'firewall' to facilitate the communication. Is that acceptable as well?

---

1. No changes to the firewall.
2. Both the proxy and the application will include the Rebex components.
3. Outside of the firewall we do not have any constraints, meaning that we are in control.

Answer 1

Consider the following approach:

1. Set up an SSH server outside the firewall that is accessible both by the proxy server and by the application.
2. Configure that external SSH server to allow incoming tunnels (reverse port forwarding) from non-local endpoints.
3. At the proxy server, run an application that uses Rebex Terminal Emulation's port forwarding API to establish an incoming tunnel (or a set of tunnels) from an address/port at the external SSH server to a machine behind the firewall.

Details:

(1) Any SSH server that supports reverse port forwarding is suitable. An OpenSSH server on a Linux VPS would be fine. (Our Buru SFTP Server is intended for file transfer and doesn't support reverse port forwarding yet.)

(2) In OpenSSH, this can be enabled by setting the GatewayPorts configuration option to yes.

(3) At the proxy server, an application based on Rebex.SshShell assembly would establish tunnels from the SSH server to local machines:

// connect and log in to an SSH server
var ssh = new Rebex.Net.Ssh();
ssh.Connect(hostname);
ssh.Login(username, password);

// create port forwarding rules
var tunnel1 = ssh.StartIncomingTunnel(
    "sshserver.example.org", 10022,  // server-side source address/port
    "192.168.1.1", 22); // client-side target address/port

// create port forwarding rules
var tunnel2 = ssh.StartIncomingTunnel(
    "sshserver.example.org", 11022,  // server-side source address/port
    "192.168.1.2", 22); // client-side target address/port

...

With this setup, an application outside the firewall would be able to connect to (for example) sshserver.example.org:11022 and the connection would get tunneled to 192.168.1.2:22.

With this approach, all of the possible tunnels would have to be pre-configured. Or, in other words, the application outside the firewall could not connect to any IP/port behind the firewall. This could be both a drawback or a benefit, depending on your scenario. But if you need the ability for the application to connect to any IP/port behind the firewall, this could be achieved by first connecting to an SSH server behind the firewall and using forward port forwarding to reach the other machines.

Comments

Thank you for your answer, now it's time to try your suggested solution :).

---

Maybe a clarifying question: in your example, it means that both 10022 and 11022 ports must be exposed by the sshserver.example.org (or a valid IP address) to be able to create the tunnels, am I right? If so, if I need access to 1000 machines, then I need 1000 ports to be open, which I don't think it's an option.

Would it then be possible to implement some kind of "splitter", like web servers do, which, based on some kind of discriminator, redirect the calls to a target machine or another?

---

You are right, and I briefly addressed this in the last paragraph of reply. One way to resolve this is to set up another SSH server behind the proxy, and use that as a kind of "splitter". This way, you would only have a single incoming tunnel from the external SSH server to the internal SSH server (let's say sshserver.example.org:10022 to 192.168.1.2:22). Now, the application outside the firewall would be able to connect to other machines as well: First, the application would connect to sshserver.example.org:10022, and that connection would get tunneled to the SSH server at 192.168.1.2:22. Next, the application would use this SSH session to establish an "outgoing tunnel" (sorry if this is getting a bit confusing, but it makes sense from the application's point-of-view) to any other server and port in the internal SSH server's network. To facilitate this, the application could use Rebex Terminal Emulations outgoing tunnel API (https://www.rebex.net/terminal-emulation.net/features/ssh-tunneling.aspx#outgoing-tunnels). There is one drawback - the tunnel would involve a tunnel running over an SSH session that runs over a tunnel within another SSH session, which means data would be encrypted/decrypted twice, which might negatively affect performance and throughput. The benefit of this solution is that all the parts would be based on open standards with no vendor lock-in.

---

Thanks again, I think this is what I need. So I need to get to work now and try this.

---

After trying to understand how can I use Rebex to model such a scenario, I think I found a logical issue. Let's name the external SSH server: E, the internal SSH server/behind the firewall: I, and other systems behind the firewall S1, S2, so on.

1. The firewall allows *outgoing* connections on port 22, meaning that I can connect/tunnel to E using port 22 (I -> E).
2. I can create *outgoing* tunnels to S1, ..., Sn (I -> S1, ...Sn), meaning that for each such tunnel a new port will be made available on I (which is fine).
3. E needs *outgoing* tunnels to S1, ..., Sn through I.
4. On E, I need a connection object to send commands/bind a terminal.

The problems here are related to the firewall between I and E.
P1: I need an *outgoing* tunnel from E to I, provided I can have only an *outgoing* one from I to E.
P2: Even if I can have multiple ports open on I and E, however the firewall allows only one (outgoing 22).

Maybe I missed something, but to me it seems that this cannot work.

---

The approach you describe doesn't seem to correspond to what I trying to describe in my comment. I'll try again.

The first part of the second solution (the one with "splitter") is to make it possible for applications from the external network to connect to the SSH server at *I* through the SSH server at *E*. The following approach can be used to achieve this:

A1. A .NET application (using Rebex SSH) running on the internal network (possibly on *I* itself) would establish an SSH session to *E*.
A2. Once it connected and authenticated, the application would then start an *incoming* tunnel from an external port (let's say 10022) on *E* to the SSH server at *I*.

Now, as long as the .NET application is running and its single SSH session is active, applications on the external network would be able to connect to "E:10022". That connection is going to get transparently tunneled through the SSH server at *E* and through the .NET application (at *I* or possibly on another machine in the internal network) to "I:22".

Once this is ready, the rest of the process would look like this:

B1. An application running on external network would use Rebex SSH to connect to the SSH server at *I* through the previously-established tunnel. Simply said, it would just connect to *E:10022* . That connection would get transparently tunneled to *I:22*, the internal SSH server.
B2. The application would authenticate to the internal SSH server and establish a tunnel to whatever internal machine it needs (S1,...,Sn). (Technically, this would be called an *outgoing* tunnel because the client is going to connect through the SSH server, not accept connections through it.)
B3. The application would connect to S1...Sn through that tunnel.

---

A2: Sorry for my lack of understanding, but I don't really get how *E* could connect to *I* as long as the firewall between *E* and *I*, protecting *I*, allows only outgoing connections on port 22. In my understanding it is only *I* which can initiate connections, not *E* - maybe there is a way for *E*, once *I* opened a session/tunnel, to "misuse" that session, making it incoming for *I*.

P.S. Is there a way in this forum to attach some picture to depict this situation?

---

To be more clear: the firewall opens only port 22, all the other are not available.

---

There is just a single TCP connection between *I* and *E* the whole time - the SSH connection initialized by *I* connecting to *E:22* (which is allowed by the firewall).

But once an SSH session has been negotiated and authenticated on that TCP connection, the SSH client (possibly on *I*) can use SSH protocol's TCP port forwarding feature to establish a tunnel from *E:10022* to *I:22*, through the SSH session. There is no actual TCP connection established by E connecting to I, but an equivalent of this is achieved by the means of the SSH protocol whose sessions can consist of multiple independent channels established for various purposes. This is not a "misuse" of the SSH session, it's a standard (and widely used) feature of the protocol - see https://tools.ietf.org/html/rfc4254#section-7 for details.

---

The forum supports images, but only in questions and answers (not in comments).
But they have some useful pictures here: https://unix.stackexchange.com/questions/46235/how-does-reverse-ssh-tunneling-work

---

(Coming back from Easter) OK, now really Rebex specific:
1. *I* initiates a SSH tunnel. The SSH FileServer on *E* receives a TunnelRequested event, user/password is checked, everything is fine, connection established.
2. Using this tunnel, now *E* needs a terminal, what typically I did using an Ssh class. Now, how do I come from the TunnelRequested event to my Ssh instance?

Note: it does not have to be FileServer or Ssh classes, but this is what I used up to now.

Thank you for your patience!

---

1. This would be correct if Rebex File Server already supported incoming SSH tunneling. It doesn't yet (it's mostly intended for file transfers), so you would have to use a different SSH server (such as OpenSSH) on *E*.

So let's assume that the tunneling has just been started from *E*:10022 to *I*:22.

Now, it is going to be possible for applications running on *E* (or possibly elsewhere on the Internet - depending on the configuration of the tunneling on *E*) to connect to the tunnel's endpoint (localhost:10022 on *E*, or even *E*:10022 from elsewhere). Once the connection to that endpoint is established, the SSH server on *E* will essentially establish a tunnel through the SSH session (established by the SSH client on *I*) and through the SSH client on *I* to the tunnel's target, which could be any machine in the *I*'s internal network (such as *I*:22).

So you then end up with two SSH sessions: One between an SSH client at *I* and an SSH server at *E*; and the other one between an SSH client at *E* (or another machine on the Internet) and an SSH server at *I* (or another machine in the internal network).

---

So, I come up with some news. After a lot of prototyping, I managed to implement the desired behavior using the FileServer, but only to the point that I can establish a "duplex" communication channel between the SSH server and the Proxy server. How I did it?

1. The Proxy server initiates the connection by creating two SSH sessions (Ssh component): one for the input queue and the other one for the output queue.

2. The SSH Server receives the commands in the ShellCommand event from the Proxy. The session of the Proxy's input queue becomes the session of the SSH Server's output queue (ServerSession component) and the session of the Proxy's output queue becomes the session of the SSH Server's input queue (also ServerSession component).

Side note: at this point it would be interesting to see how can I use a sigle session for the duplex communication (currently, when I use a single session for the input and output queues, a Send would crash if a ReadLine is active).

The communication protocol works like this:
A. SSH Server sends a request.
  1. SSH Server out: ServerSession.SendMessage(string_encoded_message).
  2. Proxy in: Scripting.ReadLine until message is completely received.
  3. Proxy processes the message.
  4. Proxy out: Scripting.SendCommand(string_encoded_response).
  5. SSH Server in: ShellCommand handler receives response and matches the request.

B. Proxy sends a request.
  1. Proxy out: Scripting.SendCommand(string_encoded_message).
  2. SSH Server in: ShellCommand handler receives message.
  3. SSH Server processes the request.
  4. SSH Server out: ServerSession.SendMessage(string_encoded_response).
  5. Proxy in: Scripting.ReadLine until response is completely received and matches the request.

The communication works fine so far. The problem arises when I try to start a tunnel from the SSH Server to a target. So here is the flow, assuming that the duplex connection is up and running:

1. Identify a free port on the SSH Server.
2. Send to the proxy the free port number and the target host/port.
3. The proxy uses its out queue's SSH session to start an incoming tunnel from (127.0.0.1, free-port) to (target-host, target-port). BANG! This is the moment when the method crashes with the cryptical "The request has failed" exception.
   (from log: ID:3|area:SSH|Rebex.Net.SshException: The request has failed.
                at Rebex.Net.SshSession.ExecuteRequest(ConnectGlobalRequest request, GlobalRequestKind kind)
                at Rebex.Net.SshSession.StartTcpIpForward(String address, Int32 port))

This is the region which generates the crash:
            object[] result = WaitFor<object[], SshGlobalRequest>(ExecuteRequestFinishCheck, r);
            if (result == null)
                throw new SshException(SshExceptionStatus.OperationFailure, SshStrings.RequestFailed);

Trying to build a tunnel using putty indicates an error: "Remote port forwarding from 127.0.0.1:4800 refused." (a connection was previously established to the server, and the tunnel is created in this connection.)

Side note: there is no notification in both Ssh and ServerSession when the underlying connection is closed/dead (such as a Closed event). I implemented a timer that checks periodically (each minute) if the connection is alive.

Any additional help would be appreciated.

---

It looks like you choose to implement this in a different way using a lower level API, but at quick glance it looks like it should work as well. However, as I already mentioned on May 6th, Rebex File Server does not support incoming SSH tunneling yet, and that means it will reject all SshSession.StartTcpIpForward requests.

This is actually one of the features we would like to add soon, but we are currently busy with other features and this is not in our near-term plans yet. At this point, our recommendation would be to use a different SSH server, but that approach is not compatible with your solution.

However, the list of customers who would like this feature keeps growing, so we will consider reprioritizing this. What is your project's schedule? Would a beta with incoming SSH tunnel support be sufficient for a month or two?

---

Hi Lukas,

yes, thanks, it should suffice. When do you think it could be available?

---

Are there any updates on the incoming SSH tunneling?

---

We will make a decision on this next week. This is one of the most requested new features, so chances are we'll start implementing this when we are done with "2019 R4 Preview 1" release.