Hello,
to make sure that the TCP port 5877 is opened, I created a very simple client and server that only uses .NET TcpClient and no Rebex components. At first, please try to test whether you are able to establish the connection from the client to your server with these two minimalistic programs and let me know how it went.
I believe you will have to setup the Windows Firewal correctly at the server machine. At least enable the port for the application to be able to accept incoming connections at your chosen port.
Here is a link to the simple TCP client and server.
Please give it a try and let me know how it went. A good way to start is trying to connect to the server locally (client app being installed on the server itself.) if it succeeds, try to move the client to another computer on your LAN, eventually on bigger networks.