Ask a Question

Is using REBEX to write an SFTP Server a really stupid idea?

+1 vote
by (130 points)
edited by

We are looking at writing client and server pieces - for gathering files at client site, shipping them to us via SFTP and processing those files in our applications. Is writing the server piece using REBEX a really bad approach? How many different SFTP sessions can you have running simultaneously on a beefy server? (I've read your answers about separate threads for separate sessions). How much is too much?

We are also looking at speeding up transfer by creating multiple FTP sessions on the client end. If so - how many threads/sessions can we safely have running on the "client end"?

Thanks for your words of wisdom.

Applies to: Rebex SFTP

1 Answer

0 votes
by (144k points)
edited by

UPDATE:

Check out our new Rebex File Server component. It is specifically designed for writing your own SFTP / SSH / SCP server. It is a perfect counter part for our client side Rebex SFTP client library.

It's not a stupid idea at all - you just have to find out the optimal number of SFTP sessions.

One of our clients had 6000 active SSH sessions at the same time on a beefy server and although this used a lot of memory (1GB if I remember correctly), it was usable. However, the SSH sessions were not heavily utilized - they were idle most of the time.

But your situation is different. With SFTP, you would be transferring files over encrypted communication channels. In that case, the first bottleneck you would hit is either the maximum throughput of your network connection or the CPU speed (SSH encryption is a CPU-intensive process). The optimal number of sessions would almost certainly be much lower even on a beefy server.

I would suggest the following approach to find the right number of SFTP sessions for your application:

  1. Make the number of sessions configurable.
  2. Start with a single SFTP session and pay attention to CPU utilization.
  3. Keep adding more sessions until the CPU utilization gets too high. If you have multiple CPU cores, you would need to use several threads to utilize all of them.
  4. If the CPU utilization doesn't get higher even though you add more SFTP sessions, then you probably reached the network speed limit and adding more sessions won't result in higher overall speed.
...