UPDATE:
Check out our new Rebex File Server component. It is specifically designed for writing your own SFTP / SSH / SCP server. It is a perfect counter part for our client side Rebex SFTP client library.
It's not a stupid idea at all - you just have to find out the optimal number of SFTP sessions.
One of our clients had 6000 active SSH sessions at the same time on a beefy server and although this used a lot of memory (1GB if I remember correctly), it was usable. However, the SSH sessions were not heavily utilized - they were idle most of the time.
But your situation is different. With SFTP, you would be transferring files over encrypted communication channels. In that case, the first bottleneck you would hit is either the maximum throughput of your network connection or the CPU speed (SSH encryption is a CPU-intensive process). The optimal number of sessions would almost certainly be much lower even on a beefy server.
I would suggest the following approach to find the right number of SFTP sessions for your application:
- Make the number of sessions configurable.
- Start with a single SFTP session and pay attention to CPU utilization.
- Keep adding more sessions until the CPU utilization gets too high. If you have multiple CPU cores, you would need to use several threads to utilize all of them.
- If the CPU utilization doesn't get higher even though you add more SFTP sessions, then you probably reached the network speed limit and adding more sessions won't result in higher overall speed.