Rebex.Net 8.0 Performance Issue on Windows Server 2008 R2 SP1

Question

Hello,

We are using the Rebex.Net library on Windows Server 2008 R2 SP1 to support TLS 1.2/1.3 communication.

We did not experience any issues with Rebex.Net 7.0.8943, but after upgrading to version 8.0.9797, we started seeing significant performance problems when making requests in parallel.

A simplified example of our code is shown below:

Parallel.For(1, 10, i =>
{
    var creator = new HttpRequestCreator();

    creator.Settings.SslAcceptAllCertificates = true;
    creator.Settings.SslAllowedVersions = TlsVersion.TLS12;
    creator.Settings.SslAllowedSuites =
        TlsCipherSuite.ECDHE_RSA_WITH_AES_128_GCM_SHA256 |
        TlsCipherSuite.ECDHE_RSA_WITH_AES_256_GCM_SHA384;

    creator.Settings.UseLargeBuffers = true;
    creator.Proxy = null;
    creator.Register();

    HttpRequest request = creator.Create(url);

    request.Method = "POST";

    byte[] buffer = Encoding.UTF8.GetBytes(data);
    request.ContentLength = buffer.Length;

    using (Stream stream = request.GetRequestStream())
    {
        stream.Write(buffer, 0, buffer.Length);
    }

    var response = (HttpResponse)request.GetResponse();
});

With Rebex.Net 8.0.9797, both GetRequestStream() and GetResponse() take a very long time to complete.
In addition, although we are executing requests in parallel, it appears that only two requests are processed simultaneously, while the remaining requests wait in a queue.

We spent considerable time investigating the issue and tried various approaches, but were unable to resolve it.
As a test, we downgraded to Rebex.Net 7.0.9796, and the performance issue immediately disappeared.
Parallel requests worked as expected again.

Because the problem only occurs with the 8.x version, we suspect there may have been a change in the networking, connection management, TLS handling, or threading behavior between versions 7 and 8.

Are there any known changes in Rebex.Net 8 that could explain this behavior?

Also, is there anything we might be missing in our implementation or configuration?
Any guidance or recommendations would be greatly appreciated.

Thank you for your help.

Answer 1

Yes, the 8.x version introduced a new asynchronous HTTP core, which impacts your use case. Citing from the 8.0.9673 release notes:

Asynchronous HTTP client core
Rebex HTTP, Graph, EWS and WebSocket feature a new HTTP client core that has been upgraded to fully asynchronous mode, making it less thread-hungry and more scalable.

The synchronous methods now internally wait for completion of their async counterparts. That is fine in general, but each synchronous call now occupies one thread-pool thread while waiting, and needs another one to complete the underlying task. With Parallel.For() this means more threads are needed than before, so the pool is quickly filled with waiting threads and has none left to complete the tasks. The pool injects new threads slowly, so most requests end up waiting in a queue, which matches what you are observing. This is called thread pool starvation.

The suggested solution is to rewrite the code to use asynchronous calls (GetRequestStreamAsync()/GetResponseAsync()), which makes better use of threads and avoids blocking them.

On modern platforms use Parallel.ForAsync() or Parallel.ForEachAsync(), on older ones use Task.WhenAll(). It can look like this:

var tasks = Enumerable.Range(1, 10).Select(async i =>
{
    // ...
    var response = await request.GetResponseAsync();
    // ...
});

var parallelForTask = Task.WhenAll(tasks);

Note: Task.WhenAll() starts all operations at once. If you need to limit concurrency (such as due to server-side limits), you have to throttle it yourself, for example using a semaphore.

If you cannot change the code right now, you can make the pool create threads immediately:

// call once at application startup (process-wide setting)
ThreadPool.SetMinThreads(100, 100);

This has its limits:

• The value must exceed the peak number of concurrently blocked threads, which is hard to predict. If it is exceeded (more iterations, other load in the process), starvation returns.
• Each thread consumes memory (~1 MB of stack), so you are paying for threads that do nothing but wait.
• It is a process-wide setting that affects the whole application.
• It only hides the problem, and the code does not scale the way the async version does.