Tls listener closes sometimes unexpectedly

Question

Hello, I'm facing some weird problem with tcp listener. Every now and then it stops listening for new connections, but ongoing ones are being processed normally. Having problems in reproducing it. ServerCommunication class is class is initiated as singleton service. Main server code snippet below. Any ideas?

   public ServerCommunication(PaymentServerConfiguration configuration,
                           ServiceConfig serviceConfig,
                           HandlerFactory handlerFactory,
                           ICertificateManager certificateManager)
{
    Rebex.Licensing.Keys.Add("");
    _configuration = configuration;
    this.serviceConfig = serviceConfig;
    _handlerFactory = handlerFactory;
    CertificateManager = certificateManager;
    Task.Run(StartServer);
}

private async Task StartServer()
{

    var certificate = CertificateManager.GetTlsCertificate();
    var listener = new Socket(SocketType.Stream, ProtocolType.Tcp);
    listener.Bind(_configuration.IpEndpoint);
    listener.Listen(524);

    while (true)
    {
        var plainSocket = await listener.AcceptAsync();
        var socket = new TlsServerSocket(plainSocket);
        socket.Parameters.Certificate = CertificateChain.BuildFrom(certificate);
        await socket.NegotiateAsync();
        Task.Run(() => HandleClientAsync(socket));
    }
}

private async Task HandleClientAsync(TlsServerSocket tcpClient)
{
    var socket = tcpClient.Socket;
    string? terminalId = null;
    LogConnection(socket);

    while (true)
    {
        try
        {
            terminalId ??= GetTerminalIdFromSocket(tcpClient);
            var client = TryGetClient(tcpClient, terminalId);

            if (!IsSocketAlive(socket))
            {
                DisposeConnection(terminalId);
                break;
            }


            var dataBytes = await PullData(tcpClient);

            if (dataBytes.Count() == 0)
            {
                _logger.Error($"Available bytes were >0, but we received no bytes.");
                continue;
            }

            await ProcessMessage(client, dataBytes, terminalId);
        }
        catch (Exception ex)
        {
            _logger.Error($"Received unexpected error: {ex.Message} \n{ex.StackTrace}");
        }
    }

    LogDisconect(socket);
    UnregisterExtender(terminalId);
}
public void UnregisterExtender(string terminalId)
{
    if (terminalId == null)
    {
        return;
    }

    if (!_connections.ContainsKey(terminalId))
    {
        return;
    }

    _connections.Remove(terminalId);
    _logger.Info($"Terminal '{terminalId}' connection has been dropped.");
}

Answer 1

This looks problematic:

while (true)
{
    var plainSocket = await listener.AcceptAsync();
    var socket = new TlsServerSocket(plainSocket);
    socket.Parameters.Certificate = CertificateChain.BuildFrom(certificate);
    await socket.NegotiateAsync();
    Task.Run(() => HandleClientAsync(socket));
}

The first problem is that the three lines below AcceptAsync can throw an exception, which would make the StartServer method to fail. For example, NegotiateAsync will fail when an outdated TLS client attempts to connect, or if the client does not accept the certificate.

Another problem is that await socket.NegotiateAsync() can take a lot of time to finish in some scenarios. During this time, new connections won't be accepted. For example, if the client is a slow device, or if it asks the user whether to accept the server's certificate, this could make NegotiateAsync take several minutes.

Consider calling HandleClientAsync as soon as a connection is accepted to prevent this. For example:

while (true)
{
    var plainSocket = await listener.AcceptAsync();
    Task.Run(() => HandleClientAsync(plainSocket));
}

And make sure to handle errors from NegotiateAsync (writing them into the log might be useful).