The delay occurs during the calculation of Diffie-Hellman shared secred. On non-Windows platforms, managed implementation of Diffie-Hellman algorithm is used to perform these calculations, which is substantially slower that the native implementation provided by Windows. On low-performance devices such as Raspberry PI, this calculation takes a very long time for long Diffie-Hellman keys. Relative slowness of Mono compared to .NET or .NET Core also makes this problem more severe.
To mitigate the issue, lower the maximum allowed Diffie-Hellman key size to 2048 bits on low-performance platforms, which is secure enough for the near-term future:
var server = new FileServer();
Alternatively, use a plugin to enable Elliptic-curve Diffie Hellman support. Elliptic curve cryptography offers equivalent security with smaller key sizes, which makes it particularly useful on low-performance platforms.