Rebex DLLs for .NET 4.6 might be a bit faster in some scenarios (mostly TLS 1.3) than .NET 4.0 DLLs because they utilize .NET's await/async infrastructure. They can also take advantage of custom implementations of RSA class because .NET 4.6 finally made it possible to extend it. Other than that, Rebex DLLs for .NET 4.6 and 4.0 are equivalent and it's safe to use .NET 4.0 DLLs in .NET 4.0-4.8.
Visual Studio's assembly reference resolving logic seems to be a mess. It looks like it never occurred to Microsoft developers that DLLs for both .NET 4.0 and .NET 4.x might be present - it looks like they simply pick the first suitable version without searching for the best one. To work around this, it's best to add the reference via the "Browse" tab in "Reference Manager", which adds an appropriate HintPath as well.
Alternatively, a better approach is to use Rebex NuGet packages. This way, Visual Studio does exactly what it should - you get .NET 4.6 DLLs if your project's version is set to 4.8.