The sender is most likely right. The first error was most likely triggered by a recently-discovered bug, while the second error is caused by a known limitation.
Update: This has been fixed in Rebex Secure Mail 2017 R6.3.
1) "Unsupported OAEP encoding parameters source" error occurs when the encrypted email used a non-empty input parameter (also known as label or P), which is a kind of salt for the OAEP calculations. This bug has been reported by another user few days ago. We have already fixed it and will release a hotfix shortly.
This issue was not detected by our automated interoperability tests because of lack of real-world test data - we have apparently only been testing messages with empty labels (which seems to be a default behavior in OpenSSL and GPG's gpgsm
utility as well).
2) "Unsupported OAEP hashing algorithm and mask generation algorithm combination" error occurs when the hashing algorithm used to compute the hash of the label is different than the hashing algorithm used in the OAEP mask generation function.
This is actually correct as well. However, Windows cryptographic APIs do not support RSA/OAEP with non-matching hash algorithms, and we expected such messages to be very rare, so we decided not to initially support this.
However, based on your feedback, it looks like such messages might not actually be as rare as we hoped, so we will add support for it shortly as well. (Due to lack of support in Windows, we will have to use a custom implementation, which means it will not work private keys stored in Windows key storage unless they are exportable.)