Problem with SignedData.Decode when verify is done with sign -outform pem using openssl

Question

Hello,
I have implemented a Rebex provider that verifies signature and decryps signed and encrypted files.
In general works fine, but when i encrypt and sign a file with the following commands using openssl a get an error on decode method of SignedData object in Rebex.Common.dll:

Exception thrown: 'System.Security.Cryptography.CryptographicException' in Rebex.Common.dll
Additional information: Unsupported ASN.1 type encountered.

opemssl commands i use:

openssl cms -encrypt -outform **pem** -binary -aes256 -in C:\OpenSSL-Win64\bin\TPS-218\initfile.zip -out C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc pemMVgl.pem

openssl cms -sign -outform **pem** -nodetach -binary -md sha256 -inkey privateMVgl.pem -signer pemMVgl.pem -in C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc -out C:\OpenSSL-Win64\bin\TPS-218\signedEncryptedfileDER2.zip

But when i use der instead of pem, it works fine.
commands that my Rebex wrapper can verify-decrypt:

openssl cms -encrypt -outform **der** -binary -aes256 -in C:\OpenSSL-Win64\bin\TPS-218\initfile.zip -out C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc pemMVgl.pem

openssl cms -sign -outform **der** -nodetach -binary -md sha256 -inkey privateMVgl.pem -signer pemMVgl.pem -in C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc -out C:\OpenSSL-Win64\bin\TPS-218\signedEncryptedfileDER2.zip

Related issue that i found but not helped
http://forum.rebex.net/6813/signing-verifying-a-file-with-a-pem-file?show=6834#a6834

What I need is my Rebex wrapper to do the operation regardless the above difference. Please advice.
Extra info regarding my implementation:
I have the certificates imported in trusted windows store, so using a certificate finder that i pass to the SignedData object i decode the data. But in this case it seems in the beginning of the decode method that it is passed an unsupported content.

 byte[] content =...read bytes(the signed file)
signedData.Decode(content);// this usually works except fr this case which throws the exception above

FYI: I cannot change the ssl commands from pem to der because they are used by another system that i have no access.

My client has a similar issue decrypting :

system.Security.Cryptography.CryptographicException: Unable to decrypt data (0x80090005).
at kftm.rxkm(Byte[] ggv, kfwo ggw)
at Rebex.Security.Cryptography.AsymmetricKeyAlgorithm.Decrypt(Byte[] rgb, EncryptionParameters parameters)
at Rebex.Security.Cryptography.Pkcs.KeyTransRecipientInfo.smqc(Boolean gnd)
at Rebex.Security.Cryptography.Pkcs.EnvelopedData.GetSymmetricKey()
at Rebex.Security.Cryptography.Pkcs.EnvelopedData.cggr()
at Rebex.Security.Cryptography.Pkcs.EnvelopedData.Decrypt()

Comments

We will look into this. We aim for interoperability with OpenSSL. Which version of OpenSSL did you use? Output of "openssl version" command could be useful.

---

OpenSSL-Win64
OpenSSL 1.1.0c  10 Nov 2016

Answer 1

Hello,

There is only slight but important difference between OpenSSL's der and pem formats. The DER file only contains binary ASN.1 data. The PEM file contain the same data, but Base-64 encoded, with a header and a footer. Rebex SignedData and EnvelopedData are currently unable to parse this - we'll add support for it to one of the next releases.

Update: This has been added in Rebex 2020 R5 release.

In the meantime, you can work around the issue by converting a PEM string loaded from a PEM file to DER data:

public static byte[] DecodeBase64(string text, out string typeName, out string info)
{
    if (!text.StartsWith("-----BEGIN ", StringComparison.Ordinal))
        throw new FormatException("Invalid header.");

    text = text.Replace("\r", "").TrimEnd('\0').Trim();

    int p = text.IndexOf('\n', 11, 64);
    if (p < 8)
        throw new FormatException("Invalid header length.");

    string type = text.Substring(11, p - 11);
    type = type.TrimEnd();

    if (!type.EndsWith("-----", StringComparison.Ordinal))
        throw new FormatException("Invalid header length.");

    typeName = type.Substring(0, type.Length - 5);

    string end = "-----END " + typeName + "-----";
    if (!text.EndsWith(end, StringComparison.Ordinal))
        throw new FormatException("Invalid footer.");

    for (int i = 0; i < text.Length; i++)
    {
        int c = text[i];
        if (c == '\n' || c == '\t')
            continue;

        if (c < 32 || c > 127)
            throw new FormatException(StringUtil.Format("Invalid character {0} at position {1}.", c, i));
    }

    text = text.Substring(end.Length + 2, text.Length - 2 - 2 * end.Length);

    int e = text.IndexOf("\n\n", StringComparison.Ordinal);
    if (e > 0)
    {
        info = text.Substring(0, e).Trim();
        text = text.Substring(e + 2);
    }
    else
    {
        info = null;
    }

    return Convert.FromBase64String(text);
}