Problem with SignedData.Decode when verify is done with sign -outform pem using openssl

0 votes
asked May 8 by michalis.viglakis (220 points)

Hello,
I have implemented a Rebex provider that verifies signature and decryps signed and encrypted files.
In general works fine, but when i encrypt and sign a file with the following commands using openssl a get an error on decode method of SignedData object in Rebex.Common.dll:

Exception thrown: 'System.Security.Cryptography.CryptographicException' in Rebex.Common.dll
Additional information: Unsupported ASN.1 type encountered.

opemssl commands i use:

openssl cms -encrypt -outform **pem** -binary -aes256 -in C:\OpenSSL-Win64\bin\TPS-218\initfile.zip -out C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc pemMVgl.pem

openssl cms -sign -outform **pem** -nodetach -binary -md sha256 -inkey privateMVgl.pem -signer pemMVgl.pem -in C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc -out C:\OpenSSL-Win64\bin\TPS-218\signedEncryptedfileDER2.zip

But when i use der instead of pem, it works fine.
commands that my Rebex wrapper can verify-decrypt:

openssl cms -encrypt -outform **der** -binary -aes256 -in C:\OpenSSL-Win64\bin\TPS-218\initfile.zip -out C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc pemMVgl.pem

openssl cms -sign -outform **der** -nodetach -binary -md sha256 -inkey privateMVgl.pem -signer pemMVgl.pem -in C:\OpenSSL-Win64\bin\TPS-218\encrypted_fileDER2.enc -out C:\OpenSSL-Win64\bin\TPS-218\signedEncryptedfileDER2.zip

Related issue that i found but not helped
http://forum.rebex.net/6813/signing-verifying-a-file-with-a-pem-file?show=6834#a6834

What I need is my Rebex wrapper to do the operation regardless the above difference. Please advice.
Extra info regarding my implementation:
I have the certificates imported in trusted windows store, so using a certificate finder that i pass to the SignedData object i decode the data. But in this case it seems in the beginning of the decode method that it is passed an unsupported content.

 byte[] content =...read bytes(the signed file)
signedData.Decode(content);// this usually works except fr this case which throws the exception above

FYI: I cannot change the ssl commands from pem to der because they are used by another system that i have no access.

My client has a similar issue decrypting :

system.Security.Cryptography.CryptographicException: Unable to decrypt data (0x80090005).
at kftm.rxkm(Byte[] ggv, kfwo ggw)
at Rebex.Security.Cryptography.AsymmetricKeyAlgorithm.Decrypt(Byte[] rgb, EncryptionParameters parameters)
at Rebex.Security.Cryptography.Pkcs.KeyTransRecipientInfo.smqc(Boolean gnd)
at Rebex.Security.Cryptography.Pkcs.EnvelopedData.GetSymmetricKey()
at Rebex.Security.Cryptography.Pkcs.EnvelopedData.cggr()
at Rebex.Security.Cryptography.Pkcs.EnvelopedData.Decrypt()
commented May 9 by Lukas Pokorny (104,330 points)
We will look into this. We aim for interoperability with OpenSSL. Which version of OpenSSL did you use? Output of "openssl version" command could be useful.
commented May 9 by michalis.viglakis (220 points)
OpenSSL-Win64
OpenSSL 1.1.0c  10 Nov 2016

1 Answer

0 votes
answered May 9 by Lukas Pokorny (104,330 points)

Hello,

There is only slight but important difference between OpenSSL's der and pem formats. The DER file only contains binary ASN.1 data. The PEM file contain the same data, but Base-64 encoded, with a header and a footer. Rebex SignedData and EnvelopedData are currently unable to parse this - we'll add support for it to one of the next releases.

In the meantime, you can work around the issue by converting a PEM string loaded from a PEM file to DER data:

public static byte[] DecodeBase64(string text, out string typeName, out string info)
{
    if (!text.StartsWith("-----BEGIN ", StringComparison.Ordinal))
        throw new FormatException("Invalid header.");

    text = text.Replace("\r", "").TrimEnd('\0').Trim();

    int p = text.IndexOf('\n', 11, 64);
    if (p < 8)
        throw new FormatException("Invalid header length.");

    string type = text.Substring(11, p - 11);
    type = type.TrimEnd();

    if (!type.EndsWith("-----", StringComparison.Ordinal))
        throw new FormatException("Invalid header length.");

    typeName = type.Substring(0, type.Length - 5);

    string end = "-----END " + typeName + "-----";
    if (!text.EndsWith(end, StringComparison.Ordinal))
        throw new FormatException("Invalid footer.");

    for (int i = 0; i < text.Length; i++)
    {
        int c = text[i];
        if (c == '\n' || c == '\t')
            continue;

        if (c < 32 || c > 127)
            throw new FormatException(StringUtil.Format("Invalid character {0} at position {1}.", c, i));
    }

    text = text.Substring(end.Length + 2, text.Length - 2 - 2 * end.Length);

    int e = text.IndexOf("\n\n", StringComparison.Ordinal);
    if (e > 0)
    {
        info = text.Substring(0, e).Trim();
        text = text.Substring(e + 2);
    }
    else
    {
        info = null;
    }

    return Convert.FromBase64String(text);
}
...