X-Unsent header is a custom e-mail header that just happens to be supported by a number of e-mail clients. However, it's not a part of the standard, and e-mail clients don't have to support this. If it doesn't make Apple Mail treat the e-mail as "unsent", then apparently it is not recognized by Apple Mail.
Are you able to create an email in Apple Mail, save it as .eml file, open it, and still see it as "unsent"? If you are, it would indicate that Apple Mail uses another custom header to indicate the message state. Opening the saved .eml file in a simple text editor should make it possible to determine which header is that. Then, adding the same header via Rebex Mail should result in equivalent behavior.