The problem is in the behavior of Exchange server. IMAP flags are not fully linked with their Exchange/Outlook representation. This means:
If a user sets a flag to a message in Outlook, it is not reported using IMAP protocol. However if a client sets a flag using IMAP protcol, it is shown in Outlook as well.
Similar is clearing flags. If a user in Outlook clears the flag it is still reported as Flagged in IMAP protocol. However if a flag is removed using IMAP protocol, it is also cleared in Outlook.
This is behavior of Exchange/Outlook, not the IMAP protocol. Other servers do this differently e.g. Gmail.
If your server supports keywords, I suggest you to use it. If not, you have to use something else.
For example, if you use ImapMessageFlags.Draft
in the INBOX folder, the user will not see this flag (nor change it), but you will be able to search for it using IMAP protocol.