There is no way to determine whether a file has been encrypted by Rebex Security. This is by design – the encrypted file was intended to be indistinguishable from random data.
However, if you wish to identify encrypted files, you can easily just insert a custom wrapper stream in between the base FileStream and the XtsStream that adds a fixed-length header to the beginning of the file and translates the positions and offsets in Seek method and Position property accordingly.
So instead of XtsStream on top of FileStream (like here), you would have XtsStream on top of CustomHeaderStream on top of FileStream.