I think that option 1 (Cancelling the operation in FileSystemNotifier.RenamePreview event) is correct.
I understand that message returned to the client is confusing.
What message do you need/prefer on the client?
1) We can prepare beta for you. In the beta we will return better message whenever FS operation is canceled in the preview event.
2) We are considering to add better support for the error messages generated from the custom file system.