You are right. When creating an attachment, content of the file is loaded into a memory object. You can remove the file from disk just after the Attachment constructor has finished.
If you would like to restore the file later, you can always save the attachment to disk like:
Just for the information: If you would like to save memory and you don’t want to load the attachment content, you can do it by setting the MimeOptions.DoNotPreloadAttachments option on the Attachemnt object. In this case you must not remove the source file.
MailMessage msg = new MailMessage();
Attachment att = new Attachment();
att.Options = MimeOptions.DoNotPreloadAttachments;