Since you can't use POP3 directly from a Silverlight client application, you would have to create a webservice offering methods that correspond to actions you would like to perform from the client, such as "Get list of messages", "Get information about a particular message (including attachment list)" or "Get data of an individual attachment of a particular message". The last of these methods could take a message uniqueID and attachment index as arguments, perform Connect/Login/GetMailMessage and then return an array of bytes that contains the attachment. To convert an instance of Attachment object to an array of bytes, the following code can be used:
MemoryStream temporaryStream = new MemoryStream();
attachment.Save(temporaryStream);
byte[] attachmentData = temporaryStream.ToArray();
(In contrast to Attachment.Save, this doesn't save any data to your filesystem.)