More SubjectInfos in message.Recipients than the number of mail recipients (message.To) is a common practice - mail agents usually encrypt the message for the sender as well (to make it possible for the sender to decrypt the message later), and this appears as additional SubjectInfo in message.Recipients. This reflects the relationship between S/MIME and CMS (PKCS#7), where the two layers are independent but the whole thing relies on them being in sync.
The only connection is actually recipient.Certificate which features GetMailAddresses() method to retrieve a list of certificate owner's e-mail addresses. This can be then used to map the recipient info to a particular recipient in message.To, message.Cc (or a sender in message.From or message.Sender). However, this is somewhat complicated by the fact that these certificates don't have to be embedded in the encrypted message (although it's a good practice to embed them). If they are not embedded in the message and not found in relevant Windows certificate stores when loading the message, it is not be possible to match CMS recipients to S/MIME recipients.