To be someone's mistress there would have to be something secret and underhand going on, which there most certainly is not.

Although... I can see where the anxiety would come from, especially if he isn't used to relationships that aren't strictly monogamous in nature.