Having friends is one thing.......having friends that you can not share is another. Is there a reason why you can not join their activities? There is no reason why a friends gender should come into play in your relationship unless it makes you uncomfortable or he is hiding something in regards XXXXX XXXXX friend.
You may want to sit down with him and let him know that you have no issues with them being friends, but why can you not be friends together and do things together instead of them being alone? Let him know that you love him but the fact that he wants to be alone is suspicious in itself. If he has no desire to change this then you will need to follow your heart.
In the end why is he putting this issue ahead of your relationship? While he has the right to request that he has freinds......the fact is friends tend to be such as which are acceptable in the group not alone.