Given the history with your girlfriend's ex and his mom, I could definitely see why both you and your girlfriend would be apprehensive about allowing his mom to see their daughter.
I think what needs to happen, although it may seem like it would be uncomfortable, is that your girlfriend (or, MAYBE both you your girlfriend) need to sit down with the child's father and have a serious conversation about the whole situation. Now, you may have already spoken with him about this, perhaps informally, and/or argumentatively, but this is a delicate situation which needs to be handled very carefully and maturely, perhaps over dinner or at least a cup of coffee. There is a good chance that the child's father has repressed a lot of what happened, or that it makes him very uncomfortable. He may simply not want to accept that it has happened, and wishes to live a normal life (you only get one mother) and share the joy of his child with her, forgiving her and turning a blind eye to what has happened in the past. There is no way to know how his psyche is currently dealing with that situation.
To answer your question, no, you are not wrong for thinking its weird and for not wanting to allow this. If he has forgiven her then perhaps you and your girlfriend should search your hearts for that same forgiveness, because on one hand, the victim (your girlfriend's ex) was a male teenager when the events occurred, and from what I gather the child in this situation is a female infant. His mother obviously served her time and probably learned her lesson. However, on the other hand, yes, it is a creepy and terrible thing that she did, and as a parent you do not want your child around someone capable of such disgusting acts. Only you both can weigh out the options (after speaking with her ex) and see what you come up with after careful thought and consideration.
If it were me, I would allow only limited supervised visits, meaning your girlfriend should be there when his mother visits with the baby.
If I have answered your question, please click the accept button. :-)
Thank you in advance and good luck with everything.