Why is he staying with you?
How old are the two of you?
Where did the two of you meet?
How long did you know him before you started seeing him?
Is this the first time he hasn't ome back?
Why was he going to this other woman's house?
Did you try to reach him?
Its something to be concerned about when a man goes to visit someone he's been intimate with and doesn't call or come home in days. Have you heard from him yet? It seems like he's having a hard time figuring out what he wants, he wants to be close, but not too close....he doesn't trust you to go in the other room but he disappears for days? It doesn't sound very healthy, and he doesn't sound like he knows what he wants. You have to pay attention, not just to the things he says, but the things he does. If you allow him to walk out and not come home for day with no repercussion, he's going to assume it's ok to do that and will continue to do it.
I welcome your thoughts, let me know if you want to talk more