I agree that you have the right idea with starting fresh. That always is a great option when things go off track like here.
My concern however is that shift from nice to aggressive kind of. That can mean lots of things so I will give you an examples 1) he is aggressive sexually when he doesn't get what he wants and you didn't know that; 2) he doesn't like women who tell him no; 3) he is sexually immature and doesn't take no seriously 4) he was having a bad day ; 5) theres an overly aggressive person behind a nice facade; 6) he is not really a nice guy; 7) he just misunderstand the sexual cues and made a mistake (this is very possible)
Return his contact and explore a little bit where this has led as far as his expectations. Find a compromise and go for it