This is a difficult situation because it is hard for any parent to hear their child criticized. At the same time, you'll need to let him know what is going on. I recommend that you discuss it with your boyfriend at a time when both of you are calm. Let him know you have some concerns that you'd like to discuss with him. Explain that you feel like there are some issues going on at the house when he is not there, and that you want to make him aware of them. Try to tell him about it in a non-critical manner. Let him know what is going on, what you have been doing with his son. Explain to him why he might be getting the impression that you are not taking care of his son (perhaps you aren't picking a fight over little things because you don't want to be on his case all of the time, especially since there are so many more important things you have to stay on him about?) Reassure him that you do care about his son and that you love both of them, and that you are only telling him these things so that he will know what is going on with his son during the day, and so that he will understand your perspective on things.
I hope that helps.
It sounds to me like he is reacting to what he is seeing as criticism of his son. It might also be that he is trying to keep things as positive as he can with him since he only has him for a short time each year.
Have you let your boyfriend know how much he is hurting you when he says these things? Perhaps the two of you could sit down together and come up with some basic home expectations for both of the kids, some general "what is expected of you" policies for the household? That way he could address any issues he might be having, and the expectations would be clear and equal for both of the kids.