Hello. Thanks for your question.
I work in the studio often and also perform, so I am familiar with the very thing you talk about. Playing to records and mp3's is a different ball game then playing with a band live. When playing to a recording, you have something to keep you locked in. Obviously, with a band, you are keeping the beat.
It is tough to tell who is right and wrong on this one, because I've played with some musicians who have trouble locking in with the drummer, and thus "they" are increasing the tempo. In this case, I've also seen drummers try to "reign" in the beat to compensate, and that shouldn't be.
I suspect they are wanting you to lay down more of a groove, than be so mechanical. This is a "feel" thing. I remember years ago when I started playing in the studio. I was concentrating too much on the click track and trying to lock in which made my playing rigid. After I relaxed a bit, and took my mind off of that, I played better. I'm not talking about ignoring the track, but rather not focusing so heavily on it to where your playing is mechanical.
In terms of your speed - this can happen when playing live, especially at certain parts where the music tends to build (a pre-chorus and/or a chorus for example). However, from reading your post, it seems to me that you are probably speeding the songs up (whether conscious or unconscious) because you feel that the songs are too slow in tempo. Again, I go back to the musicians. If this is the case, the other musicians are holding back the song, not you. This is probably why they think you are speeding up. They want a certain tempo for the song, which is not what you are used to as the tempo on the recording may be faster. If this is the case, then you all need to decide on the tempo (and all stick with it).
If you want to work on your timing, playing to a click track is really the way to go. Studio work will really help you with that. And again, you can't be so focused on the click track that you are rigid and mechanical in your playing. Be conscious about it, but let go a bit, and play! It takes practice.
Now on your feel - that groove you mention about laying back - that is a stylistic approach that unfortunately, not all drummers possess. For instance, John Bonham of Led Zeppelin was great at laying out the groove, laid back and still in the pocket. This is something that doesn't come over night, as many drummers have practiced years to attain that groove I speak of.
If you have more questions, or more specifics, by all means let me know. There are just many variables to your questions. You also might want to seek out a good drum teacher, at least for a period of time to help out with some of these issues.