Yes it could be taken as a racist comment especially if the people you work with are black themselves. Though you didn't mean as racist they will take it that way because they don't think a Caucasian person could possibly know what a black person went through during that time and maybe it's in your best interest to try to refrain from saying it that way try saying it like this: "I worked like a dog" or "I worked my tail off" especially in the work place you have to have a sense of compassion for others you might work with. A working relationship is a lot like a marriage or relationship you have to be understanding and sensitive of others needs so not to hurt their feelings. Just explain to them that you didn't mean it that way and apologize if you hurt anyone's feelings.
Hi Kimberely - I am still waiting for your additional comments to my supplimentary question to you above
I'm sorry I wasn't able to answer you there was a problem with the system on my end, they may have said that because they don't want to have to deal with repercussion. If you are afraid that they will take you to the board then apologize and tell them that you didn't realize it could be seen as a racial slur and that you wouldn't purposely insult anyone in that manner and you will no longer use that saying. In this day and age you have to be very careful what you say because you don't know if someone will take offense to what you are saying. It's sad but true, there are some people that may be threatened by you and use that against you so use your words very very carefully from now on. Keep your professional relationship very professional even if there is someone there that you think you can trust that just isn't the case anymore. The old days of chivalry is gone you can't really trust to many colleagues as far as your working relationship is concerned so you have to keep it professional and on the level all the time. There are many restriction now when it comes to freedom of speech so keep that in mind.