Thank for asking this question. Yes you do attract negative karma, because in theory, no one really "knows better" to different varying degrees. If everyone truly understood the nature of karma as "cause and effect", no one would ever make a wrong move. The important thing is for us to examine our own conscience, and right and any wrongs we may have created, either knowingly or unknowingly. Every person, whether sighted or blind, hearing or deaf, mentally typical or differently-abled, has a role to play in their own, and other's karma. Sometimes that karma is played out through one life time, or many. The picture is so big, and time is so long and unending, we as individuals are not able to see it perfectly.
I hope this answers your question - if not, please continue to ask more, until you feel comfortable that your question has been answered adequately.
Are you creating bad karma when you are five and have no concept of what karma is, though?
Are you creating bad karma if you have a severe mental illness and shoot someone to death because you think they are the devil?
I get that once you understand that there is a karma, you are held to account, so to speak, but what about cases where individuals are children who are innocent or ill people who are delusional?