I can not in the environment advise you not to go to the physician.
let me explain. I have to be responsible in the information and advise I provide. In the end, how you use that knowledge is your responsibility.
I am at a disadvantage here that I can not view the damage. I can not look into your ear with an otoscope.
BUT i can tell you that from a triage perspective,
An urgent and emergency care is care that is required to save life, limb, vision, or hearing if treatement is not received within at least 2 hours of time.
Your situation is not within the 2 hour time. BUT it is serious enough, in my opinion, that if you do not seek medical attention within 24 hours you could permanent damage you hearing from any infection that might set up.
Big question for you: how do you know, that you did not leave a piece of cotton off the swab, even a single strand in the middle ear when you pierced the drum.
How do you know that you did not end up causing a peice of ear wax, that was imbedded with contaminates, to get pushed into the middel ear when the drum broke.
that swab acts like a plunger to some extent.
Ear was is a protectant that when your ears are working right, will adsorb (that is a.d.sorb, not absorb) contaminates and debrie; when the wax forms a large enough ball, it rolls out, often onto your shoulder. (this is why your external ear canals are sloped downward). So we know that the wax contains contaminates. AND we know the wax can carry contaminates outside the ear canal. BUT in your situation, you placed a cotton swab into your ear and pushed in far enough to break the ear drum.
My advise is to see the physician.
He may want to put you on prophylactic medication to ward off disease, or he may do nothing. BUT he has to look at to know.
If you decide to not go to the doctor, be very aware of any discharge and go to the doctro emmediately if you see any discharge such as brown, yellow, green, blue or white.