It is unlikely that the magnet harms your pacemaker.
It is not strong enough.
You don't need to worry about it.
Would it be a good idea to keep it a minimal distance from the device?
You don't even need to think about it.