Hi there and thanks for the question.
Pain is your guide here as far as urgency. If you are having trouble moving the arm, or if the pain is over your threshold that you can tolerate, then yes, you should go ahead and be seen in the ER. You might be seeing swelling that is making something look 'out of place' but you could also be seeing a bone that is misplaced also. With a fracture that requires surgery (open reduction internal fixation), you should be seen tonight. Again, pain will guide you to seek treatment. Just from your symptoms you have listed, however, it does seem as though you need medical attention this evening.