This is not normal. Some swelling after cast removal is to be expected, but the swelling you describe is not normal. Especially in the distribution of the areas of concern, is cause to contact your orthopedist today.
A blood clot in the upper extremity would be rare but is a consideration.
A compression neuropathy is unlikely but a possibility. This would be a consequence of a cast that was too tight by a hair, and when removed your body would cause an inflammatory reaction and the swelling and pain would result.
Infection is also unlikely but a possibility.
In short, this may be very temporary and go away over the next day or two, but, it is not worth waiting... Time to get on the phone or go to your Orthopedists office for a quick look. This is unlikely to be a 'missed fracture', but they will likely want another Xray and certainly a good exam in person.
I am standing by if you have questions about my answer.
What is compression neuropathy? The swelling is better today but the soreness and tenderness over the back of my hand is still there. It gets worse with movement.
Compression neuropathy is simply numbness and sometimes pain from a part of your body being compressed. You will notice this if you sit too long without moving or, sometimes, if you have a cast on.
Wouldn't be unreasonable to use some ice to the area and maybe an ace wrap or a neoprene splint from the pharmacy. If you still have pain on Monday it's time to call your Orthopedist again.
It is not uncommon to have pain and swelling after casting, but it shouldn't last more than a day or two.