Hello and welcome to JustAnswer!
I believe the reason you are seeing this is because of the mechanical link between the front and rear wheels when in four wheel drive.
Obviously in two wheel drive power is only transmitted to the rear wheels and the front wheels and driveshaft are just freewheeling.
When you put it in four wheel drive then there is a mechanical link between the front and rear wheels. When on a hoist the front and rear wheels will either spin or stop together because they are locked together through the transfer case. There is no hydraulic or mechanical reason that the rear brakes would work or not work in four wheel drive, I believe the reason you are seeing the wheels stop when in four wheel drive is simply because the front brakes are stopping the rear wheels through the locked transfer case.
The rear brakes may not be working for a few reasons. It could be rear wheel cylinders or the shoes could be that far out of adjustment. The problem could also be hydraulic.
If no fluid is reaching the wheel cylinders then they can't apply. The problem could be in the master cylinder, proportioning valve or ABS hydraulic control unit. More common though would be a collapsed brake hose.
There is hard line running to the rear which then goes into a flexible brake hose between the frame and differential, and then it goes out to the wheels. Brake hoses will often collapse internally, there is a good chance your rear brake hose has collapsed internally and is preventing pressure from reaching the wheel cylinders.