Hi there and thanks for your question. Congrats on your new arrivals!
There really isn't any problem with handling kittens at any time after they are born. I would however, avoid taking them away from mom's sight as she may get upset at you.
But there will be no harm done in handling the kittens right away. It will actually be good for them to have human socialization right from day one.
I do believe that this is an "old wives tale". In all of my years of practice I have never had a client complain that their animals would not nurse after their kittens had been handled by humans. Now, there may be some truth to the fact that baby birds should not be handled. However, there is no reason not to handle your new kittens.