First of all dont panic this is quite normal, your three year old probably has no idea that knickers and bras are not to be held close to him at all times and are a private item. He is likely to be hiding them as you may be taking them from him.
I am sure its the texture of the item which is what he likes and the association that it is something Mummy wears. Could his Mummy perhaps spare a pair of favorite underwear and cut a square of the material for him to keep? Explain that this is his special silky to keep and that you must not take Mummy's knickers anymore as she needs them. This should solve your problem. Good Luck, if I have helped please click accept.