I'd take your clothing out. He needs to feel secure in "his" space. Just take him out for small amounts of time a couple times a week. That will get him used to your scent and let him "trust" you. So he knows you won't hurt or drop him.
Just go slow and wake him slowly if he's sleeping so you don't alarm him, and you and he will be fine!