I personally wouldn't but its totally up to you. What ever your comfortable with. You can try handling him for a few minutes every 2 days or so, and if he stops feeding for you, or you notice some signs of stress, than you should back off and let him adjust for a while.
Cheers.