I normally wait 2 days for a very breif handling, like to have them hang out on my shoulder while I clean out their tub if they made a mess, and then put them back.

I wait for three if I'm going to spend a descent bit of time with them. I have never had any problems doing things this way, but then again I've never done it any other way, just operating on what I've heard here.