Handling can stress him out but as long as you don't overdo it (try to avoid more than a few times a week, and right before/after feeding) then you should be fine.

One of my pueblan milk snakes went off feed for 6 months, and she was a tiny little thing. I'm sure he will be okay