I tried handling him that day and he seemed to be okay with it as far as I can tell. I guess I thought since he didn't eat that he would be agitated and it would be better to leave him alone, but I gave it a try and it turned out fine (and now I feel a bit silly asking that in the first place ).
I even let my 4-year-old nephew handle him today for the first time and it also went very well, both for the snake and the kid.
Anyway, thanks for all the replies guys!