Sounds like everything is fine to me. You're right to not handle him. If you HAD to handle him, it wouldn't harm him, but it would probably be unpleasant, to say the least. Shedding is a vulnerable time for them, so they tend to be stressed and edgey...and their skin is likely very sensitive, as well.

I've got my fingers crossed for ya!