Definitely let him be. Even though his eyes have cleared he is still enduring the process of the shed cycle.

Your snake won't forget you or "de-socialize" over a few days. They don't need regular handling, nor do they enjoy it. Handling daily only further stresses the animal. Snakes are not social animals, and will definitely NOT miss being handled.