If he was fasting a long time and you weren't handling him much then and not much lately he may just not be used to you or he could just be having a "moment" and couldn't be bothered being bothered LOL. Maybe just getting back into a routine of short, appropriate handling at the same time every so many days will help him calm down for you again.