You need to remember that you are a big scary predator to your snake. It is natural for him to be fearful.
My advice would be to leave him alone completely for a week or two, then start handling again.
Edit: If he is so stressed from handling that he won't eat, then don't handle him. Perhaps there is something wrong with his enclosure that is making him so stressed out?