The thing you have to remember about snakes is that they don't WANT to be handled, they just tolerate it. Handling is for your personal enjoyment only!So don't feel obligated to pay attention to him for specific amounts of time, as long as you take care of his needs of food, security, water, humidity and temperature, he doesn't care whether you interact with him. By the sounds of it, he's just a little stressed at all the attention you've been giving him. If you back off a bit, give him some time to adjust, then start short handling sessions, he'll probably be just fine!