Stop handling him and leave him alone for a week, then try again. Handling can stress them out a lot when they're trying to get accustomed to their new home.