I would only hold him every other day or so, so he can get used to you but also get time to rest. Don't hold him on feeding day and 48 hours after a feed! Just be relaxed when holding him and he will start to relax. It you tense up then he will feel that and it will make him feel scared!