When my boyfriend and I got our snake Syd he was completely fine with me holding but was squirmy and mad whenever he would hold him. Still today he doesn't really like my boyfriend but I figure it has to do something with my scent, sometimes snake can tell who their owner is by their scent. If your snake starts to associate the smell of whatever room he's in with home, then it will probably get better, and give him that scene of security. Just give it time.