He's terrified of you. Handle him more often, but not for long periods of time. I have an albino cali king that does the same thing. She doesn't scare me, so I just pick her up, and once out of the cage she settles down fine.