I disagree with holding him more often and for longer periods of time so that he can get used to your scent.
He's clearly stressed out by being handled, so forcing him to endure more stress makes no sense at all when you think about it, it will only serve to make him associate you with a negative experience.
I would take him out for VERY brief handling sessions once a day (but not during the 48 hours immediately after eating).
The moment that he relaxes even a little bit, put him up. What you are doing, is reinforcing calm, relaxed behavior by rewarding him with something that he desires (which is not to be held and to be put back up).
Good luck!