Ok, I would just slow down a little bit with handling for a bit til he gets more comfortable. Once you build a trust for one another and he's got a bit more size on him he'll likely chill right out.
Instead of three consecutive handling days maybe cut out Friday so he's got a day in between to just chill.