You guys have convinced me, I'll feed him in his tank :)
Just handled him now for a bit by removing his hide to pick him up, all he did was just stay in a ball. Put him back in and he found his way back to his hide. I'll take your advice and feed him before handling him the next time (obviously waiting a day or 2 after he's fed)
Thanks!