If you have a general idea of when he gets to #5, I think ideally you want to put him back before that while he's still reasonably calm. Keep an eye on how fast he's breathing - faster breathing probably means he's nervous, even if he's holding still. For lots of prey species that rely on camouflage to avoid getting eaten, holding really still can be a defense/panic response, so that makes it harder to tell whether he's freaked out.
You want to reinforce the idea that you aren't scary and being out of the cage is OK. If he stays out until he's really panicking every time, and you put him back when he's panicking, it reinforces the association of panic with handling. He won't necessarily put together the pieces to figure out that you aren't going to eat him; his little brain will interpret the incident as a close escape from something scary and that won't help him be less scared next time.
If possible, you might also try to avoid grabbing him while he's asleep and instead take him out when he's already in an exploratory mood. If he's out and about and you open the cage, he might even venture out on his own.
Reply With Quote