You are going to get a bunch of differing opinions on this topic.

In my opinion, as long as they don't seem stressed (balled up, aggressive), you can handle them as long as you see fit. Sometimes I'll have my guy out for an hour and other times only a few minutes. Depends what he feels like. If he's exploring, I'll keep him out but if he is just sitting balled up or not moving I'll put him back.