My personal rule of thumb is that if the sun is shining, I only handle them if I have to for hubandry reasons. Once the sun goes down, it's on like Donky Kong.

There really are no hard and fast rules, and your mileage may vary.