What exactly are you basing that on? How many animals have you applied this thought process to? How many animals have you raised from egg to adult? I must be doing something terribly wrong with my room full of healthy thriving animals.
Based on the fact that daily handling stresses them out and you even said yourself, the care sheet has good advice. 2-4 times a week is good for handling if you want to keep stress at a minimum.