Depends on the colony and how used to your presence they are.
While I do not recommend it I can change cages, picking up the babies hours after birth without any issue.
If one day has passed you should be pretty safe but again it really depends on the female.