With mice you can never really predict, it really depends on how social and used to your presence they are, how skittish they are etc
I have seen somewhere you would clean the cage and they would eat all the babies afterward.
I my case when it is cleaning day it is cleaning day, I never had any issue with them even if the babies were born a few hours before.
It is all about knowing your colony and try, if you try and there is a problem you will learn and know for next time.