usually at around 3 weeks old they can actually survive fairly well... that's when they actually start hopping and wrestling with each other and get the zoomies. They'll be steady on the feet, not wobbly. But that's the earliest. 4 weeks out is safest to move them as they should all be eating solid food well by then.
It's sometimes a bit hard to judge, but if you watch the next couple of litters, you'll recognize the size and be able to move them easier. Any time you see them getting water from the spigot with mom and grabbing the blocks that slip through the wire should be fairly safe. On the downside, if they are not ready to be moved, they have a very high metabolism so they can weaken quickly. I'll sometimes leave a teenager with them just so they have a guide to figure the ropes out while leaving mom to rest.