Says who? I've been breeding mice for 18 years. It does not stunt their growth. Maybe you are working with an inferior strain of mice. Maybe you are outcrossing your mice with weak strains. I will have 2 litters by the time you get your first. The only time I've seen sick mice is when temps are too cool and when they get old......8-10 months.
The mice I worked with are 50-60 grams at retirement. Would you call that anorexic? Optimal health? We are talking about animals that we feed to snakes. I have never had any trouble with cannibalism or losing mothers and babies. My average first litters have always been 8-10 babies. Average litters after that have been 14-16 and topping out at around 20.....and I inbred for MANY generations for YEARS with zero negative results.
Yes, start breeding them at 4 weeks. They are weaned at 21 days and another week is when they are usually breeding anyway. Keeping weanlings in too long is what wears out the females. I can start with a group of 5.25 mice breeding and in just a few months be producing thousands of mice a week. I guess my goals of maximum production are different than everyone elses'. Wait if you wish.
Good luck.