I fed off all the males except two, and I set up two colonies. There were two females which were already pregnant and I put them in a separate box.
Those two pregnant ones gave birth to litters of 6 each and they are all raised up fine. One of the colonies, none of the mice got pregnant. The other colony, pretty much all the females got pregnant.

Seems like you've had these mice for at least 4 weeks, so new environment stress is probably not a factor.