I try to keep written records of how many babies I pull from each mouse colony (although when I'm short on time, I take care of the mice and skip the records, so it isn't a perfect system). Sometimes I'll get a colony that goes through a round of litters with low numbers, then production picks up again. Maybe the moms were just a little overworked from constantly raising babies, and having a small litter gave them a bit of a break?
If I have a colony with low production through 2 rounds of litters, I feed them off and replace with a new colony.