I've heard that mice reach sexual maturity at around 6 weeks of age. So basically your mice are not too small. I would get another male and see what happens, don't have both males in the same cage though, that could be trouble.