It is very possible that you are just being a little too impatient. However, it is not uncommon to have a male that just doesn't get the job done, or he'll get a few females pregnant here and there, but not nearly as fast & as often as you want.
I haven't really made a scientific study of this, but it seems with my mice, when I let a male grow up with females (like when I want to breed him back to his mother, so I never take him away from her), he takes a while to get started breeding. On the other hand, when he is separated from females for a short while when he is weaned, then put in a colony at around 4 or 5 weeks of age, he'll start breeding right away.
Maybe it would jump-start your male if you separated him from the girls for a while?