That's pretty common. You are likely to have better luck moving the male from one cage to the next. Or just leaving the male and 2-3 females together and pulling weanlings out.
Good luck.
^This^
I prefer to move the male and leave 3 females together at all time, I chose not to leave the male at all time because I don't want back to back breeding but harems are also a viable option.