Mice, unlike rats, do just fine with harem breeding. Leaving the male in with them full time is just fine, and many breeders on this site, including myself, do it. Males are also not prone to eating the babies, that's usually the mothers. The males also help tend the babies, as will other females most of the time.