If you mean all as in me and Wild Bill, than yeah, leave em together.. lol

I think you should do whats comfortable. I don't think it'll hurt either way, unless you have ruthless, cold-hearted mice.
Okay:
senerio 1- you leave em all together.
they could a) all live happily ever after
b) feel crowded and eat babies (which wont leave a mess)
Senerio 2- you take male out
they could a.) be fine raise babies, and you introduce male back and all is well
b.) be fine, raise babies, and you introduce male back and they go on a murderous rampage on male
c.) eat babies, put male back, and they make more babies.
d.) eat babies, and you put the male back in and there is STILL a murdereous rampage.

As far as Im aware, the eating of babies is a maternal instinct, not a faternal one. *shrug* I still certainly say try what makes you feel more comfortable.