I don't remove my males and have never had a problem with them being in there with the mother. I think there are several people on here who do separate and have not had a problem with that either, so you should be good. Obviously if you remove your male you can control how many litters you have which is good, but I have heard of people who breed mice for pets having issues re-introducing the male after he's been separated from the females.

Good luck with them!