If you are breeding mice for feeders there is nothing wrong with leaving the male in. In fact it's better that way because new males introduced later will often be killed.
The general rule is once a colony is established and producing don't mess with it.

Unless you need the male for a feeder, that is.