Do what you feel comfortable with, personally I would take the male out, this is what I do with my colony move the male when the females are pregnant and I never had any issues upon re-introduction.
Some people are successful with harem, some people are successful with rotating the males do what work and what you feel comfortable with.
The only way you will know what work best for YOU is by experimenting.