I've left my males alone for a week or so and they seem to be fine.
I've read posts that say lone rats get depressed so now I try to rotate the females so there is one with them at all times. I currently only have 2 males and my goal is to produce at least 2 litters a week. I keep the pregnant females together until about 2-3 days before they are due to deliver.
So far it seems to be working out fine.