I do both. I like to add new blood to keep up the genetic diversity. I have added adult males once or twice. Since they are with females or by themselves - no problems.
I have also added very young outsiders to the colony to be brought up by my best moms - both have worked without a hitch - for me.