I move male all the time I rotate them so they spend two weeks at the time with each female.

What it boils down to is finding what work for your colony and yourself.

Some animals do well even when moved some not so much all you can do is try and see.