Deborah's advice was spot on. Mice are quite a bit different than rats. Rats typically don't mind being moved around and can adapt pretty easily with other cage mates, however mice seem to do better with a stable colony where they have established their roles within that colony, and that can take some time. My advice would be to leave them alone as much as possible without handling them or moving them around to different cages. Remove the young ones after they are weened to a different cage to raise them up to size, but keep the same breeding adults together.