Same time. I usually pull at 3.5 - 4 weeks and have never had issues holding rats back for breeders. I choose my holdbacks usually a week or two after weaning, and I've never had issues. Tons of people wean at that age, and it has been done for many generations.
5 weeks is the absolute maximum age you should pull the male pups from mom's tub, since they can impregnate her at that age. 6 weeks is when you need to separate young males from young females. They are usually ready to wean as early as 3 weeks though, so weaning at 4 is not too early.