The first group the has helped themselves to the babies, Feed them off and get another set. From what i have seen the will always do that.

Dont give up! it takes some time but it will come around.

Edited:
I use a rack system. the buckets are 15x20 and i have 1.5 in each. i leave the colony setup and do not remove the males.