We are lucky in that ours give huge litters (16-18) and are also very hand tame. I've never been bitten, or attacked and even have a couple that take treats and things direct from my hand.

With time, I'm sure your colony will move towards being less aggressive.