It is possible it's just bad stock, and stress. I suggest getting the best mice that you can find, and try breeding them. That's always my suggestion. I personally breed mice for show, and feeders. So I have very high quality animals, that almost never have litter problems, or personality problems.
There are also high quality feeders vs low quality feeders. There's almost no way to tell just by looking at the mice themselves.