Yea, I know people use it just fine and keep their numbers up, it's just something to consider.
Nowadays I only give aspen for the first week or more just because it's cheaper and fluffier for nest making, plus it's only one girl and she doesn't need the pellets high level of liquid/stank absorbency, aspen works just fine.
After the two week mark, its definitely all pellets.