Whether for animals or fish, I always pitch out all the rocks, sand, wood, anything porous at all. Anything with tiny crevices. You wouldn't believe how many tanks I have, that I was sold/given with the proud declaration "You can have all the gravel in it too!". Sometimes I told them that I would throw it all out, sometimes I didn't. I filled all the potholes in front of my shop with fish tank gravel over a 5 year period.