what are you running for a biological filter?
It's true about the gravel holding beneficial bacteria, but I was assuming you had enough bio filtration running in addition to that.
If you don't explicitly have a biological filter then it's true you shouldn't remove or clean all the gravel at the same time.