I don't remove anything.

I guess it will depends on how furnished your enclosure is and whether or not there would be spaces for the mouse to hide.