It's cruel to the animal in question because they have been bred and raised in captivity and are not prepared to be dumped into an alien and hostile environment where they are responsible for finding their own shelter, food and water.
Captive animals can carry viruses or bacteria that they have developed immunities to, but pass those on to native populations who have never been exposed to them before and are not immune.
One little mouse may not seem like it would make much of an impact...but every single time someone releases an unwanted animal into the wild, it adds to an already serious problem impacting local ecosystems. One little candy-wrapper tossed on the ground does not make much of an impact either...but we don't encourage people to throw their garbage out their windows just because it's more convenient to do so. Why? Because if everyone does it, the problem becomes much more severe. We can't control what others do in this regard...but we can each make our own choice to be responsible for taking care of our world and its ecosystems.