Perfectly fine. I'm pretty sure nobody provides bottled water for them in Africa.

Although, if you smell a lot of chlorine in your tap water, it's a good idea to let it sit out for a few hours before giving it to them.