Flippant answer: An infant is not capable of playing with a gun, so no immediate worries there.
I know people who carry a gun in a holster all the time - yes, even at home. You can't schedule a home invasion. If your gun in in a proper holster, on you, then you won't have to worry about your toddler playing with it when you are in a different room.
Sounds extreme to wear a gun all the time, but some places are getting THAT bad. I wear a sheathed fixed blade all the time (and we have three dogs, plus locked gates on a fence.) Yes, our neighbourhood is getting THAT bad.
Get some training, and find the good sites on the net.
If you don't do stupid careless things with your gun, you and yours won't be harmed by it.
If that pastor's wife had had a gun handy, she would still be alive today.