Babies do tend to have a rather high mortality rate that's only compounded by the hands of a novice keeper. If you decide to get your first tortoise I strongly urge you to pick up a yearling. They are pretty much bullet proof by this time and still small enough that you'll get the baby feeling.
Most tortoises don't enjoy being picked up and held but they do seem to thrive on positive interactions. I'm greeted by my tortoises every day when I make my outside rounds. They follow me around looking for neck scratches and food handouts. Tortoises are some of the most rewarding reptile pets you could possibly own - definitely tops in the intelligent interaction category.