They don't ALL die... you just have to have a source for ants to feed them and care for them correctly. Getting the ants to feed them is the major obstacle and a real pain in the rear even if you have the 3-4 decent sources of ants.
Not many people want to bother with all that since things like bearded dragons eat easy to get prey items and have much simpler care, while looking "similar"(to a layperson).
We used to have massive amounts of horny toads/lizards where my family lives in Mayport/Jacksonville Florida. They are exceedingly rare now though. That's over about 3 decades, that they went from everywhere to nowhere. Sad, because they are really neat little critters.