A lot of people who are more deeply into reptiles may think of themselves as collectors or breeders and that means they may be picky about getting reptiles wth the most vibrant colors and patterns that they like. Like anything else, when people buy reptiles they're going to tend to pick the best looking ones first, so if want the best selection you probably want to buy babies before they get picked over. Secondly, these guys already HAVE multiple adult animals so there is no sense of wanting to get a big one because it's cool. That need has already been filled.
For a new/casual collector it's not a big deal. Feel free to get any age you want. My advice would be to go to a reptile show if possible and pick out an animal there. You'll get to see and handle LOTS of different animals and this will help you feel comfortable about your choice. You'll see firsthand that they are healthy, what their attitude is like, and whether you feel intimidated by a bigger snake or whether you find babies to be nervous or feel "fragile" to you.