I'm not sure a snake would have the ability to associate neglect with a person; neglect being the absence of care rather than actively harming and badgering an animal. I just don't think they're complex enough to have a concept of a person's obligation to care for them, so they can't understand the concept of a person not caring for them. I do however think most snakes have a certain set of 'triggers' that will cause them to go nuts (coming straight at their face, picking them up from the wrong angle, pressure from below not being a problem but pressure from above typically causing a defensive reaction) and that people familiar with snakes instinctively avoid these triggers when handling them. This is why you can handle wild snakes without getting tagged if you play your cards right; it's all about not tripping those natural defense mechanisms. Someone neglecting a snake probably never bothered to learn how to actually act around them to avoid getting a bad reaction.
That's just my take on that anyway. I think initially handling a snake is down to how good you are at reading them, and that trust through conditioning is something that develops over time with a good owner. I agree though, they are smarter than most people give them credit for.