I wouldn't call not handling one neglect. IMO they tolerate being handled but are fine without. Just expect an animal that hasn't been handled to be a bit more skittish when you get them out (or not, some animals are super chill)

Sounds like he may have seen a reflection in the glass that either scared him or looked like food.