Is this just with people you are unfamiliar with? I have this happen when I have not spoken to the person enough and haven't adjusted to their tone of voice almost. Familiar people I am fine with.
What sends me off the deep end is water dripping. Or water in a fish tank is too low and its trickling. I WILL find the noise and I WILL make it stop...
Dogs licking their feet or just obsessively licking in general, Oh man look out.
I know I am not going deaf. I could hear empty rat water bottles two rooms away and through closed doors. I could hear my quietest frogs call in their glass aquariums with the TV on loud. I could hear my opossum waking up on the floor behind me with the TV on loud.