Yes and no.
It happens with certain familiar people too.
I was born here in the US. English is my first language.
But now that I'm really thinking about it, I can understand people better if they have an "Asian fresh off the boat" accent when they speak English.
Like, I am constantly asking my American born boyfriend to repeat what he says. He speaks very clear English.
But I never have to ask my parents, who are immigrants, what they just said in English even though no one else knows what the heck they said.