I always keep 'em until I'm sure they've gone bad.. pretty much when you make the call "I'm sure it's gone bad", it's the right call.

Caved in, molded, and smells bad as mlededee said are the things i look for.