4-5 Times a week isn't excessive, and as long as you're not throwing them around shouldn't stress them out very much. Others on here handle theirs every single day with no issues.
I take mine everywhere I go.