Here is how you know when you've gone too far: You start getting weird/deformed/odd behaving babies. What do you do to remedy this? Buy new ASFs and start again.
How do you prevent this? Work with multiple breeding groups at once or find places to get new breeding groups, then separate offspring and put future breeding groups together and crosses between the new and old (maybe all old line females, all new line males or something).
If you run multiple breeding groups you could start new groups from the old groups, with perhaps males from one females from another. The more initial breeding groups you start with the less likely genetic homogeny will crop up and leave you with mutant babies.