Honestly they can have you evicted for breaking an agreement. While it sucks that it exists, you did sign it in the beginning. Whether or not you say they're a collectors item or not, they're still pets because they're live animals. You're lucky to have gone so long without him finding out and they can charge you per animal or something like that, especially since it's in the contract that it isn't a pet property. I'm assuming he's making an exception for one of two reasons.
1.) Because they don't pee or poop inside the house then it's no risk involving damage.
2.) He wants to keep the money for himself because it's a no pet property.
Honestly if it were me, depending on how long on the contract you have left there's two ways to go about this.
1.) You don't pay anything and you move out, but if you're still under lease if they take you to court the court will make you pay the remaining money you owe for the time left on that lease(sucks but it's the truth).
2.) You meet him and tell him that you know it's a no pet policy so the money is going to be for him and make a middle wage with him, offer him $350 just for him to keep it out of the air.
Now while I'd do option #2, there's always a chance when he wants money he could threaten you with this again so I wish you luck.