I'm not going to lie, the shove probably would of led to him getting punched before he could of said any of that stuff, but I wouldn't recommend it...

Better to get some kind of authorities involved instead of possibly getting yourself in trouble too. Sense this has already transpired, I recommend not being around this clown anymore and making sure whatever way he knows to get into your house is no longer accessible.

If he says anything further or actually shows up at your house I would just call the cops.