Simple actually. Birds, some insects, some reptiles and a few others use a ZW sex determining system in the ovum, which is in contrast to the XY and XO system which uses sperm.
This particular sex determining factor regarding Bananas/CG only pertains and is linked to the banana/cg phenotype which is why the normal offspring void of this genetic defect are not affected by it. It also only affects male Banana/cg which again shows that it is indeed a SEX linked gene.

There has been a few fish, some insects as well as some human diseases which act in a very similar if not exact way.

Even though the "normal" is XY, XO or ZW one can also have many combinations and many defects. So things like XXY, XXXy, XXYY, XXYYY, XaYa, etc.