My biggest concern with breeding a very small male would not be fertility but that the male might not have enough reserves to carry him through going off feed in breeding season in good condition.