Polarity does matter, I couldn't find anything that identifies which of the black wires is the + and which was the - Just try both ways.

I think your problem is the fan ground. the fan ground should connect to the black wire on the output side, and both yellows should connect to the fans +