They rely on their heat pits more than their eyesight for zoning in on prey. If your finger was warmer than the mouse, then it makes sense that he was following the smell (close enough to your hand) and zoning in on the heat signature of your hand.
I just offered a f/t rat to my pastel female that I knew probably wasn't warm enough and she struck past the rodent towards my hand (missed me though - I knew what she was about to do). I took the rodent, re-submerged it in hot water for a minute and re-offered it to her and she struck true.