It depends on the animal. With most established animals, you're probably fine. There are some that seemingly refuse a meal if you even look at them wrong on feeding day. There are others that seem to know what day it is that I wouldn't trust putting anything inside their cage that you didn't want getting tagged. With time, you'll get to know their personality and learn for yourself what you can/can't do on feeding day.

In your situation, I'd resist your urge to handle him until (like GoingPostal said) he's eating reliably in your care. For me, that means 3-5 meals without a refusal. Best wishes to you on feeding day #2!