The good news: he's eating. So just leave it and let him do it his way. Don't dangle too much, just enough so he knows it's there. Then leave it.
As for handling, I would say wait one more week, assuming he eats next Fri. I would give him the weekend to digest and handle him Monday.
I would start slow, maybe 2-3 times a week for about 10-15 minutes to start. Don't handle Friday since that's your feeding day.