Leave him be for at least a week.
You can still do water changes or quick check up. But I wouldn't recommend handling until he feeds at least 1-3 times for you. This is just a way to ensure he becomes an established feeder without the added stress.
This is a 'just in case' type of thing.

And don't worry about him getting to know your scent. He can smell you from just being in the same room or general area.They have an excellent sense of smell.