Even if he refuses to eat I would still wait a couple of days before handling. Refusal is him saying that he's stressed and handling could increase that stress