Yes, they absolutely do stress from being moved. And, no, he does NOT necessarily need to eat!! Boas are very different from ball pythons and they have much slower metabolisms. Many very good reputable breeders only feed their brand new neonates every 10-14 days right from the get-go.
Turn out the room lights and/or cover the cage, give him LOTS of privacy, and wait a full week without offering anything before you try again. He will eat when he feels comfortable and he will be no worse off for waiting.