It wasn't clear from your post, but if you're touching the snake in its enclosure, maybe this is causing him stress because he feels like you are invading his space.
I think that reptiles can tolerate touching and petting, but it's not something they... want or would enjoy, just because they are not mammals and aren't programmed to know what that sort of contact is, and don't view it as bonding or reassuring. Like, I can pet my iguana but he will just sit there with his eyes closed waiting for me to stop; I know he doesn't actually enjoy it, but he knows I'm not a threat and is literally ignoring me. With our Japanese rat snake, he doesn't really like being touched while in his enclosure and will sort of twitch or move away from your hand (not dart away, but will start to relocate), but outside of his enclosure, he doesn't seem to care at all. I can touch his tiny little face and he doesn't care, but I don't think like he'd like that in his enclosure (I haven't tried because it seems mean, lol).
In the long run, I think the best thing to do if you want a handleable, trusting snake, is choice-based handling. This may take a little longer initially (like spending time near the enclosure and eventually opening the door to see if he will come out) but this would be a good way to have him be comfortable with you and have a positive association with you. It will build both confidence and trust in your snake if he thinks it was his idea that he could go explore or be handled. After going through the initial steps, you can pick up the snake from the enclosure as "normal."
https://www.youtube.com/watch?v=4ljtIS4qCKw
Lori Torrini has a lot of videos on training ball pythons as well!
I like to keep in mind that reptiles are kind of on a slower "plane" than people, so things may take longer. It may take several months for him to get used to you. But what I like about this method is that you can tell by the snake's reaction when it is time to progress to the next "step."
If you want to watch a longer video about the subject of choice-based versus force-based handling, you can watch this one (I haven't watched it yet, but plan to):
https://www.youtube.com/watch?v=LPKpTh79Oe4