The water your using might be too hot or too cold for him.

83 to 87 degrees. I usually just fill a plastic tub up with water that feels SLIGHTLY cool to the touch.
And your swimming him right? Filling your tub or whatever your putting him in to a foot deep. And then letting him swim around?