Put him in the tub with a couple inches of lukewarm water with a dishtowel folded on the bottom to rub against. Leave him in for about 15 minutes and then have him run through the damp paper towel a few more times. Does the trick for me when needed.
Ok, just finished a 15 minute soak and 99% of what was left came right off. He has a tiny bit left right at the tip of his tail, so back in the tub for a few more minutes. Hopefully this will come off just as easily. Thanks for the help!