Playing devil's advocate, but how much time DO you spend with them? I didn't read in your post that he was telling you to get rid of them, or stop spending time with them, just that he feels it is too much. Although I agree you should never give up something you love for another kind of "love", this doesn't seem to be the question at hand. Have you asked him for specific input on what he needs, too? Why should he be the only one to adapt?