Or maybe every time he goes up there, he is trying to confess his love to you, but seeing you just looking worriedly at him and he just cannot get his point across, his feelings are heart and loses interests untill the next time he feels he JUST HAS TO tell you how much he cares for you lol