Whys this guy really annoying me?

Does he do this with other girls? If not, he may have mildly inappropriate feelings for you. You're doing the right thing by avoiding him.