I understand. I've made similar experiences.
So I'm gonna tell you what I did against it.
I turned around and said loudly but calmly and slightly angrily You think this is funny? What do you want from me? No, really, what do you want? *pause* You're stupid, that's the thing. You think this is funny, but in reality you are stupid. *keep staring for a few seconds, slowly turn away*
After that I didn't have problems with the guy anymore. Note that this monologue is from my bad memory, this happened years ago... so you probably shouldn't just repeat what I said without knowing how I acted with body position and tone of voice and exact words, cultural background etc.
Basically when people get on my nerves I ask them what they want from me. Usually they don't have an answer to that, but it doesn't matter if they do, after that I just always say that I don't want to have anything to do with them and turn away. So far this has worked for me every time, although lately I haven't really had the problem anymore.