Should I continue to hang out with this person?
It's sad that some people clash so much with others, but it's also sad for people who'd have to allow others to cross their boundaries or make them uncomfortable. I don't think that's suddenly a fair request just because someone else is struggling.
If it's been a couple years and he seems more respectful of other people's space, a chance might be worthwhile. People can grow. I'd also look at the way he reacts to being told that something is making people uncomfortable. If he cares about how he affects people, that's different than not caring at all as long as he gets his way.
This happened last year.
From what you wrote, it sounds that a lot of the story is what other people said. Can they be trusted? It's not a good idea to simply believe what people say. Many people enjoy malicious gossip. They sometimes exaggerate or even lie to make someone look bad. So he might be a victim of malicious gossip. It's just a possibility. If he does something directly to you that makes you uncomfortable, you certainly have no obligation to be around him. Otherwise, you could give him a chance. That's my opinion.
You should never feel obliged to spend time with people who make you uncomfortable. That's not fair to either you or him. Even if he's not aware of your issues with him, I'm sure that if asked, he wouldn't want to hang out with someone who doesn't like him; no more than you'd want to hang out with someone you don't like. Like another user explained, it may take him a little longer to recognize personal boundaries than most people, but that's no concern of yours. It's ultimately up to him to learn how to respect peoples' space.
_________________
“Oh - You're a very bad man!
Oh, no my dear. I'm a very good man. I'm just a very bad Wizard.”
― L. Frank Baum, The Wonderful Wizard of Oz
If the question, and post are put to writing, conventional wisdom says, "when in doubt...DON'T!"
