I feel that way too. It's just part of our "disorder" as it's called. I can't offer much help but I can at least say I relate exactly to what you're saying.
When I enter a class and there are other people there, they become silent. Oh look, here comes the big dumbass who can't learn anything because he's mildly retarted. I've struggled with intellectual disability my whole life and low confidence as a result, and I've become a magnetic force for these idiots.
Just hoping that one day a secret society can be formed for people like us, where we can pursue our study of something more important instead of being destroyed and beaten down by a confused and deranged society.