I haven't met a lot of aspies, but I've met some, and I've had several friends with ADD/ADHD.
I don't really care about what most people think about me, but when I'm around judgemental people I get very self conscious, because if I don't "behave" they will think badly of me, and that makes me feel very uncomfortable, no matter if I know/care about them or not. I don't know what they're thinking about me, but I feel their feelings towards me.
The thing I've noticed lately, since I've started to read more into ASD's and similar conditions, is that most of the people that do not judge me for being different actually have an ASD, ADD, ADHD or something of the sort themselves.
So, it's probably more common for odd people to tolerate oddness in other people than it is for "normal" people. Of course "normal" people also have their oddities, but they tend to suppress them in order to look normal, which often leads to projection, most often through the need to judge others in order to look good themselves. Although I think this also applies to aspies who are in denial that they are aspies, and/or have been indoctrinated by their surroundings to belive that any aspie behavior is "bad" or "wrong".
So, actually I think it has more to do with being conscious and accepting of one's own oddities than anything else. If you know who you are, you won't be as prone to judge others, and if you think that your own oddities are bad and needs to be suppressed, you will judge others by the same standards, aspie or not.
So yeah, I think it would probably be easier for me to hang around aspies than nt's in general, but I don't think it's really connected to the condition itself, but more about the persons attitude towards being "different". Also it's not that I "care" about what they will think of me, but more about the fact that if they think badly of me they are hurting me by projecting negative energy in my direction, which I am very sensitive to.