Personally it would depend for me on whether I liked the job or not. Lots of times these kind of stupid things blow over (I did not mean that to belittle your experiance, was just referring to people's short-sighted grudges that are on-mass) with the exception of one or two people who will hold onto thier opinions no matter what. If you just live as you and show anyone who is looking (and if they are trying to prove that you are a shmuck they will be... or they won't be bothering with any of this) that you are just... you and not this a..hole your ex makes you out to be, people will come around. I don't mind being a bit of a social outcast for a while, heck, I sometimes prefer it. However if I didn't care for the job and felt like a change, I'd just pick up and go. Just be warned, if you are leaving to get out of a situation... well, the grass isn't always greener. Just make sure that if you do leave, that you are running TO something, never AWAY from anything.
Whatever you plan on doing, I wish you luck.
_________________
Humm, guess I should put something witty here, huh?