I feel that I've worsened with age as well. Some areas have improved, though I attribute those mostly to life experience and the patience that often comes with maturation. There are other aspects of my life/personality/functionality that certainly feel weaker than when I was in my 20s or 30s.
In my case, I have a [completely unprovable] hypothesis: my wife and I went through a series of significant life traumas in close temporal proximity, which has effectively "tapped out" many of my internal & emotional resources. Sort of a low-grade PTSD, you could call it.
I certainly do not function optimally on my own either. Every day I count myself lucky that I met, befriended and married my best friend (who turned out to be what Attwood calls an "extreme neurotypical"). Our strengths and deficiencies complement each other very well; we each have said that together we make one really awesome person, but that separately we each function as below-average individuals. In my younger days I had problems, mostly due to naiveté, but think that I was "more above-average" than I am these days. When my wife is out of town, I slip into a sort of cocoon mode -- even as I become more lonely, I leave the house less frequently and have increasing difficulty making necessary phone calls and similar things.
All of that said, I wouldn't trade my present life for one moment of my pimply, hormonal, perpetually-confused, outsider-y adolescence.