Honestly I think it would be quite challenging, and mentally exhausting, day after day (although likely quite rewarding at times too).
I know for me personally I would get far too drained trying to deal with that, but maybe it's just my personality type. I think it's worthwhile remembering though that while being on the spectrum would give you a lot of insight and empathy that many others would lack, as it *is* a spectrum I imagine you'd come across people that you couldn't relate to at all despite their dx. And they might even end up resenting you telling them that *this* worked for you...
If I had to do care work, I"d probably do aged care because I imagine they'd be slightly less challenging, and more predictable (to an extent). Mostly though I'd do pretty much anything other than care work, way to hard of a job for me.