It's not a sign of AS, just like introversion is not a sign of AS. It may, like introversion, be more common among Aspies.
Any one sign is never diagnostic, not with ASDs. It's not like you can do a culture and see the bacteria on a microscope slide or something.
NTs can have strong hobbies. They can have poorly developed social skills for non-neurological reasons (social isolation, abuse, etc.). Otherwise typical toddlers, especially boys can have speech delays; and hearing impairments or auditory processing issues practically guarantee them.
Heck, even stimming isn't a sign of autism--plenty of small children do it; people with sensory processing disorders do it; ADHD tends to make it more likely; people with developmental delay do it... even your average NT will fidget or fiddle with something or bite his nails or tap his fingers...
With AS, you can only say for sure when you have a BUNCH of different signs, any of which isn't diagnostic all by itself, any of which could be a sign of something else altogether, and any of which could be a trait that a typical person would have, which simply isn't strong enough to be considered a symptom.
All I'm saying is, Be careful about what you attribute to Asperger's--you're going to have more "uniquely you" characteristics than stereotypical Aspie traits, in any case.