We'd need more details: how does he seem socially when interacting with you? Does he make eye-contact? Does he know how to be reciprocal in conversation, or does he only respond when you ask him questions, or take over the conversation with one of his special interests? You say his interests are academically advanced compared to his peers: are these interests intense and highly focused, or will he jump from subject to subject almost indiscriminately? Does he have sensory issues that you're aware of; loud sounds, clothing or food textures, lights, physical touch, etc.? Does he understand non-literal language such as humour and figures of speech? Does he know how to start a conversation appropriately, or do the other children walk off because he simply approaches them and launches into a monologue? How is his emotional regulation? Does he have meltdowns when socially or physically overwhelmed? Does he have odd or repetitive motor movements such as flapping, rocking, spinning, bouncing, etc.? Have any of these concerns been brought to your sister's attention by his teacher or school counsellor that you are aware of? Have you personally witnessed his interaction with other children? Does he do better with adults, or children younger than him than he does with his own peer group?
Based on your description, you might simply have a gifted introvert on your hands. If you're very concerned, I would find an opportunity to observe your nephew's behaviour in a number of settings (home, public and, if possible, school (perhaps from behind the fence during recess if that wouldn't be too creepy)), have a list of the diagnostic criteria for autism handy (very easy to find online; make sure you have the DSM 5 version) and check off the behaviours you notice. You can then present it either to your sister for further analysis, or bring it to your doctor and ask his advice. He might also be able to suggest ways of broaching the subject with your sister if you're unsure of how to do it yourself. Good luck, we hope to hear how things progress 
_________________
"Survival is insufficient" - Seven of Nine
Diagnosed with ASD level 1 on the 10th of April, 2014
Rediagnosed with ASD level 2 on the 4th of May, 2019
Thanks to Olympiadis for my fantastic avatar!