I completely see where you are coming from. I had the exact same problem with my ex-gf. She was always getting mad at me, thinking I didn't care because I honestly couldn't understand her feelings. I was always trying to explain to her that I couldn't help it and I loved her more than anything else in the world. In fact, I still feel that way about her. She's the most important thing in my life, whether she knows it or not.
I assume your bf is a NT as is my ex-gf. All you can do is just try to explain to him how much you love him and let him know you honestly do care about his feelings. Hopefully, he will understand. I know how hard it is, however, to explain Aspie thought processes to a NT. I tried with my ex. Good luck with it.