I used to suffer from full fledged agrophobia. I could not even order a pizza on the phone, or a meal in a fast food joint.
Slowly I adapted and came out of it. At least you are not that bad!
I also had problems as a young adult with "fun" situations. I had a handful of friends and a sister a year and a half younger who got into my "click". Anyway, I had a tendency to get excited about wanting to go out. We would do some drinking to "prime ourselves" before we went out, but usually, in the "pre drinking" phase, I would go over my limit and pass out before we left my house. (Usually before 10pm.)
I was always screamed at by my family "why do you have to be a party pooper, or ruin the fun?" Honestly, this was not an alcohol problem, but it was my way of "getting out of" the whole night. I really could not stand "having fun" most of the time. Occasionally, I wanted to, and had a good time, but most of the time, I just could not handle that level of socialization.
I have also had full fledged panic attacks in really busy stores. The only place that happens.
I have no answers for you, no suggestions, only support.
Been there, done that, fun is not fun! Blah.