I have the same problem when there are multiple voices. I don't have this problem when there are multiple noises though. Here is an article that might help explain, if it is the same thing.
https://en.wikipedia.org/wiki/Cocktail_party_effect
Quote:
The cocktail party effect is the phenomenon of the brain's ability to focus one's auditory attention (an effect of selective attention in the brain) on a particular stimulus while filtering out a range of other stimuli, as when a partygoer can focus on a single conversation in a noisy room.[1][2][3] Listeners have the ability to both segregate different stimuli into different streams, and subsequently decide which streams are most pertinent to them.[4] Thus, it has been proposed that one's sensory memory subconsciously parses all stimuli, identifying discrete pieces of information and classifying them by salience.[5] This effect is what allows most people to "tune into" a single voice and "tune out" all others. It may also describe a similar phenomenon that occurs when one may immediately detect words of importance originating from unattended stimuli, for instance hearing one's name among a wide range of auditory input.[6][7]
BTW, I am so sensitive to quiet noises that somebody tiptoeing across the room while napping will wake me up, but you can stomp right on by me and I'll keep on sleeping.
_________________
I finally found an avatar.