I seem to find the sounds of small children more distracting, maybe because of their high-pitched squeals and unpredictability. I remember one time I was in a library to get some peace and quiet before my bus was due, and these two women had a toddler with them that kept running around the library screaming at the top of its lungs while the two women just let it carry on. The noise was ear-piercing, and I couldn't help but frown at them, and then one of them suddenly picked the kid up. It stopped squealing then.
I love kids if they're my own (what I mean by that is kids of friends and family members). But random kids in public do my head in. A couple of weeks ago I went out to a restaurant with my aunt and cousins, and right on the table next to us there was a baby in a stroller at that irritating "I want to speak but I don't know how so I'll just shout instead" age, and by Christ was it loud. I couldn't enjoy my meal properly, as the noise of the kid drowned out everything else and was hard to ignore. I just don't have the filters in my brain to be able to ignore the shouts and screams of babies and toddlers.
I live below two of them, but that's another story. 