Hearing is more fundamental than sight, or touch, or smell. That's why noises wake us up. Our ears are ever alert to danger. As you said, if the car alarm had turned into a scream you would have taken notice, because screams indicate danger.
I've often wondered if this accounts for the greater emotional impact of music over the visual arts. I've never burst into tears at the sight of a painting, but music can do that to me. But then maybe that's just me. Maybe I am emotionally stunted when it comes to visual art forms.