On my way to work, I always notice a large advertisement for the Museum of Contemporary Art. They usually feature some visually compelling, sometimes humorous, image from their collection with the line "It's not modern, it's contemporary." I think it is a quite interesting differentiation, and certainly one that applies to classical music as well as the visual arts.
I have no idea when the word "modern" became such a horrendous word, but just the utterance of it sends people running for the exits. The idea is expressed in the MCA's ads to say the very least. They make the distinction in favor of the word "contemporary," a simpler word conveying the idea of right now, as opposed to the cutting edge, the implication of the word "modern."
I think people, including myself, don't like the word "modern" because it implies a certain arrogance. It conveys the idea that whatever is given that title is something we are supposed to like if we were "with it." It anticipates what will be important as opposed to the idea of letting the importance be bestowed by the viewer. The word "modern" dictates instead of suggests.