I don't think saying you have to be a straight man (or gay woman) in order to fully appreciate a pair of breasts is sexist. Once it turns you on, it becomes more than just a body part, the term 'obnoxious' simply doesn't occur to you.
Well people are stupid. There is nothing beautiful about the human body from a completely objective point of view. The only reason you can appreciate a human body in the first place is because you are human. And the only reason you would fully (and that means also sexually) appreciate a woman's body is if you're a (straight) man or a lesbian. There are no other real reasons people find each other attractive, other then in purely aesthetic/artistic ways, which is what you are talking about, and though that has its value, its not nearly as potent.