What kind of rumors? Like you're bad in bed or something? How bad are they?
I think the guys who are talking down about you actually are shooting themselves in the foot. The truly attractive non-needy guy doesn't blatantly insult other guys, and also doesn't let the stupid made-up opinions of others affect him, right? I can totally understand how annoying it is, though, I've been cockblocked like that before.
In that experience, I still got the girl that this guy was desperately trying to hook up with by talking shit about me/chasing, and then during pillow talk after sex she would tell me how annoying he was, haha.
My 2 cents.