Do guys automatically think single moms put out?

I'm a single mom, I work and go to grad school. I'm apart of a group and I've met some great people on campus. There is this one guy that just seems to think that he can say whatever to me (ual stuff). I am thinking its b/c he figures I'm just some slut or something. I'm not. Was with my baby's father for years (things just didn't work out) and I've only been with one other man and I've decided NO premarital anymore! The rude guy never talks to the other ladies like that and I've never said or done anything to make him think I am slutty. And there is a guy I actually like in the group (and who I think would LIKE to talk to me) but I think if he hears this other guy saying crazy crap, he will back off. Should I just tell the rude guy to get lost? I don't like conflict, but I'm sick of this guy thinking he can talk to me any kinda way!!!!!