Cerapter have a bit of a point. When a case of non-drinker arrives, his friends does seem to try to justify the fact that he's not drinking to outsiders of their circle. Case in point, a friend of mine invited me on a cabin trip with some friends. One of them didn't drink, and I was told at least once that he didn't, but he was a jolly good guy who was just as, if not more social, than the drinkers. Which was true. But he wouldn't have needed to tell me. I do sometimes go to parties sober, or almost sober, and when I drink (which is most of the time), I often don't drink myself very inebriated (though sometimes, I very much do. Like when there's free homemade wine tasting like lemonade available).
You're from around here, Cerapter. You should go and find yourself some drinking buddies who don't care much that you don't drink yourself. They exist. I know some. Fuck, one of the members of the group I sometimes hang around with when I'm home is a christian non-alcoholic, I believe. and the rest of the gang are metalheads.
All in all, I believe that your own belief that people who drink look down upon you for not drinking has a much greater effect upon you not being able to go out with people who drink than the effect of people actually looking down on you. Because apart from some childish guys who you don't want to hang out with anyway, most people don't. They may try to trick you into drinking, but that's just messing around with you. We mess around with each other, too. Like this last party I was at, when a guy put his newly emptied beer down and went to the toilet, and came back to find it almost empty. Point is, don't take good-humoured messing around for some sort of bullying.
As for not drinking, I don't see a reason to do so myself. I like alcoholic beverages, and I find it fun to be inebriated. And I am in control, I know what I'm doing even when I'm drunk.