You're certainly not alone. I tend to feel like that a lot, as I've always been a, well, if not shy, at least the sort of person to shun attention. I often realise that I'm trying to rush a shot so that I'm not standing there conspicuously for too long. Obviously the quality of your work suffers when you start doing things like that.
I just tell myself that I don't care what others are thinking or if they're looking, I'm doing my own thing for my own reasons. I focus on what I'm doing and ignore everything else. At the end of the day, getting the shot you want is what matters. Those people looking at you weird don't care that you went home devastated because you missed that great shot, and likewise when you've got an awesome picture hanging on your wall, you won't care (or even remember) that some random stranger thought you were a bit nuts.