I've recently been doing a bit of street photography. I noticed the same thing... people look at you funny (like what the heck are you doing) or hostile (like get away from me!). I went to a saturday market and had a couple folks ask if I was with the newspaper and still other people be very concerned that I was taking their picture! So, I've been trying to practice taking photos from the waist. I don't look through the viewfinder... just let the camera hang and snap shots that way. You may not get what you were after but you do get some very interesting views that you hadn't thought of!