The mob, Richmond chapter.
I would have just ignored the person which I'm sure you did. If someone politely asks me not to I would apologize and move on. It's much easier to do this in tourist areas where everyone has a camera anyway.
One of mine taken with a D800.
Shooting street with a 28mm you really need to get into people's faces otherwise it just looks too disconnected.
People aren't my interest down there, the historical architecture is. Problem is people are unavoidable and there is always one in a crowd who gets nervous when they see a camera that could be used by the media.