I think it is a little unfair to compare street photography to rape. However, I understand the point of the hyperbole.
That being said... I think that there are two things being discussed here that might be at odds...
First is a perceived right to privacy or control over ones image. Which (at least in the US) has no basis of support in the law while you are in a public place. If I take your picture on the street, I cannot be charged with a crime, and you cannot take me to civil court as long as I use the photo appropriately.
Second, is a right to safety and personal security. This does have a basis in law. If you assault a photographer or his camera you would be guilty of a crime (possibly a felony). In fact, in some states all you would have to do is threaten a photographer (got something that could be construed as a weapon... aggravated assault).
That being said... My street style is not the "camera in the face" style. So if you threaten or assault me, you better believe I am pressing charges. So is "defending" your (not supported by the law) right to privacy in a public place worth getting a year (or more) in prison for violating my right to safety?