I'll point out from the top that I'm not a lawyer.
That said, my understanding is that you're basically safe as long as you are a sensible, reasonable person. Ever watch a TV broadcast of a sports event and see them focus on a cute kid (or a cute young woman) in the crowd? I can guarantee that they didn't get model releases first!
Nor do TV news crews get model releases when they do a crowd shot (although they do get a release when they do an interview).
When people go out in public, there is a reasonable expectation that other people will see them. That includes things like being inadvertently photographed. Imagine what it would be like to try to get a shot of Times Square on New Year's Eve if you had to get permission from every identifiable person!
There are also gray areas. If your photos are for personal use, you're definitely safe. If the person you photograph is a public figure, you're very safe (that's why paparazzi can do horribly embarrassing shots of Britney Spears on the beach).
If you take a closeup of a non-public person (such as you or me) in a public place, and then publish it on the front cover of a national magazine, things get a bit more interesting. You're probably still safe if the photo is flattering, or if it's clearly arty (look in a camera mag for candid shots). But if it's seriously embarrassing (for example, you caught me picking my nose) I might be able to argue in court that you've damaged my reputation.
Also, remember that people can complain--and sue--even if the law isn't on their side. If you publish a photo that includes a stranger, there's always a tiny risk that the subject will whine at you, or even file a lawsuit that will be very expensive to defend. For me, that's sort of like the risk I take every time I get in a car--small but real. Your mileage may vary.