I've got a Sony RX100, which is probably as close to DSLR quality as you'll get in a "point and shoot" form factor as of right now. I find that I don't use it as much as I thought I would, as the quality is still nowhere near what I'm used to on the 5D3. When I want to travel light, I usually slap the 50L and a 270ex on my 5D3, sling it on my shoulder (I've got a Joby strap), and head out. It's a very unobtrusive set-up, IMO. I think the key is how you handle yourself when you're out with a DSLR. If you're awkward about it, you will look awkward. If you're casual about it, then its no big deal.
I agree with everyone else that a good camera phone will fill in the gaps nicely for moments when you can't bring your DSLR....if only because it's something you'll have on you all the time (unlike a P&S)