When I'm walking around NYC I'll take whatever bag I need to carry the gear I wanted to bring. Usually it's just a small Gregory Iso backpack with an insert. It's not camera specific and I can fit my 50D w/17-55 mounted, 10-22, 28mm 1.8, and my flash...as well as all the little things. I can also fit a tripod on the outside. I us this bag on a daily basis for pretty much everything.
If I want to take more with me I'll take a Timbuk2 messenger bag with an insert. It's not as easy to carry but it fits everything.
If I want to travel light, it's simply a Tamrac Digital Zoom 4. I can fit my camera w/ lens mounted, memory card, battery, small blower & brush, and a lens cloth. Just the basics.
There are so many people walking around the city with tons of way more expensive gear hanging around their neck. I wouldn't even worry about being discreet in Manhattan unless you're in some really seedy areas. The other boroughs I can't really comment on as I don;t get to go to them as often.