I owned both of them for my 7D. I kept the 24-105. Why? Well, I got mine for $500 in the 5D II kit deal and to be honest it's just a better lens.
I find my 24-105 consistently sharper and to have better colors. Plus, it's just built better. I also prefer the range. When I think of "walk around" I think of that 20mm (on crop) to 100mm (on crop also). I'm often using foot zoom or lens zoom when walking around.
Less than 20mm for me is "tripod time". And in this case I prefer a lens that goes from 10mm to 20mm.
I found the 15-85 to have inconsistent IQ (especially on the edges), to have a slight dimness, to have substandard vignetting, to have lens barrel creep (come on, really?). The lens just did not provide confidence.
Now my setup is the 24-105 IS and Sigma 10-20. Both are matched with top end CP's. The 24-105 stays on my camera constantly. It's good for landscape, portraits, events, and even big wildlife like bison, moose, or horses.
I also prefer the constant aperture. I do not care for variable aperture lenses.
The 24-105 is just in another league than the 15-85, and they are way too close in price. You can find new 24-105's on Craigslist for $800. My father bought a white box (parted from kit)for $850 at Adorama that turned out to b brand new in a red box, lol.
The lens also uses a 77mm front end, which matches my other lenses.
I'd much rather pay $850 for a kit-parted 24-105 on Craigslsit (make sure you get the warranty) then spend $734 on Amazon for the 15-85. Plus, you're getting a lens hood and a nice pouch, too.
Canon is making a mistake by pricing these EF-S lenses too close to L lenses. Their EF-S lenses feel floppy and wobbly, and I don't think any of them should be priced over $600.