The 17-55 is f/2.8 where the 15-85 is variable-aperture. I also find the 15-85 to be a bit soft away from center-frame at the wide end. You won't notice it in final prints, but you will if you start pixel-peeping or taking crops from the corners.
The 15-85 also has a fair bit of chromatic aberration, but this is easily-corrected in post and IMHO isn't one of the lens attributes that bothers me in digital nearly as much as it did when I shot film, where it was effectively un-fixable.
If you shoot things that move INDOORS then the 15-85 loses badly as that fixed 2.8 aperture suddenly becomes much more important. But... you have wide-aperture primes already. Do you need the "holes" in your primes covered for available light, moving-subject indoor photography? If not then that advantage disappears.
Don't overlook that the 2mm difference in the long end is actually rather significant, and that you're giving up 30mm on the long side. This is likely responsible for the IQ difference, incidentally, as it is extremely difficult to manufacture a lens that both has top IQ and a wide zoom range. Canon did a VERY good job with the 15-85 given this constraint.
For MY uses I prefer the 15-85 when I intend to use it. But I have a number of other lenses for other purposes; I have both a 24-105 and 70-200 and am considering a 1.4x extender for the latter. I find the 24-105 has better IQ than the 15-85; the latter has a 5.6x zoom range, the former 4.4. No surprise. With a 3.2 zoom range the 17-55 f/2.8 would be expected to have superior IQ to the 15-85 and probably even beat the 24-105L -- and it does. Note the numerical range of all the new high-dollar (and very-high-IQ) zooms in percentage terms -- 17-40, 24-70, etc. All right near 3.x. There's a reason for this; materially over 3x maintaining top IQ becomes impossible.
The 70-200 is the sharpest lens I own by a material margin, especially in the corners, and if what I intend to shoot fits within its operational range that's what I mount.
The 17-55, for MY use, would require that I carry a second lens too much of the time as its f/l range is only about 60% of the 15-85s. Shoe zoom is great right up until there's water, a cliff, a barrier (e.g. in a museum) or some other impassable object in the way. Cropping in post nearly always costs more IQ than the difference in lenses does; that's a mathematical reality, so if you can you always want to compose as you intend rather than relying on post.
If I move to FF then this will change as the 24-105 covers close enough to the 15-85 effective range as to be interchangeable with the shooting I do. But the 24-105 is a much larger and heavier lens than the 15-85 and therefore makes for a bigger and heavier "as carried" kit.
I find the 15-85 to be an excellent choice to stick on my 60D or 7D and use as a "lightweight, walk around and shoot nearly anything" lens. It has its compromises but then again all lenses do.