The 17-85s biggest problem is the wide end, it has a lot of CA at 17mm and the edges are not very sharp even when stopped down. The other big issue everyone mentions is the distortion, and while it is pretty bad it's virtually the same as a 24-105L on a FF body, and you don't hear many people complaining about that lens body combo often, do you?
The CA and distortion can be fixed very easily in post with good software, but the edge softness you can't do much about. Once zoomed to 24mm the sharpness and distortion are much improved, virtually the same as for the rest of the focal range. So basically the 17-85 will only handicap you if you do a lot of wide shooting at 17mm.
The best replacement for the 17-85 is the new 15-85, which is better in virtually every way. Better build (but still not L), 15mm is much wider then 17mm and the image quality is much better on the wide side. The long side is pretty much the same as the 17-85, and so is the f range. Overall it's what the 17-85 should have been, but honestly the price difference is so large that I personally can't justify the upgrade expense just for the improved wide end, and will happily continue to use my 17-85 and buy an extra lens instead, like the Sigma 30mm which is a great addition to a crop body.
So ask yourself, what range on the 17-85 do you use the most, and is it worth upgrading to the 24-105 for the improved build and extra stop? The image quality will be better on the L but not by such a large margin, especially at the long end.