I don't believe that the 15-85 is of much better quality than the 18-55.
Do you own one, or have you used one extensively? I have both the 15-85 and the 18-55, and the two cannot be compared IMHO. Beyond that the build quality of the 15-85 is only a couple of steps behind some of the L lenses and it has one of the best IS of any lens Canon makes. I can hand hold it at 1/15 of a second or less and get sharp photos as a result. It's also tack sharp across the image wide open with great contrast. The 18-55 is a wonderful lens at its price point, but the two are not in the same league.
While I'm certainly not suggesting the OP should sell his 24-105 for it, the 15-85 is a very good lens. I chose it for the everyday lens for my camera precisely because the 24-105 did not go very wide. If it was designed as an 18-105 instead, I'd be using one right now.