OP's question mentions wanting to get some L-glass. The 10-22 isn't an L. By Canon's own definitions, no EF-S lens will ever be an L-series. The optics of the 10-22 appear top quality. The build is probably more Canon mid-grade quality.
A fisheye is a very different animal, so OP may want to rethink pairing it up with a 17mm lens. There's a huge range you'll be missing in between the 15mm and the 17-40. It's not just the 2mm difference in focal length. Even on a crop sensor where it doesn't render its full 180 degree view but about 112 degrees instead, there is a dramatic difference from a 17mm (which gives about 77 degrees on 1.6X cameras). In fact, the angle of coverage of the fisheye is wider even than the 10-22 zoom, too (107 degrees on 1.6x cameras). On a crop sensor camera, a 17mm is only mildly wide angle. Hardly "breathtaking", IMHO.
I'd probably vote for the Tokina 12-24, if it were an option in the poll. Both it and the Canon 10-22 are surprisingly well corrected, considering just how wide they are. Especially compared to any fisheye!
Alternatively, the new Tamron 10-24 sounds like it will be interesting. We'll have to wait and see.
I have the older 17-35/2.8L, and will be trading it for something wider soon. It's just not wide enough on my crop sensor DSLRs.