I'll play... only read below if you're bored
I have the same lenses that appear in OP's sig plus the 135L in question. I also rented 35L this past summer and when the time was up, it felt like giving away my favorite puppy or kitten. I'd assume 35IS's image quality should be up there on par with L lenses.
I've built my lens lineup so that the zooms are at f/4 (with the exception of 16-35, because I need that extra f stop for handheld interior shots), complemented by fast primes at key focal lengths.
Since you have the 100L in the tele range already, how about one of the 35mm choices to complement the 24-105. In fact, if I had to own just two lenses, it would probably be the 24-105 and a 35.
135L might be too long for candid shots, but that's just me. It really depends on your shooting style -- do you shoot outside of the action looking in or do you immerse yourself in with the action? I tend to be in the latter most of the time and hence one of the 35mm prime options (35L, 35IS or Sigma 35) is high on my next lens purchase.
100L and 135L are very different lenses but in a pinch, 100L can cover 135L's ground. 135L focuses faster, but if you apply the same trick 85L II shooters use, which is to manually focus close to the anticipated distance, the 100L will focus right away. Using distance limiter selection switch on the lens also helps. 100L's bokeh is not as creamy as 135L but the lens is plenty sharp enough.