I've had or lusted over many of these lenses and vacillate between being a prime/zoom guy. Here are my two cents.
Keep a couple primes and a couple zooms. Pick your focal ranges and keep lenses that complement them. I'd say a WA zoom and tele zoom, with a couple primes in the sweet spot of those ranges. You have the 17-40 and 70-200 f/4, but you don't seem to use the 70-200 for some reason. These two ranges make a great "shoot anything that comes my way" or travel combo. If you don't use the 70-200 you probably won't like the 2.8 version, since it's a heavy like a beast.
You like 35mm perspective. Do you like 85mm as well? Because they go really well together. I'd skip the 50mm, it's too close to either of those and I've had the 50L. I think the Sigma is 90% of the lens for 25% of the price. The magic of the 50L are the bokeh half-length portraits at f/1.2 to f/2.8, and honestly those get boring. Otherwise it is a competent, overpriced, and good looking piece of glass. I'd mention the 40mm pancake, but 35 is way too close for that. I had a 24/50/135 combo, so when i sold the 50 i wanted to fill the gap without breaking the wallet.
But now you have 85 and 135 primes (and a 70-200 that covers both of them). To me 85 and 135 are closer than 50 is to 35/85. Personally id sell both of them and put more money into an 85L or maybe even a 100L macro. I'd give up on the 50L, it's magical but not 85L magical. And if you play around with the 2.8 70-200 and think "this is a lens i might bring with me and use a lot" consider that as well. Personally i hate the thing as a physical object. It's white, loud, heavy and inconvenient. But damn it is extremely versatile and it gets the job done.
my current gear setup: 17mm TSE, 24mm 1.4, 28-70 2.8, 40mm 2.8, 70-200 2.8, 300 2.8. My gap is a wide angle zoom and either a 100mm macro or the 85L