Heya,
I've gone backwards on glass & body, and I have no regrets. I have a mix of old and new stuff. L quality is great, and full frame is great, but sometimes, it's just overkill depending on needs. If I'm on vacation, I don't care to drag out the biggest camera I have and the biggest heaviest lenses. I'd rather enjoy the trip and use something else for photos that is easier to lug around and less costly if I drop it or lose it or anything. So even though I have too many cameras now (1D2, 5Dc, 40D, 450D, 650D, EOS-M) I actually use them all for different things. And for glass, I have things ranging from 30+ year old vintage primes that I use more than anything else on my day-to-day shooting, to L glass (primes in my case). I have gone different directions as I've found L glass sometimes to be worth it (like my MKI 200 F2.8L prime) and irreplaceable for cost (nothing cheaper does this, I got it for $250 because it's a MKI but it's still fine glass and fast AF). Instead of 35L or 35A, I use an EF 35 F2 IS. I find it to be good enough for a 4th of the cost for my needs. I do a lot of context portrait and landscape with it, and events, so I don't need it to be F1.4, and that's the only advantage (for me) that they have (the L's optics are not legendary, the ART however does have awesome optics, but it's more than my needs require so I don't need a $1k lens for slightly better optics). As for other glass, I went to manual glass that is optically very good, instead of using expensive L primes with AF, or third party alternatives. Namely stuff by Samyang, like the 85 F1.4 and the 24 F1.4. They are either comparable, or better (the 24 F1.4 in this case) than their costly alternatives. I don't need the AF for portrait or landscape, so manual focus is fine for me. The 85 F1.4 by Samyang performs between the 85 F1.8 and the 85L, about on par to my eyes as the Sigma 85 F1.4, but for only $250. So that lens is optically great, very sharp wide open, and inexpensive. The Samyang 24 F1.4 is better than the 24L in most ways (sharpness, corner sharpness, coma, etc), other than it lacks AF. But again, I'm ok with manual, so $500 instead of $1200+ works for me. Anyhow, you get the idea. I actually prefer some older glass to modern glass for general stuff because they're sharp and look great, without the cost. Again, mostly manual, but I'm ok with that, as I'm proficient with manual focus anyways (using precision matte screens). I shoot more with an old Helios 44-2 and Takumar 50 F1.4, and I have no care in the world for a 50L, 50A, etc. These do that for me, plus have either unique bokeh, or cost nothing and still produce very good images, so I'm less concerned with the optical performance being at it's peak.
For bodies, instead of going for a 7D, 7D2, 5D3, etc, I instead started using an old 1D MKII. It's only 8MP for it's files. But I'm not printing massive prints, and I use it for wildlife and action. So I find I don't need high resolution, these can be printed around 8x10 just fine anyways, so I'm more than ok with that. Instead, I paid peanuts ($289) for an autofocus monster with decent ISO performance. It's just old and lower resolution, but again, depending on the final media you're using these files for, it may not matter. My wildlife/action is mostly web share, no big prints, so I don't care about the resolution being huge. This is just an example of stepping back and looking at actual need of features. I needed AF muscle and speed, and the 1D series gives me that for cheap, on the older models.
As for my travel stuff, I don't bother with SLR at all at this point. I used to. But now, I'm more content with my EOS-M & 22F2 pancake. Small. Sharp. Decent in low light. Big sensor for size of setup (APS-C) and high resolution. I take this with me when I'm "out" and on vacation, because it's just simpler, less worry if I bust it, or lose it, or stolen, yet still takes SLR quality type images, and I still have F2 for isolation if I want that.
Again it's all about features.
The 50L, if it were not F1.2, would it be special to you at all? Or would any 50 F1.4 basically do what it does?
The 135L has it's own magical look. But is it really vastly different from the 100 F2 or an 85 F1.4 for YOUR needs? Maybe not. That's up to you to figure out.
As for the 24L, there's not a lot of alternative, if you want fast 24mm. 24mm F2.8 is fine. But I wouldn't want to lose F1.4 as an option if possible, but this is why I like manual (Samyang 24 F1.4). But if you love that lens, keep it. Especially if you actually use it.
Finally, if you're not using it, who cares about the regret? Most lenses have value that can be sold for near no loss and re-acquired later if you have a change of heart/need, like the 135L. That lens is traded/sold/bought more than anything else I've seen probably, short of a 70-200 of any flavor.
Lastly, try out alternative lenses. See for yourself. See if you can live with them instead of the L's. If you can, you know you're answer. EF 28 F1.8, EF 35 F2 IS, Sigma 50 F1.4 (non-ART), EF 100 F2 or Sigma 85 F1.4. Assuming you require AF at all times. If you threw an EG-S screen into your 6D, and were proficient with manual focus, you could get really great glass for cheap too (24 F1.4, 50 F1.4, 85 F1.4, 135 F2.. fully manual).
Personally I'd take more trips and see more stuff, than worry about glass that isn't used much on a shelf!
Very best,