Let me start this by stating that I just do photography for fun. I do enter some contests, but I don't do any paid work. I spent a fair amount of money on some L-lenses. In my opinion, they are not worth the price, UNLESS, and this is huge, they are the only option for doing what you want to do.
I had a 135L and 200 2.8L and let both of them go for a Sigma 70-200 zoom. I am happy with the lens, the photos I get from it, and the $800 I put back in my pocket. My mid-range zooms are the Tamron 28-75 and the Canon 15-85. I could cover that range, more or less with the 24-70 II, which costs more than twice what I paid for them both.
I kept my 100-400L because there are few other options that don't cost as much. There is the Sigma 120-400, but I went with the Canon. I tried non-L primes for awhile. I don't do anything that requires an f/1.n aperture. I don't care for the razor thin DOF. Therefore, I have little need for L-primes. Heck, the only prime I have left is the EFS 60mm f/2.8, for obvious reasons.
As mentioned, if you need something only an L-lens can do, and you can afford it, buy it. I really want an EF 24mm f/1.4L II. There are NO alternatives. I have bought the 24 2.8 IS twice, and returned it twice. It just isn't fast enough. In this case I need an "L", so I will have to pay for it. Later.
With all of that out of the way, if I could afford it my kit would be: 5DIII, 24L II, 16-35L II, 24-70L II, 70-200L II, 500 f/4L II. I think that would be most shooter's kit, if money weren't an object. Short of my dog pooping a sack-full of Bennies (that meant something entirely different 40 years ago) I don't see it happening.