Hi,
It's a hard choice. I went through this right before COVID hit with one of my kids starting track. Been a prime guy mostly and have some big teles but they're too long and cumbersome to use with a crowd of people. So I had to decide what to do and I was shooting with 1D bodies for this. I carried a 50mm prime just for quick shots of team and close range stuff with DOF control to hide the mass crowd sometimes to put subject on the specific person/face. Worked well for that. Simple. Light, and cheap in case damage or drop occurs. I got a 70-200 F2.8 just for the track and indoor stuff. Worked very well. But, I quickly found it reach limited fast because 200mm is just not long for outdoor stuff. I found myself using my 300mm more and with a 1.4X TC too. Ultimately that 400mm reach gets you where you need to be, starting out, for sports. It's outdoor and there's plenty of light. Heck, I prefer overcast days. After all that, a few years of doing it, I've now left my 1D at home and take my smaller mirroless body and a tele zoom because at the end of the day, I want to watch the event and be part of my kid's experience and not just running around trying to get shots and not even seeing the matches or anything. And almost always there's just someone in the way making a photograph become a snapshot and I'd rather just not waste time on that. I've grown to love ultrawide and wide angle lenses with flash to be able to get group shots up close without random people walking into it easier and faster. Sigh.
The biggest thing is letting go of the feel of needing shallow DOF on every shot of your kids or groups. It's an easy button to hide unwanted environmental stuff, or take the emphasis off of it, but it doesn't make the shot better really. This is a hard thing to let go of. We all are there at some point. Instead, start looking at the environment as part of the image and compose for it to be part of it. The crowd and field are part of the image and experience, not something to hide. Get creative. It will force you to do better photography too.
I would lean towards the 100-400. 200mm on full frame is not much reach for sports at all unless you're way up close. And the 1.4x on a 200 is a bandaid, it doesn't add hardly anything. 70mm vs 100mm on the short end is a difference, but 200 and 400 on the long ends is literally double pixel difference, which is significant since you're on a full frame sensor. You need as long of a physical reach as you can. It's amazing how little magnification is happening when they're on that big field or far off and how short a 200mm is for this purpose. If it's outdoor sports/cheer, the 100-400 will be a great option. If you're indoor, that's different, then yea get the 70-200 F2.8 (or really if you're indoor honestly get a fast F1.8 or F2 tele prime like the 85 or your 135, as its really dark for a camera in most gyms and stuff).
Keep the 24-105, it's good when are not trying to hide the background. It will be excellent for trips, vacation, and group shots where you need DOF. Portrait isn't about "shallow DOF" every single time. You'll appreciate having a "do all" lens when you're swimming around with all the kids or on trips. Why hide the beautiful nature background for a portrait in a park that you can't identify from the blur? Right?
If you truly love the results of the 135L for portrait and you have the physical space outdoor to benefit from it, then keep it. If not, then sell it and replace it with an 85mm F1.8 to get a very similar effect but with less money tied up and a lot less distance needed. Plus it's a fantastic low light lens if you're ever indoors for cheer.
Unless you're completely married to that 35L, you could probably lose that one. You don't need it for landscape when you have a generalist zoom at F8~F11 anyways. For indoor tight grouping, you can use a F2 version and get some cash back towards something else. Flash is far more important indoor anyways, so I wouldn't stress having a 35L. Again unless this is something you just love, everyone has a preference. But you already have a wide end lens and flash, that's going to be better for group shots than natural indoor light (with variable temperature lights conflicting) at F1.4 and have someone soft and someone sharp.
Get more used to using your 580EX. Bounce flash and HSS TTL flash is a game changer. I'd value this over focal-ratio speed for most of what you described (except for sports on the field and landscape).
Or, keep everything and just add the 100-400. Done!
Or, heh heh, time to add the second body so you have a general zoom and a tele zoom for your events and trips and go akimbo. Camera dad.
If you wanted to squeeze you could swap out a few things.
Swap 35L to 35 F2 IS.
Swap RF 24-105 to STM 24-105.
Keep 135L (or trade out to fast 85mm).
Add 100-400 II or RF version or the 100-500 for outdoor sports.
Very best,
:: My 
