The biggest things to consider are: What are you looking to shoot? The needs for football are way different than for basketball and the needs for soccer are different from baseball. What is your overall goal for your photography? Are you just wanting to take quality photos of your kids/nephews/family members? Or are you building a portfolio to try and pick up stringer work for local media outlets? There really is no reason to drop $1500 or a lot more if you are just doing some family lens jockey work.
However, if you are looking to really dig in and make a big commitment of time and money then my suggestion is keep the 70-200 f4 and go for the 300mm 2.8 and a 1.4 tc. The guys that I know that have the 100-400 use it only as a situational lens not a primary lens. The primary lenses they (myself included) are using for sports is a 300 or 400 2.8 on a monopod and a 70-200 on the shoulder. I will also have 24-105 and 17-40 in the think tank speed racer waist bag. That is the kit that I use for Football, Soccer, Baseball, Softball, and track/field. Basketball, swimming and wrestling I will use a 24-105 and 70-200.
Equipment list: Canon 1Dx, Canon 1D mk III, 50D, 400mm 2.8L, 70-200mm 2.8L, 17-40 4.0L, 24-105 4.0L IS, Sigma 15mm 2.8 FE, (2) 580EX II