Get f2.8 if you need to stop action in poorly lit circumstances (ie if you need the shutter speed). The IS on the f4 can more than compensate for the stops of light you lose compared to the f2.8, but it will NOT let you use a faster shutter speed.
The f2.8 also gives you a brighter viewfinder, better autofocus in low light (lets more light in, although how significant this is, I don't know), is better at F4 than the 24-105 is at F4, but 2.8 is not 'much faster' than 4; it's one full top faster, while IS gives you 3-4 stops (supposedly). There's also the depth of field issue - 2.8 gives you a shallower DoF and blurrier bokeh, although I find the F4 does pretty decently.
The f2.8 is significantly heavier and a bit bigger. The F4 has a better overall range. Far as I can tell from various reviews and samples (averaged), IQ is roughly the same - one says the F4 is better, the other claims its the F2.8, so I figured 'split the difference'
Ultimately I went with 'lighter and broader range' over 'more light', because a) it was pretty cheap with the 5DII kit (400 dollars is not to be scoffed at) b) weight does matter for an all-rounder and c) if I really need fast glass, I've got 1.x primes, which are my preferred 'people shooting' glass. I shoot a fair amount of low-light stuff, but most of that is pretty static - landscapes, cityscapes, nature. Now, if they came out with a 24-70/2.8 IS zoom, I might be very, very, very sorely tempted to trade up.