What kind of low light situations are YOU talking about?
If it involves stopping motion on the subject, then it's pretty simple. The faster the better. End of story.
So for indoor concert photography, where let's say my 200L f2 would be at 200mm f2 1/160 ISO3200, the shooter next to me with his 70-200 f2.8 would be at 200mm f2.8 1/80 ISO3200 or f2.8 1/160 ISO6400, and the shooter with the f4 version would be at 200mm f4 1/40 ISO3200 or f4 1/80 ISO6400.
Further, with an IS lens, you can get away with hand holding at 200mm 1/160 and still be able to reasonable stop moderate action and keep the images sharp. Not so simple without IS.
Your 40D doesn't have ISO6400 does it? Good luck with that f4 lens.

Thanks for all the great replies. I totally understand that you get a faster shutter speed with 2.8. It seems that a lot of you can get by with 2.8. It just seemed to me that when I'm using my lenses, 1 stop of light wouldn't be enough for what I want to capture in the low light situations I'm in.
I guess for what I shoot, f4 will work, or I need something f/2 or faster. I never seem to be on the fringe where f4 is barely not enough.
I guess the camera body plays into this as well. As a 5dmk11 with a 2.8 is going to be much better with its high iso capabilities over my 40d.