The EF 300/4 IS works very well for sports. It also works very well with a 1.4X (I use the Canon Mk II).... There's almost no hit to image quality and no perceptible difference in AF speed/performance with the 1.4X. It's also quite hand-holdable. Not much larger or heavier than a 70-200/2.8.
300/2.8 and 120-300/2.8 are wonderful, but you'll want to use a tripod - or at least a monopod - most of the time.
Incidentally the EF 300/4 IS is the closest focusing of all the Canon lenses longer than 200mm. It can be useful as a near macro lens, too. Gets to about 1/4 life size on it's own, while most of the others don't give more than 1/5.
And, the IS comes is helpful. I leave it ON practically all the time on this and the other 5 or 6 IS lenses that I've been using for close to 15 years now. It does not slow down AF... IMO if anything IS helps AF work faster and track better. That would just make sense, that the camera would have an easier time locking onto and maintaining lock on a stabilized image. (Note: There is some evidence that Nikon VR slightly slows AF on Nikkor lenses, I don't know why. But I don't see that happening with Canon IS and that's what we're discussing here). The 300/4 is one of the few lenses where IS needs to be turned off when the lens is solidly locked down on a tripod, no chance of movement at all, otherwise the IS can go into sort of a feedback loop where the IS actually causes movement when there's none to correct. But even if using it on a monopod or a gimbal mount, I leave IS on. And in almost all instances, Canon IS lenses are sharper than earlier or concurrent non-IS versions that might exist. This includes the 300/4.
Also, 50D is quite capable of sports shooting. I agree with the suggestion that a 1.6X crop camera can be an inexpensive way to "get more reach". Anyone who suggests otherwise simply hasn't used a 50D very much. Sure, a 1D series camera or a 7D is more capable and ideal for action shooting, with their more sophisticated AF systems and discrete chip to drive those systems. But, the 50D certainly would be better at sports shooting than a 6D. That's for certain.