Having have owned the 300 f/4L IS prior and buying the f/2.8 II recently, I would have to say the f/2.8 II is somewhat in a league all its own. The sharpness wide open is probably better than the f/4 IS will be, even stopped down. The build quality, IS, separation ability, etc...
That being said, I don't think its crazily different when you factor in the price/performance ratio. If you can get by with f/4, that model is excellent. The MFD makes it a cool macro-like lens, its lightweight and optically very good. I never had a problem with it! It does show some purple fringing in high contrast areas, I do remember that.
So while the f/2.8 is quite a bit better, not sure if its worth it for everyone. It just depends. I like the option to add the 1.4x and still have an f/4 at 420mm, or a 2x III and get 600mm at f/5.6 with still really good IQ. The f/4 is good with a 1.4x, but not so much with a 2X, plus your a stop slower.
I think it was worth it for me... But I really enjoy shooting with telephotos for most things, not just birds, wildlife, etc... I do much of my amusement park photography with 200mm+ and the f/2.8 helps with separation and low light.
The f/4 IS is an excellent lens though... For the price, its a great option. I sold mine when I got the 70-200 II because the 70-200 + 1.4x was basically sharper at 280mm than the 300 prime was (or close enough). So there was no reason to keep it.