I've had the pleasure of owning all of these at one point or another, and currently own the f/4 IS and the f/2.8 IS II. I'd put them in this order:
70-200 2.8 IS II
70-200 4.0 IS
70-200 2.8 NonIS
70-200 2.8 IS
I think the bottom 3 are really splitting hairs. The f/2.8L IS II and the f/4 IS are clearly the top 2. From my experience I'd say the old f/2.8 IS is probably the worst wide open.
You can see in these comparison crops that the regular f/4 does well against the old f/2.8 IS.
I'm going to agree with this assessment. I've used all the different versions of the 70-200 except for the F4 non IS.
The 2.8 IS is probably the most overrated in terms of sharpness wide open. I've tried 4 copies of this lens and the version 2 of this lens is leaps and bounds better in every single way except for the slight increase in weight and the massive increase in price.
The F4 IS is probably the most reasonable in terms of weight, price, and quality. Its a great lens.