If you plan to do a lot of tripod work with it, keep in mind that the f4 lenses don't include the tripod mounting ring... it's sold separately. (It's included on the f2.8 lenses.)
There is some image quality difference, but you really have to look for it. The two most recent models of 70-200 are the f4 IS and the f2.8 IS Mark II... and of all the Canon 70-200s they are the only two using fluorite elements for better control over chromatic aberration. Makes for a "crisper" looking image, particularly at the largest apertures. The IS on both these are also the later, 3 to 4 stop variety, while the earlier f2.8 IS has 2 to 3 stop rated IS.
Besides size, weight, price, and the extra f-stop, the biggest difference between the f2.8 IS Mark II and the f4 IS is focus speed in certain situations. In low light/low contrast, the f4 lens will struggle a little more... but under good conditions you won't notice any difference. Both use fast, accurate, reliable USM AF drive.
Personally I use the 70-200/4 IS and the 70-200/2.8 IS "Mark I" (can't justify the chunk of money to upgrade it).