I used to own the 70-300 non-L, but sold it and got the 70-200 f4 IS instead, and have since swapped that for the 70-300L. Honestly, I'd say it depends on the type of photography you intend to do. "Vacation" covers quite a range of possibilities. For us, "vacation" means the beach, and for that, I personally would prefer the faster focusing of the 70-200. For touring and visiting, it's probably less clear-cut. What you will be losing in the exchange is some sharpness in the upper part of the range (the 70-300 I had was not particularly sharp above 250 mm), some colour and contrast rendition, the internal zoom (the 70-300 extends when zooming), and above all, focusing speed; the 70-300 is a slow-focusing lens (it's not "true" USM). That may not matter if you're photographing stationary subjects, but it should be a consideration if you think you'll be photographing fast-moving things (e.g. birds on the beach).
If you haven't tried the 70-300, I'd suggest you do so before making the switch. Having owned the lens, I personally wouldn't do what you're suggesting. But the addition of IS may be worth it to you; as I said, it all depends on how you'll be using it.
Good luck with your decision!