I don't think the issue is so much a matter of IQ loss, so long as you use a high quality filter and be aware that it might be an issue in some circumstances (if you have light sources in the picture for example, causing flare).
Most of the people who say they are necessary talk about damaging the front element, even wiping off the coatings (see one of the posts above). Believe me, the front elements are a lot tougher than these people think and the coatings are designed to withstand the lens being cleaned, believe me.
I have lenses that have seen many years abuse, they are dented and scuffed, they have had the front element wiped and cleaned hundreds (possibly thousands) of times, with an assortment of soft cloths and even towels and shirt sleeves, if that was all I had available at the time. Yet, the front elements are fine, they have no scratches, no loss of contrast or any other issues. They have rarely seen a filter and only when I have needed one for it's effect (CPL etc).
Somebody mentioned it is easier to change a filter than a front element, this is true. You will probably have to send the lens in for repair to change an element. However, they are also much, much harder to damage in the first place. One thing that IS quite good at damaging them is the flimsy thin piece of glass that people stick on there to 'protect' it. Filters break easily and can break in a manner which forces sharp shards of glass into your precious element, which can scratch it. I have known two people scratch elements by breaking a filter but have never known someone where a filter actually saved a lens from damage.
In the end though, it comes down to personal preference and how much you believe the guy in the store that tells you that you 'need' a filter (in reality, he just wants another sale). The reality is that anything coming in hard and fast enough to damage the element, will go clean through the filter and take out the element anyway.
Do the math and see what works for you. Filters are just too crazily expensive to be justifiable for me. I have quite a number of lenses and, if I was to put a decent filter on each one, it would cost me more than buying a complete new L lens (and some other stuff). In addition, the way my gear gets thrown around, I would likely have regular broken filter replacements to buy which cost even more money. I would rather buy another lens and have that to use instead.
The best protection for your lens is a good quality lens hood, they take the brunt of most impacts and keep the majority of foreign objects away from your element. In addition, they have the benefit of actually improving the image quality.
If the worst should happen and you break the front element, it doesn't cost a huge amount more than a good filter to replace anyway (although there is the inconvenience of sending the lens in to get it done).
You will get people advocating both sides passionately though, it is a common argument in the forums as to whether you need a filter or not. I simply offer up that in my experience (and I have been using SLRs for over 30 years, usually carrying two or three at a time banging into each other and everything else) I have so far not managed to damage an element in any way (not counting the lens that got trashed when it came loose from a fast moving car and bounced along a hundred yards of road - I don't believe a filter would have saved it though) despite never having used a 'protective' filter.
You need to make your own mind up though.