Aye very true, though I have to say that thus far you're the only person I've noticed to get this level of performance with the combo. I think it a very bad case of bad luck resulting from the camera + TC + lens having some kind of calibration tolerance issue (which with photography gear is always a risk with any purchase - though in general most gear is ok - the 100-400mm being one of the oft quoted riskier examples)
Edit - now two people
I say it's a problem with 2x TC's in general, rather than some lens-specific issue. When I stick a 2x II on my 300 2.8 IS, image quality pretty much falls off a cliff (in relative terms; I pixel peep)--it's still quite spectacular with a 1.4x II. The effect is more pronounced (rather, easier to detect) on my 70-200 II
I've also got Kenko MC4 DGX 2x and a Kenko Pro300 DGX 2x, and I see about the same performance--so I can't claim that my Canon 2x II is just an aberration/dud adapter.
I avoid using my 2x TCs whenever possible--moreso because of the poor AF than the loss of IQ. Also, you can't track BIF properly with a 70-200 II w/2x TC That killed the combination for me.