I have the 400 prime, but then again I pair it up with my 70-300L when I use it, primarily at racetracks. Sure, this is going to require me to change lenses, but since I use my 70-300L for other things as well (zoo, family stuff) it's a price I'm willing to pay. Not to mention the total weight of those two lenses in my lightweight hiking backpack feel like nothing to carry around.
Also- consider if you want to use a 1.4TC on your 400mm lens. I tried a non-reporting kenko 1.4TC on my old 100-400L, and it didn't work well at all, or give too good of results with my 7D. The 400 prime on the other hand takes the TC much better than it was designed to with my 7D, and the AF and IQ are both manageable as long as you're outdoors.
Just some things to consider beyond simply the IQ between the two, as both are very capable lenses.