I have both and the 200mm f/2 is somewhat sharper across the frame at f/2 than the f/2.8 is at f/2.8.
However, the bigger difference is the color rendition and contrast of the prime is significantly better than the zoom. This is not surprising, since the prime is 3x the price and optimized for one focal length and one aperture setting -- no one shoots the prime at f/2.8. SOOC, the raw files from the prime "pop" more than the ones from the zoom. I also suspect that once Canon releases a high megapixel FF camera (40+ megapixels), then we can see the resolution differences between the prime and the zoom more clearly.
That said, the zoom is often more convenient for me personally (events, constrained spaces, etc...), but if I have enough space and time, I'll take the prime every time.