I have had my 24-70 II for about a week now and I have owned my 35L for several months. Prior to buying the 24-70 II, I have found myself in a dilemma where I debated whether or not I even needed the zoom.
Short Review : 24-70 II is an extremely sharp lens, but 35L is just as sharp at 2.8. It is soft at 1.4, but comparing 35L wide open to the zoom wide open is not fair. At short focal lengths, 1.4 aperture makes a massive difference when it comes to subject isolation and after using 35L, the zoom is not satisfying at 2.8 for subject isolation. That being said, while you could take a step forward and backward with the prime, there is the perspective problem. With zoom, you get proper perspective all the way to 70mm. What you can accomplish at 24mm with the zoom is also something you will never be able to with the 35L unless you can fly through the walls. So without going into further details and scientific experimentation, my opinion is that 35L produces far more "magical" shots that most amateurs may be looking for. The zoom is good for wedding photographers who don't want to miss a shot, but the shots will be rarely as magical as 35L's. It produces more standard looking shots unless you are way close and at 70mm shooting for head portraits. At 35mm however, 2.8 vs 1.4 makes a big difference.
If you are shooting indoors, I'd say 35L no question. Zoom's focal range is hardly useful in a small apartment or a medium size house. I can use my 35L for 99% of the shots I deal with. If you are outdoors however, 24-70 II has the upper hand. 24mm makes a big difference, so does the 70mm for body portraits.
Afterall, I'd say if you are a professional, get the 24-70 II, otherwise, you have to be looking to burn cash to get the zoom over prime. I'd say 70-200 is a far more useful zoom than 24-70 will ever be for me.
Thanks for reading.