I have both the 24l and 35l and 5D.
I much prefer the 24mm focal length for wide angle work, so use this lens more often.
From what you write though I would reccomend the 35L to you. 24mm is very wide on FF and you have to be quite careful about distortion.
35mm I find to be a more general purpose FL- not too wide, can get in closer with less distortion etc, can shoot people with it more easily, yet is a great lens for all but the tightest wide angle situations for landscape/urban scape.
The 35mm is much more forgiving. Its also probably the slightly better lens optically than the 24L mk1- may be a tad sharper, maybe slightly better bokeh, colours maybe slightly more rich.
But these are mere quibbles- both lenses are outstanding optically- my 24L smokes the 17-40 I have for contrast and sharpness, and was sharper than my very sharp 24-105L.
But I use two bodies :- with a 24 on one and a 50 on the other. If I only had one camera and had to choose one prime only I would choose the 35L- its so versatile. But conversly, having two bodies with a 24 and a 50 combo, I hardly ever use the 35L. Go figure
Recently I was using the 17-40 a lot for convieniance.
I went out with my 24L at the weekend and the shots were so much more interesting because I had the thin DOF to play with.
You wont regret getting a 1.4, whichever FL 
Theres no going back.............