As others have said already, it comes down to focal length!
I owned the 35L for some time, and it is a superb lens, probably the best prime Canon makes (no weather sealing is it's only drawback, IMO). It's sharp, has nice AF speed, beautiful color rendition; images from this lens sing! But, 35mm FOV didn't work with my vision, so I hopped up to 50mm, which is my "sweet spot" focal length. Currently, I shoot the 50L, and while it has it's quirks (focus shift, though it's easy to work around), it is simply awesome. It has weather sealing, fairly quick AF speed, gorgeous bokeh, practically flare-proof, and has an uncanny ability to lock focus in near darkness where other lenses will hunt and hunt.
**Note that the focus shift may interfere with your style of photography (if you shoot stopped down, in the "danger zone" MFD to ~6-10' range), so it'd be prudent to maybe rent the lens for a bit and see if you'll be hampered by it.
AlanU is bitter about the 50L, as he insists on demonstrating in every single 50L-related thread. No one will disagree with him regarding the known focus shift of the 50L; what most 50L OWNERS will disagree with Alan on is just how little it will affect your photography once you get to know it.
Alan, while you hunt for your mythical unicorn 50L, those of us shooting with the lens will enjoy the fantastic results it delivers.