This is a question I've been thinking about as well.
A few months ago I picked up a used 35mm f2 and have come to absolutely love this lens. Before getting it I'd read some reports of it having a noisy focus motor and questionable build quality. What I found when I bought it - yes, the motor is a little noisy but not something that bothers me one bit, focusing is superb (probably my most accurately focusing lens), the field of view (close to a classic 'standard' lens when used with a 1.6 crop camera) is great, pictures are super sharp and the relatively short focal length, combined with the f2, makes indoor low-light pictures a snap - and it's relatively cheap. The small size and light weight make it a joy to use when fitted to your camera. It has become my most used lens, and I'd recommend this any day over the 50mm lenses that are so often mentioned here.
Now I'm toying with the idea of getting the L version - but it's not a clear cut decision.
To get that extra stop, you have to pay five times the price. There are questions about the L's sharpness at f1.4. There is also the problem of increased size and weight. Sharpness from the L and non-L seem close in tests, as shown in the table posted above. Yes, you might also get better colour and contrast -- but these are two things than can be tweaked in photoshop.
On balance, I eventually will probably go for the L - mainly because I can see this focal length becoming my most used and also because I take a lot of low-light indoor portraits. But, if you tend to shoot mostly in reasonable light, it would be a close call. If both were stopped down to 2.8 or 4 I doubt if you would see much difference in actual prints taken with these two lenses.
I would have the same problem with the 85mm lens. There are very good alternatives avaliable -- the 35mm f2 and 85mm f1.8 -- that are really more sensible choices for most people. If you've got the money, sure, go for the L versions. If you really need that extra stop, go for the L (or increase your iso from 100 to 200).
But if, like me, you have a limited budget you will probably get 90 per cent of the quality for 20 per cent of the price by chosing the non-L lenses.
So, my decision. I'll probably go for the 35mm L because I do use that length a lot and can make use of it's low light abilities. Even then, I'll probably keep the f2 version for parties and social occassions because of its small size and weight, and the fact that it is a great preformer. I'll probably stick with the 85mm f1.8, however, because I use this length less and find the non-L perfectly acceptable.
Hope this helps..