I shoot a 24mm f/2.8, and I also use the 50mm f/1.8 II and 100mm f/2.8 Macro. I like the 24mm f/2.8. It's sharp enough and it's really quite wide on a full frame camera--wide enough to make 'interesting' shots, while not being so wide that I can't use it for more general photography.
It's not as sharp, contrasty or as good in the colour department as the 24mm f/1.4L II, but really for the price difference it's not too bad. Given a choice I'd certainly upgrade to the L lens. It's not in my budget.
One of the best things about the 24mm f/2.8 is its size. I can easily pocket it and take it with me anywhere. If I'm bringing my camera, I always have the 24mm and 50mm f/1.8 II along minimum. I hardly notice that I'm carrying either lens.
I think your biggest consideration here is focal length. The 24mm is really quite wide on a 5D. You may prefer a 35mm. You may prefer a 28mm. I think you should buy the best lens that you feel you can afford in the focal length that you like best. I selected 24mm because I like to be able to take very wide feeling shots if I get close to something, but I also want the lens to not be completely distorted too. The 24mm filled the bill for me.