Heya,
I have both the 35 F2 IS and the 40 F2.8.
The 35 F2 IS is better in every way, except size, it's sharpness is another level. It's a larger lens obviously, and with a lens hood, takes up space and no longer is as discrete. The 40 is small. No hood needed (the hood you can get is not even a hood really, what is that thing?). So discreet you look like anyone else with some P&S walking around and you get ignored.
So it comes down to what you want most.
Personally for the image quality and aperture and 4 stop IS, I really enjoy my 35 F2 IS as my all around, walk around, travel, you name it lens. Something about slightly wide, 35, on full frame, with F2 is just appealing. Wide enough for some general landscape. Great for portrait, great for group, street, etc.
I'm not as put off by size, because even though the 40 is smaller, the camera is not, so you're still walking with a bag of some kind, and a gripped 6D (or any dSLR) is big enough that frankly a small lens doesn't make a big difference. I have two bodies currently and the 40 lives on one of them, it's a funny look (big gripped body with a 40 attached). My other body usually has the 35 F2 IS, or a telephoto attached.
Very best,