Is this a religion? Will you be excommunicated for keeping a zoom?
I personally like a mix of primes and zooms. I keep the zooms around for those times when I need more than one focal length handy and when I don't need an especially fast aperture. I use primes when I need a fast aperture for the most part.
I'd probably just keep the 17-40L if I were you, and that seems like a good match to the 85 and 135 primes. Unless there is some very specific thing you are looking for on the wider end from some specific prime, I'm not sure where getting rid of the 17-40 is a great idea.
I personally own a 24L II which is a great lens for when I want a really fast wide angle (which is a unique look). But there are times when I think my 16-35L II is a better lens, and not even just for the ability to adjust the focal length. The zoom is actually also better at controlling flare in those situations as well.
So.....get what you want and all, but your thinking about a 35mm prime over the 17-40 zoom doesn't seem to be based on any clear rationale.