My 24-70 is my professional workhorse, and I can't afford to be without it ... personally, I'd love a 35L but can't justify sacrificing my professional workhorse for my personal fancies ... but if you can afford to be without your 24-70 (and it looks like you have the wide end covered by the 16-35, and the long end covered and furthered by the 70-200) then I see no reason not to get a prime of the focal length that suits you best for the extra creativity, and opportunity that those lower apertures can bring. Play around with the lenses you have to mimic the primes you're interested in and see what might suit you ...
On paper, 16-35, 50, and 70-200 looks great, keeping gaps pretty minimal, but if you don't really shoot at 50mm, but are particularly fond of either 35 or 85, then having an expensive 50mm prime makes less sense than having a great lens for your preferred focal length ...
In short, I see no reason not to, but having no reason not to, isn't really a reason to - so find the lens you really want ...