I've owned 2 copies of the 20mm over the years and it's probably my least liked Canon prime. It was just so soft and the colors so off, especially for its price.
The 15 is essentially a fisheye and so would get less use than your 16-35.
If you still use the 16-35 for filming, I'd just keep it. Another one to consider would be the F4 IS. It's a stop slower but much lighter and compact.
Personally, I have the 16-35 F4 and the 24L. I find they are both indispensable tools for different types of photos. I wouldn't use any other Canon wide prime that wasn't L, except maybe the 28 1.8. I have a soft spot for that one, it's highly underrated. I also loved the 20-35L for its IQ and its size (so small!). Another highly underrated lens, IMO.