Based on the OP comments, I would keep the 24-105 as it is a great primary lens. I really like the overlap of the 70-105 range since this is important if you do any portraits (I like the 24-105 for stopped down studio work as well). I considered the same lineup as the OP, but I ended up going with the 17-40 and 35 L instead of a 16-35.
If you don't want to get into the prime thing, I think you have a nice consolidated setup as is. You have a fast UWA that can be used for low light at f2.8 from 24 to 35mm if f4 is too slow on the 24-105. You have a walk around general purpose lens with the 24-105, and a fast sports/portrait lens with the 70-200 f2.8 IS.