I do exactly the same thing. I always take all four lenses on a trip, but every time I leave the hotel room, I select only two. During the day it's usually the 24-105 and either the 35L or 70-300L. At night it might be just the two primes. Sometimes it's just the 24-105 and camera attached to the tripod slung over my shoulder.
I do the same thing. ( unfortunately, I've done little travel as of late ) I do have many of the lenses in question ( 24L/50L/135L and I also have similar zooms 17-40, 24-105 and 70-300L.
Seems like your really debating 24/50 vs 35. I've actually considered going to the 35 just to simplify things as I find 35 to be very practical but I would not want 35 to be my widest lens and the 24 is excellent ( IMO better than the 35L ) The 50L has a look that I really like and I find 50 to be a great FL for me. I could easily live without the 24-105 but considering what it goes for used I haven't bothered to sell it and it does come in handy when I want to keep it really simply or for studio portraits.
Can you effectively use the 50L as a general purpose lens... Or is it more of a portrait lens given how soft it is off center?
I use mine as a general purpose lens and street lens. I love the photos I get with it but many people seem to have issues with it.