I like to break down travel photography into three scenarios.
Walk around daytime:
On a crop your 17-85 is fantastic, the 15-85 is a bit better optically but either will do a great job.
Indoor low light ( museums, churches...):
I like something on the wider side and optically fast. On FF the 35/1.4 is my first choice on a crop I like 24. A wide 10-22, for crop can be handy in these situations with a body that has good high ISO since it is on the slow side.
Night walk around:
For this I usually like a fast prime but it really depends on the location and the sort of things I expect to be shooting. Since I focus on the two previous categories I generally just mount what ever I bring along.
One thing to consider is weight. You can very quickly get sick of heavy kit, L glass is great but not only costly but heavy. Do not over look the mid-range lens over the L glass due to weight. While the 17-55/2.8 if a step above the 15-85 optically it is a heavy beast. The same holds for primes, the 35/1.4 & 24/1.4 are great but quite heavy/bulky compared to their non L relatives.