If you were mainly using your 15-85 for street, yes, the X100 could be a worthy replacement. I added a X100S to my kit, and absolutely love it (and no I don't planning on getting rid of my SLR). But, the X100 doesn't do landscape as well as an SLR, and of course, you'll lose the versatility of the 15-85 range. But, I'm finding I'm loving the 35mm FL... just learning I need to get a little closer than I used to.
I agree, IMO keep the 15-85 and your 60D for times when you need the reach and wideness that it can provide.
but also buy a smaller portable camera that you can bring with you all the time.
main reason why I invested on a X100 and now the X100s.
for people telling you to skip the X100, don't listen to them as IMO it's still a perfect usable camera with a decent AF allowing me to capture some people walking or running, and the X100 goes for half the X100s.
I used mine 95% on my trip in Orlando/Miami and it was such a pleasure to carry around compared to DSLR.
funny enough, I never liked 35mm on FF DSLR, however I love the X100/s and before I used to shoot Rebel + 24 2.8