I have owned the 24-105 for many years now, and recently I rented the 24-70 2.8L II for a week when I went on a trip. I left the 24-105 behind. My impressions were thus.
The 24-70 is incredibly sharp indeed. AF was quick on my 5D3 and corner to corner it was pretty much completely awesome. Sunstars were nice and pointy. I can't complain about the image quality at all. It's top notch.
I missed the extra 35mm on the long end quite a bit. If I wasn't so used to having it maybe I wouldn't, but that reach to 105mm is incredibly useful in a walkabout lens. I did a lot of indoor people shooting during this week, and to compensate for lack of IS I ended up cranking the ISO which is not something I normally do.
A week or so after I returned the rental, I was pondering the switch. Then there was a night-time arts festival and I took my camera and the 24-105 to it. This was really challenging to shoot and incredibly dark. I had the IS on and was able to get sharp images down in the 1/8th second range. Such a shot would have been utterly impossible with the 24-70 2.8 unless using a tripod, which wasn't a very good practical option for this event.
Before I sent it back, I did a series of test shots on a tripod with my 16-35 2.8L II, the 24-105, and the 24-70 II shooting the same scene, in the same light, at f/8 and f/11 at 24mm. First place was the 24-70 predictably. Second place was the 24-105, and it was a CLOSE second. Sharpness was almost equal at those apertures. Distant third was the 16-35. Since I do landscapes mostly, this complicates things with my thoughts on trading the lenses. I was expecting a huge sharpness difference even at f/8 and f/11 but didn't find it (although I found a big difference with the 16-35)
The conclusion I came to was that I will keep the 24-105. Someday I will add the 24-70 II to my kit, but the 24-105 is just too useful to let go of. B&H selling the 24-70 II for $1699 after rebate makes it really, really tempting. If I had both the 24-70 would be on my camera for most landscape stuff. 24-105 would remain the walkabout choice for most of the time.