It doesn't matter if it makes sense to us, it matters if it makes sense to you!
I guess the big question is do you feel like you'd like a wider lens when you have the 24-105 on your camera? Or is 24 usually wide enough? I guess I'm trying to ask if you are certain you want to replace the 24-105 with the 17-55. If so, then by all means make the switch. It does not do you any good to keep a lens if it will end up collecting dust.
I just got the Sigma 17-50 f2.8 OS HSM, which is primarily my indoor lens. I also recently got a refurbished 24-105 from Adorama that is primarily my outdoor lens and travel lens. I, too, and looking to get something longer than 105, but I'm not sure what I want yet.