I own 24L and 50L. Recently added 24-70 L II for the same reasons - the convenience of not having to change lenses. I also shoot only for fun, focusing mostly on my family, kids, travel. I took the 24-70 II with me on a family vacation as a one lens solution, I got some great photos, really can't complain about the quality. With that said, I'm going to depart with it for several reasons:
1) I missed having large aperture for low light videos. this is where the 24L really shines.
2) big reason why I love photography is the process of making photographs, not just the outcome (photos); I didn't enjoy using the 24-70; don't know why, perhaps because it was too convenient to use
- missed the challenge of using just one focal length
I will continue using my prime lenses for times when I want to enjoy photography or deal with challenging situation. I may add smaller zoom camera (m4/3, NEX, etc.) for the times when I need something smaller, portable and convenient.
My recommendation: considering you can't afford to keep what you have and buy the 24-70, wait until Canon offers rebates (probably around thanksgiving), BH had it for 1799 after rebate few months ago, buy it with BillMeLater 0% 6 months financing, use it for 6 months along with your other lenses, then sell what you don't want to keep and pay off the bill. If you time it right and buy the lens with rebates, you should be able to sell it used later with only a small loss.