No, it's not a bad idea by any means.
However, I would say that you'll see little difference in your images comparing the 28-135 you have now and the 24-105 you're considering. Focus speed and close focusing ability of the two lenses aren't all that different, either. The main difference will be the build quality, sealing and durability that the L lens offers. The 24-105 is top L quality build, while the 28-135 is more mid-grade. Considering that a 28-135 does the job nearly as well and I can buy one off Craigslist for $200-250, I would probably only do that particular "upgrade" if I were getting the 5DII in kit with the 24-105, which is often a pretty good deal. And even then I'd be sort of tempted to sell the 24-105 to recoup some of the total cost just to be able to keep the 10-22 to use on the 50D or to help fund a 24-70/2.8.
The 17-40 also might be fine. You're right about the 10-22 not being very usable on the FF camera (there are modifications to allow it to mount, but it still vignettes pretty strongly... I use a Tokina 12-24/4 instead, which does happen work on FF as wide as 18 or 19mm.) Personally I like and use a Canon 20/2.8 as my widest full frame lens. But, you may need something wider and the flexibility of a zoom. Lanscape shooting isn't a high priority for me.
Not saying your plan is bad in any way... just that there are some alternatives you might want to consider.
For me, getting back to full frame for the first time in a long time a few years ago, the most important lens and one I bought immediately was the 135/2L. But, it sounds as if I do more portrait shooting than you. I do a lot of sports as well, but for me that's not a consideration because I don't use 5DII much for that. It's just not an AI Servo/sports/action camera... It's a great One Shot/portrait, landscape, macro camera.