If you can't control your backround, then get the 85/1.8... anyone who says it is too long for indoor "portraits" probably doesn't have one or doesn't shoot indoor portraits. The 50mm, imho, is too short. You can almost shake the subjects hand if shooting a head shot. Being that close is too intimidating even if it's your own kids and is not as flattering as an 85 or a 100/135 (if you have the room). The result is, basically, shots that just don't look "good", imo. I am limiting my opinions to portraits, btw.
If you can control the backround, have additional lighting then you already have what you need in the 24-105. I use my Tamron 28-75 at or close to 75 for indoor controlled situations 95% of the time and almost never more wide open than f5.6. The 85 comes in when I want those extra shots, when I'm done, with shallow DOF etc.
For outdoor portraits, I use up to 200mm.
And lighting is everything.