To answer the question the OP asked, set your 24-105 to 24,35,50, and 85mm. Figure out which focal length works for the type of shots you shoot, then start researching the prime that's the focal length match what you've found you shoot. If you shoot head and shoulder candids of your kids primarily, a 24L would be a horrible fit, despite being a stunning lens. If you primarily frame the whole room with the kid in it, then the 85L despite being a stunning lens you'd find miserable indoors.
Alternatively, go back and find your favorite indoor shots and see what focal length they were taken at.
Now, with all that being said, EverydayGetaway is right on the money that most of the time people say "I don't like using flash" means, "I don't understand flash or I'm not comfortable using a flash". If that's the case, then learn to use a flash. Indoors it's a night and day difference between a properly, well lit shot, and using the average ambient lighting. That all being said, it is obviously much more intrusive and disruptive to use a flash, especially an E-TTL and can make it much harder to get true candids.