I ended up with the 28 f/1.8 USM when I wanted a normal walkaround prime for my original 7D. It's never been optically the best, but it's smaller and lighter than a zoom, and definitely faster. Still have it and use it occasionally, though I really should ditch it- the 35 f/2 IS is better in every measure. There's also the Tamron 35 f/1.8 VC, which hasn't been mentioned yet. I have the 45mm version and love that lens, but it's a bit longer than "normal" on a crop camera. The 45mm can be had for a song (I've seen under $300), the 35 is usually a bit more expensive, but not much...
I do think the EF-S 24mm STM is a great little lens, but I don't grab it for most of the reasons you mention wanting a prime- I grab it when I want the smallest, lightest least-obtrusive camera possible. It pretty much lives on a Rebel SL2, but I will occasionally throw it on the 7D2. And I find it maddening that the EF-M pancake is f/2 and the EF-S is f/2.8, but that's a different rant...
Anyway, my first choice would be the 35 f/2 IS, followed by the Tamron f/1.8 VC, followed by the Canon 28 f/1.8, followed by the EF-S 24.