Portraits and candids....
Primes are nice for those purposes.... They can be relatively compact and unobtrusive than a big, fast zoom, as well as offer larger apertures than any zoom for more control over background blur when you want it, not to mention lower light shooting. Going to the extent of L-series, nice as they are, sort of defeats that. They aren't small or unobtrusive.
Plus you'll most likely need several primes. For the price of one of the L-series lenses, you can put 3 of the "lesser" models in your camera bag.
If you shoot fast, USM lenses are nice. It's generally faster and more accurate than micro motor lenses. Also, as a rule, Canon only puts USM on their mid-grade and premium lenses. Sigma HSM is similar to Canon USM. Tamron is starting to offer a similar USD drive, too, but has only put it on a couple lenses so far.
Taking all these factors into consideration, among Canon lenses it sort of narrows down to: 20/2.8, 28/1.8, 50/1.4, 85/1.8, and 100/2 (I'm not including the 24/2.8 and 28/2.8 STM/IS lenses because of cost and smaller max aperture). You might also consider Sigma 30/1.4, 50/1.4 and 85/1.4, though the 50 and 85 in particular tend to be pretty large and heavy compared to the closest Canon equivalents.
You say that the 50/1.8 covers about 80% of what you like to shoot, but only have that one prime so can't really say if you'd find other primes equally useful. But apparently you like to shoot with primes, so I suspect you'd enjoy using them.
I also think it would be good to upgrade from the 50/1.8 to either the Canon or Sigma 50/1.4. I prefer the Canon for the size/weight, but the Sigma is a little sharper wide open (I tend to stop the Canon down to f2.2 when I want to insure sharpness, but I'll use it wide open without too much concern if needed). Either one... the Canon with USM (though it's "faux" USM) and the Sigma with HSM... will give faster, more accurate and more consistent AF than the 50/1.8. Quieter too!
Yes, the 50/1.2L is superior in a number of ways... but it's also tons more expensive, as well as bigger, heavier and a lot more intrusive.
The 28/1.8 is one of the most compact of the Canon prime USM/mid-grade lens line-up, even with it's lens hood. It also shares 58mm filters with the 50/1.4, 85/1.8 and 100/2... which I believe your 18-55 also uses. The Siggy 30/1.4 is a popular alternative.
Personally I'm not a big fan of 35mm focal length on a crop camera such as your 60D. It's sort of a long standard lens or a very short telephoto. But, again, that's just a personal preference.The 35/2 is a fine lens optically, but it is not a USM lens and its build quality is only a notch about the entry level such as the 50/1.8. I'd rather have the 28/1.8 or 30/1.4. I left the 35/1.4L off the list, even though it's a wonderful lens (especially on full frame, IMO), due to price too.
If you like the 50/1.8, which behaves as a short tele/portrait lens on your 60D, I suspect you'd really like either the 85/1.8 or the 100/2. Those are essentially the same thing, but longer and with considerably better build/AF. Both are reasonably compact, with high image quality. The Sigma 85/1.4 is a viable alternative.
Yes, the Canon 85/1.2L is a premium lens alternative with amazing image quality, but very expensive, big, heavy and more intrusive. It's also less of an "all purpose" lens... more of a portrait-specific lens. It's AF is slower than the 85/1.8, by design (due to the large aperture, the focus is "long throw", emphasizing accuracy over speed). That makes it let useful for action/sports or other purposes where focus speed is important.
I didn't include the 135/2L on my suggestion list because of price and, even though it's a wonderful lens and fast focusing, it's starting to get pretty long on a crop camera. It's also bigger, heavier and a bit more intrusive than either the 85/1.8 or 100/2.
So, I'd suggest you look at, consider...
Canon 20/2.8
28/1.8 (or Sigma 30/1.4)
50/1.4 (or Sigma 50/1.4)
85/1.8 (or Sigma 85/1.5)
100/2
One other thing... do you ever shoot macro or want to? If so, you have two choices... use macro extension tubes or a close-up diopter on one of the short tele primes above... or instead get a macro lens that might serve dual purposes. You might consider a 60mm, 70mm, 90mm or 100/105mm macro lens, if you'd have some use for it and it can as a short telephoto. Maybe even a 50mm... though that will put you pretty close to your subjects.
Most macro lenses are slower focusing though. Might be fine for portraits, a little less so but still usable for candids in less stationary situations, and not so great for faster action sports. The Canon EF-S 60mm and 100mm macro lenses have relatively quick focusing USM, but are f2.8 at the largest. Tamron offers the only macro with larger aperture... their 60mm f2... but I sort of doubt it's AF is comparable (watch for a USD version eventually, that's Tammy's new drive system that's more similar to USM/HSM).
At any rate, if you are into primes, there are lots of options.
Have fun shopping!
