If you don't need the wider aperture for skinny DoF, or the fast AF (especially compared to the macro), or if the purple fringing of the 85 (either the 1.8 or the L!) is getting to you, then yes, the 100L will do the business. You may even need to apply a bit of blur in your portraits, it's that sharp.
As for the utility of IS, it may be a bit limited in practice, since most portrait subjects are not truly static, and you'll need 1/60 or faster to be reasonably confident that subject movement can''t mar the result.
I have the 100L, but I use it just about exclusively for macro work; I do few portraits. That said, if the need arose, I'd not hesitate to grab it even if I had an 85/1.8 in the bag. If there was an 85L there, well, I'd grab that if I wanted marginally superior bokeh (by my taste, from what I've seen of 85L images), or needed to get just one eyelash in focus.
For context, I use a 6D almost exclusively these days; my 30D has earned itself a rest.