I did just that and have not regretted the decision. However, if you are looking for a lens specifically for portraits, its hard to beat the 135L. I would loosely argue that the 135 does extremely well with portraits, so well in fact, that it almost seems to be a specialty lens with not a ton of versatility. Just my personal experience and I'm sure you can find plenty of exceptions/examples to counter that statement.
IMO, one major drawback to the 135L is, ironically, the 135mm focal length. During events and difficult lighting I use the 85 far more often than the 135. I found that when I want/need the 135mm length, the 70-200 fills in nicely (even for most portrait scenarios). The one thing I didn't realize prior to purchasing the 135 was that despite it being f/2, it really isn't suited for low-light situations due to the focal length. To eliminate camera shake, I always had to keep the shutter at about 1/250, so I essentially neutralized the benefits of the wide aperture by having to select a faster shutter speed. I'm not sure why I couldn't get away with a 1/200 speed, but for me, it really had to be 1/250 to get crisp shots consistently. (Maybe something to do with the size/weight of the lens???) With the 85 1.8, not only do I gain an additional 1/3 stop via aperture, but I can also stay at 1/100 to gain an additional 2/3 stop over the 135L. Therefore, I was gaining a full stop of light in real-world use of the 85 1.8 over the 135L.
So, while the above may not apply to your originally intended use, I think it's something worth considering before purchasing the lens.
I agree with mannetti21's post 100%. As you know, The 135 is really awesome. IQ is top notch. However, it is a bit of a specialty lens. A bit long a lot of times. But the real kicker for me was that I had to have good light to get sharp shots and that meant SS's >1/250 and preferably ~1/500. This limited it's use. The f/2 was more for throwing the BG OOF/subject separation rather than light gathering. So I sold mine.....to mannetti21!
I think the prevailing message in this thread is to simply pick up the 85 as it is a very good lens and quite affordable (as far as camera gear goes).
I would love to see a 135mm with IS
EDIT: You do know there is an 85 1.8 sample thread...right?