If it's a choice of one lens or the other....
...If macro is a high priority, stick with the 100/2.8.
...If you do a little macro and a lot of portraits, you might consider getting the 135L and a set of macro extension tubes.
I use both.
To me, 100 macro portraits are a bit "clinical" looking. Most portraits don't need (or want) the extreme detail and crispness that the macro lens can deliver. It's a great macro lens, though, on either crop or FF cameras.
Wide open the 135L has a more dreamy look... something the macro lens really can't do at typical portrait distances and that's nice for some types of portraits. Stopped down, the 135L is very sharp, tho perhaps not quite as sharp as the 100mm macro. The 135L is also faster focusing than the macro. So it would be a better choice if you shoot from the hip a lot or do some sports/action work.
However, if the 85mm was too long for some of your work, then the 135mm is going to be just that much worse. (MFD can be improved by adding a short macro extension tube, BTW. I'd suggest you get a set.)