The best portraits I've ever seen are taken by the 135mm f2 on a FF camera. Please explain what you don't like about the results of the portraits you've taken with this combo. Is the lens too sharp? Is the bokeh too creamy?
About your choice between the 35mm and the 85mm, the 85mm is a great portrait lens, the 35mm is not.