I have the 85 and I also have the 100 macro and the 24-105L. I wouldn't sell a lens that I liked just because I haven't used it in awhile. If that were the case, I'd keep my 35f2 and sell the rest of my primes.
I have a Tokina 12-24F4 zoom. I don't use it a lot, but there have been times when it was needed... like last week. I want what it can do for me. Just because I don't use it every week, doesn't mean I won't use it on my next trip, for example.
Besides, it is fun having a variety of lenses to choose from. I just avoid needless duplication. I am an amateur, so I wouldn't get the 24-105L, and the 24-70F2.8, and the 17-55IS, and the 28-135, etc. The 24-105L may include the 85mm focal length, but nothing that I have seen will focus as fast as the 85f1.8, and it is a fast lens, i.e., wide maximum aperture. This is not duplicated by the zooms. The 85 is perfect for sports, and for some events. I'd rather bring the 85 with me to a concert than the 24-105F4 L, if I knew where I was sitting and the 85 were the appropriate focal length for that position.
I am keeping mine.