I take it off when I'm shooting with the cam on the tripod or when I want to be less noticeable or intimidating. If the person being photographed is kind of shy or not very self confident it may well be intimidated by a large camera + huge chunk of glass like 85L.
I never take it off because of weight...





