I take all of my gear whenever I go away. I guess the reason for this is because the only reason I go on vacation is to take photos of wildlife and nature. It would seem silly to me to head off for a week or two of nature photography and leave a few lenses at home. I can't take it "everywhere", as there is too much to carry on my person. So what I mean by "having it with me" is that it is in my car.
When I hike into the bush away from the road, I decide what lens(es) I will most likely need, and just take those. Usually that means the Sigma 300-800mm mounted on the tripod, and my 100-400 hanging from a strap around my neck (or in a coat pocket). Sometimes the 24-105 or the 50-200 will be in another pocket, but this is rare. The 1D4 will be attached to one of these lenses. I don't usually hike with a spare body, although there have been times when having two complete rigs has been quite helpful.
I keep a 14' orchard ladder on my car at all times now, as well. One never knows when one will find a photo opportunity that might be most effectively captured from a height greater than "tippy toes" will allow.
I have always been glad to have all of my gear with me. I don't really see any downside to it.