That is all I put on mine, too. Lens hoods offer far more protection against knocks than filters do, although filters can be useful in conditions of blowing sand or fast flying stones.
I have given my 100-400 many bangs which would have broken a filter, but not affected the front element (which is much tougher), so I have had no damage. A filter only protects against flying debris, if you knock the lens it does nothing to protect it and the filter can break and the broken glass can scratch the element, thus doing more harm than good.
Filters for protection are simply not economical for me, I have several lenses so that adds up to several expensive filters. buying that many filters would cost me far more than replacing a front element, and would be a definite cost whereas the element is only a possible cost that probably will never happen. For me, spending several hundred pounds to avoid the possibility of a £150 or so repair bill isn't an option, I could have another whole lens for that.
In over 25 years of photography, during which I have ill-treated my gear in many ways, I have never damaged a front element. I do have two lenses with severely deformed filter threads, due to knocks that would have certainly shattered a filter, but they still perform flawlessly and without a mark on the glass. I couldn't fit a filter on them now though, even if I wanted to.
For every lens I have seen that has been 'saved' by a filter, I have probably seen another one that has been damaged by broken filter glass.
There is no right answer though, there is an eternal argument on here between the filters / no filters brigades. You just have to decide what to do that is best for you.
The 100-400L is known for wanting high quality filters though. There have been a few threads on here by people complaining that their 100-400L is very soft, only for them to find that it became sharp once they took the filter off.
did read the whole thing (im editing - you know how that is)
I just figure everyone knows a lens hood is a given, it also off improved IQ by not allowing stray light to enter from angles on the side of the lens and or out of your angle of view. I hear what your saying but things happen (ie sand rocks hehe) also one branch on a tree facing you will go right through the lens hood and scratch your coating on the front element, thats a hefty fix out of warranty, not too bad IF you even have to replace the 80$ filter 
For me its a piece of mind that i can focus more on what im doing and not falling on slippery rocks and running through low bushes while no worrying about where my lens is in relation to my direction



