Yes, I do handhold my 500 II almost 99% of the time
It's a little beast, takes a lot of getting used to, but definitely do able. For me, it strikes the right balance between aperture, focal length and weight. Cost was a pill to swallow though...
I shoot small birds, need all the light I can get. Not only does the f/4 vs. f/5.6 help, I have more "reach" when using a full frame camera (5D4), which has about a 1-stop noise advantage over my 80D when it comes time to push ISO. This advantage applies for the DO II as well...
Being able to get to 800mm if need be is awesome. At 560mm f/5.6, it's razor sharp with no IQ loss. Here's the cool part: when the zoom is extended to 400mm, the DO II is physically shorter! Not only that, the 100-400 at 400mm isn't really 400mm, more like 389mm or something like that, so you get a tiny bit more reach with the prime. The DO blows the zoom out of the water when using TC, so there's that too.
Stunner of a lens, the DO II. If I had to choose between that and the 100-400 II now, I'd personally take the prime. For you, OP, doing BIF is a challenge because of the inherent variability... definitely go with the zoom.