In the first couple of crops of the plants it looked just like undeveloped bokeh should...slightly OOF but not so much that the object cannot be discerned.
The photo of the number plate doesn't look too nice and neither do the ones of the swan. But when it comes to long lenses like this one, I think a phenomenon known as heat haze sometimes causes a deterioration of image quality when photographing far away objects. I've had the same issue with mine.
With near subjects, it brings in good sharpness, but far objects have this moth-eaten appearence. That said, the 100-400mm seems to produce softer results in less than ideal lighting, although I don't know the real reason why. Pity I'm not working from my own computer at the moment, otherwise I'd post up a crop of a butterfly at the zoo. That photo was so sharp I could almost see the scales on the wings. I will edit later and post up a crop.
I hope (and believe) that there's nothing wrong with your lens. It's just a bad habit of this piece of glass to show some softness when shooting into shade. Give it light and it's really good though. 
EDIT: Alright...now that I'm home, here's some 100% crops. It's not a precise test since too many factors are involved, but it compares nicely the results between a near subject in good light and a far away one in more subdued lighting, with roughly the same camera settings in terms of saturation, contrast and sharpness.
Butterfly

@ 275mm, 1/640s f/8, ISO 1600, near subject & good lighting
Lizard
@ 400mm, 1/500s f/5.6, ISO 400, near subject & good lighting
Dog
@ 400mm, 1/400s f/7.1, ISO 400, near subject & good lighting
Purple Heron
@ 400mm, 1/250s f/5.6, ISO 400, very far subject under shade