Having second thoughts

After a lot a back and forth I went with the 400 5.6 over the 100-400. Took it out for the fist time and not thrilled with my shots. I was taking photos of an osprey that was about 150 feet away. Lens and camera mounted on sturdy tripod. shutter set to 800, AV at 8.0. I did not use timer or cable for shutter release. All of my long shots are OOF. Lens or pilot error?

There is a learning curve with new glass, especially longer lenses.
Try hand holding and take a lot of shots at higher shutter speeds.
That lens has a stellar reputation.

I'd perform some static tests shooting against a focus chart before saying either. Could be you or a calibration issue.

You made the right decision. Now, learn how to use a long lens. Hope you did not expect your first shot to be stellar. It is all about technique.

Try it with Live View on a tripod and manual focus at 10x magnification with some static object with texture. If the result is razor sharp, you know it's an autofocus issue that could possibly be resolved with microfocus adjustment. That was the case with my 400 f2.8, where it was backfocusing a little too much for my liking. Now it seems to be right on the money.