It is not a focus issue... It's a motion blur issue... probably mostly subject movement, but might also be some camera shake blur happening .
50mm lens... you must have been pretty close. 1/160 was slow enough to produce a lot of subject motion blur. That's fine, if it's what you want. Panning rapidly with a 50mm lens, 1/160 might not be enough to fully deal with camera movement blur either.
Another thing... The shutter moves vertically, two curtains with a space or "slot" between that, along with the actual speed the shutter moves determine the "shutter speed". The higher the speed, the narrower the "slot", as well as the faster the movement of the shutter itself. But this means that different parts of an image are exposed at slightly different times. This was more apparent with older style shutters that moved horizontally. Motion blur would be different when the subject was moving left to right than it was moving right to left. But, even with modern, electronically controlled, vertically travelling shutters (based upon the venerable Copal Square invented in the late 1950s), there's still some effect with really fast moving subjects.
Add to that another variable... yourself. As you pan you probably don't do so exactly the same plane as the subject, nor at the exact same speed or even precisely the same direction. You probably "wobble" a little during the pan. You can minimize, but not eliminate this entirely with a monopod or a tripod (such as with a gimbal head and the panning axis left loose).
Whenever you "drag the shutter" (i.e. use a slow shutter speed deliberately to produce subject motion blur), there are a lot of variables and there are bound to be more shots lost to too much or too little blur. As best you can, experiment with different speeds and apertures and techniques. But differences in subject speeds, focal length (distance from the subject and the subject's distance from the background) are always going to happen. You never really "perfect" panning, per se, though you certainly can get better at it with practice and by refining your techniques. But even then there will always be some chance and luck involved. So take lots and lots of shots and simply plan to throw away more than you usually do, don't expect a high keeper rate.
For the above, the car wasn't going as fast as in your shot (autocross, so probably no more than 40 to 50 mph)... but also I was using a smaller aperture (f16), a slower shutter speed (1/60) and a longer focal length (200mm) so was standing considerably farther away. The lens I used is stabilized, too (IS was left on, but in mode 2 which only corrects for vertical movement).