I almost never turn IS off when shooting sports, even when using a fast shutter. Even if it isn't necessary to remove movement blur, I find it helps when framing at long focal lengths.
As for the different modes, I can only talk to the Canon system.
Mode 1 is 'conventional' IS, i.e. it stabilises the image on both axes, reducing both vertical and horizontal movement. I use this mode for still subjects and for subjects approaching me more or less head-on.
Mode 2 is designed specifically for panning and so only corrects for vertical movement. Obviously, I use this mode when panning in either direction. It seems to work fine, though I haven't performed a rigorous comparison between modes 1 & 2 for panned shots. I 'think' I get more keepers with it on.
Mode 3 is designed to actuate the IS only just before the shutter fires, thus (I presume) saving battery life and wear on the IS mechanism. I never use this mode because I find permanent IS useful for framing and exploring, just like IS-equipped binoculars.
5D3, 7D2, EF 16-35 f/2.8L, EF 24-70 f/2.8L II, EF 24-105 f/4L, EF 70-200 f/2.8L II, EF 100-400 f/4.5-5.6L II, EF 1.4x III, Sigma 150mm macro, Lumix LX100 plus a cupboard full of bags, tripods, flashes & stuff.