Although the IS will be of little importance for camera shake, once you get past about 1/600th, it does still help as it steadies the viewfinder and makes it easier to keep your AF point exactly where you want it. that isn't always an isue, but sometimes a slight movement of the camera can move it off target enough to spoil the shot. When shooting birds for example, I want my AF point right on the eye at all times, with small birds it takes very little camera movement sometimes to slip off the eye and onto a twig behind the head.
The best thing is to just play with your setup and see when and how you are finding an improvement with IS and at what speeds and in what situations you feel better off without it. A lot depends on your personal shooting techniques and habits.
Personally, I rarely turn IS off, even when using fast shutter speeds. I would only do so if tracking an unpredictable target which can suddenly change direction, as that can cause the IS to fight the panning as you adjust your movement. But, that's me. Others may turn it off more than I do, some never turn it off. Experiment to see what works for you.