When using Servo AF, the camera has to carry out at least one focusing operation between each shot. Otherwise it would not track focus, which is exactly what Servo AF is all about.
Now it shouldn't be too much of a surprise that to do the metering, evaluation and focus drive of the lens takes at least some time, even if the camera is quick.
Compare that to using manual focus, where none of these events, i.e. metering focus situation, evaluation and lens drive has to occur.
Which do you think take longest, do something or do nothing?
Now the 7D is designed in such a way, that in good conditions it has enough capability to actually execute all the AF related tasks in between two shots, even in high speed continuous. But this takes that the subject has a reasonable contrast somewhere, that the lighting is such that this contrast is bright enough and that you, the photographer, manages to keep an active AF point on the target. If any of these conditions fail, then the camera operation will slow down. If you have it set to AF priority, which is the default, it will slow down more than if set at drive priority. In the second case, the risk you get an image out of focus increases instead. Sometimes a slightly dizzy image is better than no image at all. That's up to you to decide, hence the setting.
Note that under good conditions, you can't test the difference between AF and drive priority. Under good conditions, there is none, since then the camera is capable of doing everything it should even during the short time frame allocated to AF in drive priority, so there is no difference to see.
Noteworthy is also that for the camera to be able to measure focus, it needs a contrast. If the camera is shaking a lot, the contrast gets blurred on the AF sensor, and then evaluation is everything from slower to impossible. Using IS for action photo is kind of a double-edged sword. Although it sometimes ends up with lagging behind your intentional camera movement, and therefore causes blur instead of reducing it, if you take the picture when the camera is catching up with your movements, IS stabilizes not only the picture, but also the viewfinder. And since the AF sensor sees the same thing as you see, it also stabilizes the contrasts presented to the AF sensor. Which makes AF calculations faster, which in turn may keep drive speed up.
Now I notice that your lens doesn't have IS, so if it has any effect on your images, it's slowing down AF in that case. Since I've never seen you work, I can't tell if this is really something that happens, but when you know this, you have to make your own judgement about that.
Another issue is of course how many AF points you use, and to which degree you make them stay on target. I can't judge that either.
Finally, it's true that in really low light, the continuous drive speed of the 7D goes down to about four frames/s. But this is in such low light that in most cases you'll loose drive speed due to slow shutter speeds anyway. Count on that you have to use 1/500 s or faster for the camera to do its best.