Manual focus and One-Shot focus are fine. It is only AI Servo that causes the slow-down. It is a physical limit due to the design of the older Arc-Form Drive AF lenses.
I asked Chuck Westfall about it and here was his answer:
THE DIGITAL JOURNALIST
by Chuck Westfall
I have recently noticed that my older EF Arc-Form Drive lenses will cause the frame rate to drop to about 4fps on my 40D when shooting in "AI SERVO". Even with a stationary subject and the camera on a tripod.
This doesn't happen in "ONE SHOT". In "AI FOCUS" I will get 6.5fps, but as soon as the subject moves, it again slows to 4ps or less.
Can you please explain why this is so?
Except for the EF80-200mm f/2.8L and EF20-35mm f/2.8L, all other EF lenses with AFD (Arc Form Drive) focusing motors were designed in such a way that autofocusing and diaphragm operation (via the Electromagnetic Diaphragm, or EMD for short) were carried out sequentially rather than simultaneously. This has no impact on continuous sequences in One-Shot AF, because no autofocusing occurs after the initial AF lock in this mode. But in AI Servo, both the focus and the diaphragm are operated for each photo, and consequently the camera's framing rate decreases.
For what it's worth, AFD motors were phased out of the EOS system for all new lenses introduced from 1990 onwards.
Best Regards--Chuck Westfall