What would be the optimal shutter speed for the prop blur while still photographing a jet @ 500mph?
Well, you still use the settings for the prop plane. They're both traveling at the same speed, it's not like the jet is overtaking the prop plane at some very high speed.
In order to get "some" prop blur, you'll need 1/250s. 1/200s mostly acceptable. If you're able to do 1/100-1/160s (you're getting pretty good). And under 1/100s, you're in "panning God" territory.
It can be done. Your success will depend on the angle at which you're shooting and also how good you are at panning. 




