I have the previous models you mention, the 1D Mark II (non-N) which I got recently for under $800 and the 40D. I prefer my 1D in general with the one exception being birds that are very far away where I need to do a lot of cropping; there the 1.6x sensor offers more reach and the higher resolution allows for more cropping. In all other situations where I can frame appropriately, the advanced autofocus (45 AF points), spot metering linkable to 11 points, higher frame rate, better build and integrated grip make it a better camera overall. 1D Mark IV would be best of course but out of the price range...
The 100-400L is great with both the 1D Mark II and 40D, one of my most used lenses. All your lenses will work well with either body.
The 50D will be more like your 20D. The 1D is much bigger. Both will be an AF upgrade but the 1D will be more of an upgrade in that regard. The 1D will not give you higher resolution while the 50D will give you quite a resolution upgrade.
The 50D will give you a self cleaning sensor, better LCD for reviewing images, micro focus adjust. It is new technology while the 1D Mark II is old but I don't find I miss the new stuff much (I also have an even newer T1i).
Personally I'd get a Mark II which is nearly the same price as your 50D refurb they are offering.