I went through the same progression you are, started with a 400 5.6L, went to the 300 2.8L and I'm now shooting with and planning to stay with the 500 f/4L. The 300 2.8 and 500 f/4 are both a notch above the image quality of the impressive 400 5.6L, they both focus faster as well. In my experience the 300 2.8L + 1.4TC wide open beats the 400 5.6L wide open which is impressive, with the 2x TC the image quality is still good but improves stopped down and you do lose some autofocus speed but it's not bad.
The only problem with the 300 is that even with converters it's short for birding in my experience. As far as carrying lenses around I didn't really notice the extra weight of the 500 as I just throw it over my shoulder with a tripod/monopod or have it in my Kinesis backpack. I've never had any issues walking for miles with the 500, granted I'm in pretty decent shape. Where you will notice the weight difference is when you are shooting hand held with your arms extended, the 500 is still useable hand held just not for long periods. You should handle both if you can, neither are really walkaround lenses (not that I haven't done it) in general and you might find the weight difference not to be that big. When it comes to birds, reach is king.
I'm taking pictures of everything, so go ahead and take this place away from me.
5DII, 16-35mm f/2.8L II, ZE 28mm f/2, 35mm f/1.4L, ZE 50mm f/1.4, 100mm f/2.8L IS, 580 EX