There is a physical difference on a per-frame basis due to the difference in sensor size.
For full frame versus crop, the difference in high ISO performance will be about 1.3 stops in favor of the full frame camera. This is assuming the sensor technology used for both cameras is identical.
Beyond that, the differences will be attributable to the differences in the sensor technology being used. The 5Dmk2 probably gets somewhere between 1.5 and 2 stops improvement over the 50D in terms of high ISO noise based on the comparisons I've seen, which suggests that the 5Dmk2's sensor tech is a little better than that of the 50D.
Now, keep in mind, that's with per-frame noise. You can't determine per-frame noise by examining 100% crops, because a 100% crop only tells you what the per-pixel noise looks like, and that is a function of the size of the physical pixels. The size of the physical pixels, combined with the physical size of the sensor, can be used to determine the resolution of the sensor.
Consider the 50D and the 5Dmk2. The 5Dmk2 is a 20 megapixel camera with a full-frame sensor. The 50D is a 15 megapixel crop camera. If one were to build a crop sensor with the pixels from the 5Dmk2, they'd get an 8 megapixel sensor because of the difference in sensor size. The 50D's sensor packs nearly twice the number of pixels into the same area as the 5D2's sensor, so the 5D2's sensor is gathering twice as much light per pixel as the 50D. Which means the 5D2 has a 1 stop advantage over the 50D on a per pixel basis from the pixel size difference alone. Since the 5D2's sensor seems to be a little better than the 50D's in terms of the technology (since its performance exceeds the difference you'd expect due to the physical sensor size difference alone), it follows that on a per-pixel basis, it'll actually be doing a little better than one stop over the 50D on a per-pixel basis -- somewhere between 1.2 and 1.7 stops.
If all that sounds confusing, just think of it like this: the difference in per-frame noise is determined by the difference in sensor size and the difference in the tech. The difference in per-pixel noise is determined by the difference in the pixel size and the difference in the tech. Either way, unless there is a very large difference in the tech, the difference is going to be dominated by how much light can be gathered, either on a per-pixel basis or a per-frame basis.
The per-pixel noise is most useful when determining how a crop from a high-ISO shot will look. The per-frame noise is more likely to be what you really care about in the end because it will determine how the overall shot looks. The latter is why the claims that the 40D's high ISO performance equals or slightly exceeds the 50D's are misleading: they're based on examining the per-pixel noise, while the 50D's resolution exceeds the 40D's by a factor of 1.5. If the per-frame performance of the two cameras were the same, then the per-pixel performance of the 50D would be worse than that of the 40D by nearly 2/3 of a stop. Conversely, if the per-pixel performance of the two is about the same, then the 50D's per-frame performance would win by nearly 2/3 of a stop.