It has to do with compression. A high ISO image has more noise, noise is random, and random data is more difficult to compress than non-random data so it takes more space.
A simplified explanation...
So if you have a pic with a pure blue sky a row of pixels could get compressed as "3072 pixels of color 12,12,252 in a row". In computerese this might stored "12,12,252 x 3072"
But now add noise, so that every n'th pixel is some whacky color, and "n" is random. So now a line of pixels gets compressed as
"12,12,252 x 130", "19,211,43 x 1", "12,12,252 x 642", "231,47,188 x 1", "12,12,252 x 87", "22,119,55 x 1" and so on.
You can see how the noisier, more random line of pixels takes much more room to store the same line of pixels.
So higher ISO images don't compress as well, so they take more room on the card, so the card holds less images.