I started Digital with no knowledge of card speed and a bare understanding of capacity. My first cards (CF) were in Kbs and had the blazing speed of 33X
As I upgraded camera bodies over the past decade I've come to the conclusion that WRITE speed is key, followed by READ speed and then capacity.
It really behooves a person to check those Read/Write specs because the way they are presented is often misleading.
That said, I've kept my older cards as back-up but my working cards are Lexar Pro 64GB 1066X.
I'm using a Canon 1D X and I find that 64GB will give me well over 2000 RAW shots.
The 1066X speed equates to guaranteed minimum write speed of 65MB/sec and up to 160MB/sec if your camera is able to do it.
With the 1D X I can fire long bursts at 14fps without stalling or significant slowing.
Using a UDMA 7 enabled reader allows card-to-computer transfer at the top end of the speed.
The "all your eggs in one basket" approach has merit, as does using multiple cards. I've seen more cards "fail" outside the camera (Damaged, Corrupted, Lost or Stolen) and there's more chance of pin damage as card changes increase.
So, "YES", size matters to the extent of your personal needs, but I think one can easily go overboard.