For me, too. When I was shopping for my first DSLR back in February, I was looking for a basic camera on a budget. My criteria were: 1) superb IQ; 2) durability; and 3) comfortable handling. Nothing else mattered.
(My eyesight is failing, so I need big viewfinder and big LCD.)
I tried the XSi, D60 and D90, and I found them very uncomfortable to hold and operate. I also didn't like the plastic construction.
But when I tried the 40D, I instantly fell in love with it! Perfect grip, fantastic ergonomics. All-magnesium body. Large and bright viewfinder. And at CAD$1180, it was easily the best photographic bargain on the market. Well worth the price difference. And this is for a basic DSLR camera.
(I'm still on a budget.)
Yes, the viewfinder definitely matters.
Similar story for me. I was going to get a T1i didn't like the tiny grip on it. Picked up the 50D and looked into the viewfinder. Done deal I was getting a 50D over the T1i.