I got my oldest son a refurb HP something or other for Christmas when he was four. We chose the camera we did because it was cheap, and it looked like a brick, with nothing hanging off or that would pop up that could be easily broken. The battery cover is now held on with a rubber band (their design flaw, nothing he did), but he still loves to use it three years later. We love to see his perspective on the world and what's important, and how that's changed over the last three years.
As that camera is very simple -- the only options are on/off, zoom, and flash -- he's outgrown its capabilities. Since I've moved from a P&S Nikon Coolpix 4500 to the 20D, he now gets to use the much more sophisticated 4500. It has the full range of program modes, plus manual capability. He can choose an appropriate shooting mode and go to town with it, or I can manually choose settings for the situation and hand it over. As he's now sufficiently computer savvy (he's been teaching his grandparents and his mom how to do things on the computer for two years :lol
, he's able to learn how to do some PP work, too.
When he gets physically bigger, I'll let him use an SLR, too. Right now, if it's not on a tripod, it's just too big for him to be able to handle comfortably and safely. Something in the Rebel size might be possible, but the 20D just doesn't fit his hands right now, and would make it harder for him to take pictures. He does like to look at the pictures I take and we talk about why they look the way they do (flash, depth of field, shutter speed, etc.), and can experiment with those things on a smaller physical scale with the 4500.
Our middle son is now starting to get more interested in taking pictures himself, too. Right now he loves to borrow his grandparents' and auntie's cameras. Soon, we'll pass the HP down to him (he's now the same age that Brady was when he got his camera), and start the cycle over again. In four more years, if the HP has survived, our youngest son may get to do the same thing. 