Toughest PARTS for me:
1. Getting a shot that impresses ME. I can't seem to appreciate my own shots or critique them.
2. Breaking out the SLR in public/crowds. I've always wanted to wander around a city and take shots of people. Figured I might get punched in the face, told off, or my stuff stolen so I don't do it. Shyness.
Don't be afraid to do it. I've walked around many parts of Houston with my camera, and for the most part nobody pays attention to you. Obviously you don't want to wander around with a tripod, or stand there forever trying to compose a shot of someone. I usually put the camera in AV mode and then go for it.
Toughest things for me are definitely getting a shot that impresses me, and gear envy.