I've worried about it in pickpocket heavy neighborhoods like Rome and Spain. I did have one encounter with a group of muggers/aggressive panhandlers in Paris that I managed to get out of.
I ended up buying a backpack that has opens in reverse (Manfrotto Veloce V) to reduce the risk of being pickpocketed, but it is definitely not as quick as a normal bag with a side/top opening. I often pass up shots rather than stop to get what I want out.
Maybe I'm worrying about it too much, have you ever had your bag pickpocketed?