
I can't help but wonder what you are doing to get this reaction. I'm a father of 4 (tweens and teens now) and, at playgrounds when they younger, while using a 7D/70-200 f/2.8 (or my older 75-300) to get good candids - I've never had a reaction even remotely like this -- and my children don't even look at all like me (i.e., to a casual observer, I was photographing kids other than my own).
Clearly, unless: you are on private property, there is a reasonable expectation of privacy, or you are taking lascivious shots of children; you can point your camera where you please. However, even though legally you can, common sense dictates that it is not a good idea to take pictures of other peoples children without the parents permission (there are many exceptions, e.g., when your child is also in the frame, sporting events, etc.) I am not saying you did anything wrong, but just wondering what is happening that is causing that reaction. For example, if you showed up in this van

there are places where you cannot legally photograph children in public parks without the parents' permission. I'm not saying that's what he was doing, but some parents have gotten hyper-sensitive to photographers around their kids.