This topic is the sort where there are no absolutes, and we end up arguing over where to draw the line. Such arguments are endless, with impassioned participants finding obscure what-ifs to challenge any sort of consensus. So that's a great way to explore a topic, but let's keep friendly through the process.
Legally, people can be observed when on their own property, but not through their windows. This detail is obtained from the point of view of insurance companies trying to identify work comp fraud, but I understand the restrictions/ permissions to be non-specific.
I recall a case in the news where a homeowner went around nude in his home, and was able to be seen from the street through a front picture window. Law enforcement had no recourse, as the homeowner had broken no laws, and if anything, those who watched the man were peeping toms. There's always more to the story than we ever get from our corporatized, sanitized press. Was the person parading around at opportune moments to elicit attention, or was he a hard core nudist genuinely moving about his home? Legally, I suppose it doesn't matter, but it does indicate what he person was trying to accomplish.
So most people might say that a person crosses the line when they sneak up to a window to look through, but legally, just looking through a window is an invasion of privacy regardless of where the viewer stands. So it goes in the USA, but I also recall a story about a judge that allowed artwork based on photography of street scenes that included views of apartment dwellers within their apartments. So maybe the whole 'in public view' thing is a line that is drawn differently in different nations, but since it caused worldwide reporting, I suspect that there is an expectation universally that we should be left alone when in our domicile.
So can a person use a drone, or any camera, to take pictures of you in your backyard? For non-commercial purposes, I think the legal answer is yes. For commercial purposes, drone use would require an FAA license, and use of the images would require some sort of model release. I'm not sure about how model releases work, but I know all about the FAA requirements - I'm not allowed to use a drone at work for imaging because it would be a commercial activity requiring FAA Licensing. We ended up hiring a company that used the octocopter to suspend a 5d3 - very cool stuff BTW.
So let's set aside the photography aspect briefly. Can a person look at you when you're in your yard? If they were waving hello, I doubt we'd protest. If they were taking photos with a 400mm lens, most would feel uncomfortable. Legally? I don't know that the law sees a difference.
Making common expectations into law is difficult because the expectations are so often contextual, as in the above paragraph. A common expectation would be to greet the neighbor, examine their artful landscaping but not their proverbial sunbathing daughter (returning to the topic's original premise). I'm in agreement about the expectation, but how do we legislate this to everyone's satisfaction?
Let's start a new nation up (thanks to Mike Stipe) and explore ideas for the perfect privacy laws. I'll start:
People covered head to toe at all times in public with a non-form fitting outer garment to eliminate them from at least detailed view.
8' privacy fences now required
No windows in our domicile for people to look though, so those within cannot violate the privacy of those walking by, and vice versa.
Violations of privacy laws are enforceable with lethal means by the person whose privacy was violated.
Hopefully most see that draft set of rules as tongue in cheek, but also hopefully thought provoking...