Good point, but I think these incidents are few and far between. It is still lawful to be on public property and shoot into military installations.
In those cases where a military facility is supposed to be secured against photography, I would expect that commander will erect sight shields where necessary to protect specific sensitive areas from at least casual ground-level photography. That's a pretty obvious precaution--anything less than that would be dereliction of duty--he certainly can't expect to catch everyone driving by his base with a cell phone camera. There will also be plenty of warning signs posted.
There is a big difference from taking pictures lawfully and a camera being used in a commission of a crime or being the fruit of a crime. For instance, a few years back when those idiots went around video taping themselves shoot pedestrians w/ paintballs, the camera and tape was used as evidence. If someone was on the street corner video taping whatever and they happen to catch the same crime being committed, they are not required to turn over their tape as evidence w/out a court order. They can, on their own, turn it over to police to aid in the investigation.
In a practical manner of thinking, the issue is really only an issue if the photographer is otherwise not committing a crime.
If a crime is involved, the images themselves might be material evidence--that would be determined by someone other than the arresting officer. The arresting officer could not delete the images legally. That would be like an arresting officer deciding to mop up the blood at a murder scene.
There just is no legal reason whatsoever for a photographer being ordered to delete images--except in the one circumstance I mentioned. And as you say, that's going to be pretty rare, and not a surprise when it happens.