Is there any way to validate DNG files without Lightroom?

I have a lot of DNG files and I know at least one or two have errors. Is there a tool that can validate the checksums in the DNG files? This will help detect and prevent bit rot.

I know Adobe has an API that can do it, but I don't know if anyone has developed a tool that uses it.

Right now the only method I can see is to convert them all to DNG again and manually look for error messages, a bit of a long process when you have tens of thousands of DNG files.

The DNG SDK contains a utility called "dng_validate" that you can run in a command line environment. For example:

KT-Laptop-3:TestRaw kirkt$ dng_validate /Volumes/KT_External_S​SD/5D3_Tests/Focus/_R0​00829.DNG 
Validating "/Volumes/KT_External_​SSD/5D3_Tests/Focus/_R​000829.DNG"...
Raw image read time: 0.289 sec
Linearization time: 0.033 sec
Interpolate time: 0.301 sec
Validation complete

Great, thanks Kirk.

I'll write up some kind of a script or program that can work on multiple files or directories when I get around to it. Unfortunately I haven't done CPP in many many years and don't want to bother extending what they have, so I'll probably write a Java program that runs it or something.

If you group all of your DNGs in a directory and use the command:

dng_validate *.DNG > output.txt

Validating "_R000849.DNG"...
Validation complete
Validating "_R000850.DNG"...
Validation complete
Validating "_R000851.DNG"...
Validation complete
Validating "_R000852.DNG"...
Validation complete
Validating "_R000853.DNG"...
Validation complete
Validating "_R000854.DNG"...
Validation complete
Validating "_R000855.DNG"...
Validation complete
Validating "_R000856.DNG"...
Validation complete

Thanks Kirk. I have very deep nested directory trees. Knocking up a little program will only take 30 minutes or so, I'll try it this afternoon.

Well it took a few hours to knock the program up, but it works fine. It's just a wrapper, I'll release it as open source sooner or later.

The problem is the DNG SDK only supports files up to ACR 5.6 - anything later (ie compressed DNG files) can't be checked as the SDK is too old. Anyone have another suggestion?

I've posted to the Adobe forum, but they planned to release support two years ago but never got around to it.

So far my tool has found problems with about 20 files, out of the 40,000 I've checked. As well as DNG I have RAW and jpeg versions, so I'll just juggle things about to make sure I have good copies of all.

I wonder if there's a similar utility for Canon or Nikon RAW files? I could validate them too.

tim wrote in post #17078194
I have a lot of DNG files and I know at least one or two have errors. Is there a tool that can validate the checksums in the DNG files? This will help detect and prevent bit rot.

I know Adobe has an API that can do it, but I don't know if anyone has developed a tool that uses it.

Right now the only method I can see is to convert them all to DNG again and manually look for error messages, a bit of a long process when you have tens of thousands of DNG files.

Is it possible to validate the checksums in other RAW formats? Can the DNGs be checked with Photoshop instead of LR?

tim wrote in post #17079391
Well it took a few hours to knock the program up, but it works fine. It's just a wrapper, I'll release it as open source sooner or later.

The problem is the DNG SDK only supports files up to ACR 5.6 - anything later (ie compressed DNG files) can't be checked as the SDK is too old. Anyone have another suggestion?

I've posted to the Adobe forum, but they planned to release support two years ago but never got around to it.

So far my tool has found problems with about 20 files, out of the 40,000 I've checked. As well as DNG I have RAW and jpeg versions, so I'll just juggle things about to make sure I have good copies of all.

I wonder if there's a similar utility for Canon or Nikon RAW files? I could validate them too.

I am aware of the bit rot phenomenon. Have you thought what may have led to those 20 files being corrupted? Are those files lost for good?

The DNG convertor actually does some validation of camera manufacturer raw files although I haven't taken the time to actually try to understand what it is checking. There is at least one downloader/image transfer utility that uses the DNG process to validate raw files during transfer even though it allows you to leave the raw files, not DNGs, at the target location. I think it was this one:
http://basepath.com/ne​w/detail-ImageIngester.php

The developer of that utility also has this utility that was mentioned in one of the editions of the DAM book:
http://basepath.com/ne​w/detail-ImageVerifier.php

Also ran across this thread:
http://photo.stackexch​ange.com …ity-of-a-series-of-images

Good to hear products are already available, but their SEO sucks, I couldn't find them despite searching.

Unsure what caused the bit rot. My disk has two sectors gone bad, maybe that, or maybe just that hard drives aren't 100% reliable. I haven't lost any data, I have a native RAW copy of each image onsite, DNG files offsite in two locations, plus JPEG files of every image in each of those locations.

I've solved the problem too - I was using an old version of the DNG SDK, as the Adobe website linked to the old version not the new version. Now that I have the latest version I'll re-run my verification and see what happens.

I've looked for an open source program or library to validate other RAW files but I've had no luck. If anyone knows a suitable library or piece of software that works from the command line let me know. dcraw was my first stop but it converts the broken files fine and doesn't seem to report any problems.

I've released my DNG validation tool, you can read about it on this thread. I won't respond to questions on this thread about the tool, but happy to continue the discussion about validation libraries etc.

If anyone's interested in data corruption there's a good article here.

Thanks Tim!

kirk