★ Best software to check data integrity of backups

I have over 500,000 pictures/videos backed up on several drives. What is the best software that I can run to help me check whether none of them is corrupt on a quarterly basis?

Thanks!

If you mean some software package that can run on a folder and report back damaged files; then that's a tall order and I'm not sure that such software exists. And one has to ask just how often does a file become corrupted and what is your overall backup strategy. Is this just a concern or are you actually experiencing a need for such software?

Having said that there are some good products in the marketplace that are more or less dedicated to image type files. Unfortunately they usually can only handle jpegs, not RAW. One that comes to mind is Hetman File Repair (see https://hetmanrecovery​.com/file_repair/softw​are-2.htm).

I have to say, in about 15 years of doing digital stuff, I don't think I've ever encountered a corrupt file. But I'm also not a pro so if I did I might not even know it (or care). But I do maintain backups.

I'm with John.

When copying files, the software can check that the copy is identical to the original, this is the best point (possibly the only reasonable point?) to check data integrity.

If a file is just sitting there on a regular (non-SSD) hard drive the greatest risk by far is mechanical failure. I've been working with digital media since the early 1990's and I'm struggling to remember a time that I had files go bad that wasn't tied to a hard drive failing. This is the reason people keep 2-3 copies of important files.

I'm actually dealing with copying data off a failing HD now. Carbon Copy Cloner tells me what files are corrupt at the end of the copy process. Thankfully this drive is pretty much filled with junk ... no work files.

the.forumer wrote in post #18603723
I have over 500,000 pictures/videos backed up on several drives. What is the best software that I can run to help me check whether none of them is corrupt on a quarterly basis?

.
I have never heard of such a thing.

Are you sure that what you are looking for actually exists?


.

If you're dealing with an actual backup, not just drives full of files, then your backup program should be able to perform a verification. I use Cloudberry backup and it has such an option.

If I understand what you are asking for you can do this by comparing the checksum of the local and the remote file. If the remote file differs from the local file by even one byte the file will be flagged. There are various checksum algorithms. One popular example is MD5. A google search for MD5 software programs has some hits that may work for you.

Most good backup software will have an option to perform this check during (and even after if you didn't do it when you first backed up the data) the backup process. If your backups exist in the same state as the original files (ie, not in a proprietay file type or encrypted), one that will likely work on the local and remote data that you have now is GoodSync. You tell it what folders to compare on the local and remote drives and in the options there is a setting to "Compare checksums of all files (slow)". It will then list any files that have different checksums which would suggest a corrupt file. They have a free evaluation period so you can check it out first.

Bcaps wrote in post #18604122
If I understand what you are asking for you can do this by comparing the checksum of the local and the remote file. If the remote file differs from the local file by even one byte the file will be flagged. There are various checksum algorithms. One popular example is MD5.

Jpeg images are inherently difficult for a checksum comparison. Explore the process at https://www.controlled​vocabulary.com/imageda​tabases/de-dupe.html to see what is involvedto check for duplicates.

I currently manage backups manually because there are so many drives and so many permutations (e.g. my 1TB 3.5" stores pics from 2007-2010, my 4TB 2.5" stores videos from 2009-2017 etc). Is there any backup software that is flexible enough to let me program to the fullest extent for automatic backups? (talking about offline backups in this scenario)

Separately, i have personally encountered corrupt image files before, although that was the point after I transferred files from my A7Rii to my PC, and not after X years of storage (I rarely access my backups unless I need to), hence the question for data integrity check.

So there are a few approaches here.

- Use the verification tool included in your back up software of choice. (For the enlightened, this means " /v " )
Most back up software will have this as an option. Most strangely do not use it by default, favoring speed over integrity, ( which seems counter intuitive for a back up app.)
There are different methods used by differing apps.

- Use a file/directory comparison tool after the fact. I've never done it this way, but the tools do exist both as stand alone, or your backup app might be able to do it.

eg:
https://backbox.com …ions/backup-verification/
A wiki study;
https://en.wikipedia.o​rg …_of_file_compar​ison_tools

You could probably just use PowerShell to do it if you have any coding skills... For each file, create a hash and store that off in another file or database. The quarterly, run the same cmdlet again the compare the hashes. If the file is corrupted, it SHOULD present a different hash especially when you read the file into a binary stream and then calculate it. Pretty simple to do and probably pretty quick. I can compare the hashes of around 15K objects in about 30 seconds when doing it in parallel.

Good luck with your search.

What you have is copies of your files, not backups. Read my article on backups. I'll wait here until you're done.

.
.
.
.
.
.

Ok, now you know about incremental backups, the 3-2-1 rule, and the need for backup software, cloud vs local, etc. I suggest you choose backup / archive software and get everything into a good system. Have your backup software run integrity checks say every three months. If anything fails you have at least two other copies of any file.

I keep my data in my house on hard disk, nearby on hard disk, at work in my drawer on hard disk, archived in Amazon Glacier (medium res jpg / 720p video not full res), and recent files go into a version controlled S3 bucket until they're into my standard backup system. Usually I leave them on S3, it costs little at my current volume of data. What I like about AWS S3 / Glacier is 99.9999999% durability (or something like that), as the data is stored in three different data centers and they do regular integrity checks. Glacier isn't exactly cheap, 10TB of data would cost US$40 per month, which is why I store smaller images - 120 weddings plus 15 years of family photos costs me about $1 a month to keep in Glacier.

tim wrote in post #18605114
What you have is copies of your files, not backups.


If anything fails you have at least two other copies of any file.

Copies or backups, what's the difference?

If I use a copy >>> paste operation to another "destination" is it a copy?

If I use dedicated backup software, is it a backup?

Personally I use a batch file to accomplish it all, doing full and incremental backups and retaining the full file name to make life easier when I need something, which hasn't occurred in 15 years of digital. But then again I don't have 1/2 million images!

John from PA wrote in post #18605146
Copies or backups, what's the difference?

If I use a copy >>> paste operation to another "destination" is it a copy?

If I use dedicated backup software, is it a backup?

Personally I use a batch file to accomplish it all, doing full and incremental backups and retaining the full file name to make life easier when I need something, which hasn't occurred in 15 years of digital. But then again I don't have 1/2 million images!

True backups give you a version history. What happens if you do a backup, then discover that the one you needed was a prior version of a file?

As others have stated, integrity checks in the purest form is difficult. Look for backup solutions that compare hash values, which in my experience has never failed.

I have several complex backup scenarios both from and to multiple devices. I've started using SyncBackSE last year and it easily accomplishes everything I need.

The issue here IMO is not so much that a backup will suffer corruption, but that your working files will. Errors happen over time. There are software crashes, power failures, MS can commandeer your box for a forced update, and so on, not even talking about user error or viruses. These things can corrupt files. Some of those corrupt files look fine to the operating system, but the content is bad. You won't know the file is corrupt unless you open the file. And you are not going to open 500,000 files, so you won't know if an old file has gone bad.

Then when you do your backup (or copy), the bad file is archived. Over time and after consecutive backups, you will eventually lose all copies of the good file.

It happened to me... opened two old *.JPG files and the images were scrambled. My backups all had the corrupted file too. In this case it didn't matter because I still had the CR2 files, but it is scary anyway to see that files can go like this, unnoticed. This was on my Win 10 box.

Another time I was backing up the contents of my wife's Android tablet. I copied the files over to removal media (flash drive). About 1/3 of the files were corrupted. The only way I knew was by opening them. No warnings or error messages. Jpgs, PDFs and other file types were involved. There was nothing wrong with the files on the tablet, and copying them over multiple times eventually gave me good copies of most. So the OS may not tell you that there is corruption going on.

You might have corrupt files on your system and not know it.

After my experience with the corrupt JPGs, I was worried that my HD was full of corrupt files. I wasn't able to find any software to automatically check the integrity of all my files. But there was one that could check JPGs. I ran it and it only found the files already known to be bad. From that I concluded that there probably weren't many bad non-JPG files in my system either.