Archiving 10000 images ...what to use ?

I am finally getting around to creating back-up copies of my images off of the computer for storage outside of the house. I already do have an external mirror hard drive as by back-up when ( not if ) my hard drive fails.

I use Lightroom to create DNG files from RAW so the image files are rather large 15-20 MG. I have started to use DVDs but this is becoming a long task at 5 min. per disk. I figured that it would take be about 6 hrs to complete two sets.

Anybody use a different medium for backing up files to store outside the house?

Small portable hard drives would be quick

Agree a pair of portable hard drives keep 2 copies.

I have roughly 100,000 images (raw files) archived.
1 set stored on my server and copy stored on a 1tb portable drive stored offsite.

That is just archives. Processed images exist on the main computer and Smugmug with the main computer backed up weekly.

Reading this... Reminds me to get back on that. Ive got about 50000 raw files on multiple HD's.

First before actually archiving data there is one very important question to ask yourself:

Do I actually care?

10 000 images is a fairly large collection. I've seen far larger collections of Data in my time, and I'm sure lots of photographers hang on to far greater collections than that, but why are you storing all of it?

Are they All important? Do you see yourself wanting the entire collection again at a later point in time? Is it impossible to cull this down to a more reasonable number?

After you figure out what you really need to store, then start looking at your options.

Personally I'm kind of a fan of Network Attached Storage (NAS) boxes, because of how easy some models are to use and setup. And they can be stashed in out of the way places, rather than taking up extra room right on your desk or near your computer.

For files that are really important, then you really want to use off-site backups. That way if something happens to your home, you don't lose your stuff. Well, all of your stuff. Currently I have a cross-share setup with some friends, which is basically a custom script we have setup using the program Unison to sync folders at each other's houses. We each get 1/4 of a 1.5TB drive, and every week that drive gets archived locally at each location. (The archive is to prevent someone from doing something foolish, like accidentally deleting files from their folder, and then passing that change around to all the others. In theory we can't lose more than a week of work.)

Failing tech savy friends to do something like that, then I would consider one of the many "Cloud" based services for online backup, combined with a nice little encryption system on your files before you upload. Just remember the password to your files. DropBox lets you backup 2GB of your most precious data for free.

However, I would not suggest relying on any free service as your critical archive method, or even a single service if your work is exceptionally critical.

We have over a half million images (RAW's, jpg's, tif's) backed up in triplicate. Two copies onsite, one copy offsite. 1TB hard drives cycled through a toaster drive as required.

Cloud storage makes no sense to us due to sheer volume (nearly 3TB of data) . NAS makes no sense to us because of added overhead for cases, power supplies, software, etc.

Does it make financial sense for us to spend money/time on archiving this many images? We probably have around $1000 tied up in our backup system between hard drives, toaster drive & a single external SATA case. We frequently get calls from clients looking for sports images, family photos, whatever from years ago. We'll sell somewhere between $500 and $1000 every year in archived images. So, yes.

I have images from about 13 years stored. I also have about 1000 film images that I have stored away that I should digitize. Last year I decided to make my archive/storage system match the century we live in. Before I did that, I went through a lot of images and deleted approximately 15,000 images. I probably could have deleted double that, but I just could not get myself to do so - hoping that the future technology could bring some of them back.
I figure when I am too old to hike, then I will work on processing them.

So now I have multiple harddrives storing all of my images. Including jpegs (some as small as 1mb), I have about 2 terabits of images - that equates to a lot of images. I have 90,000 images I took for work alone (those are all jpegs).

If I was to die tomorrow, most of those images would probably never see the light of day.
Maybe someone would try to revive the family images, but the others would most likely be found 50 yrs from now. Making me famous and rich long after I am dead.

Yes, more HDD's seems to be your best bet. Don't you miss film?
I think with that much data it's worth it to just get even more hard drives. To save time copying, you could use a RAID1 set up. In which it essentially creates two hard drives with the same data, for redundant redundancy. When you copy a photo, it'll go to both drives (automatically iirc), so in the event one fails, the other has an exact copy.

You could just use another hard drive. I have a Blu-Ray burner, those are 25GB per disc I believe. Still would take a while.

I don't know the age of your computer, but I think it would be painful to copy that much data to an external HDD w/o using a SATA connection. If your computer doesn't have an eSATA port, my personal recommendation would be to use an internal HDD with a SATA connection, and just remove it after you copy everything over. However, this assumes your computer has SATA ports (most modern ones do) and that one of them is open (or disconnect something like CD/DVD drive temporarily). They make empty external drive bays that you could put this raw drive into if you ever wanted to access the data as an external drive.

Not necessarily recommended, but depending on the RAID setup, you could break the RAID (remove the secondary drive), and put a new one in there. The existing data will remain on the secondary HDD (which could then become your off-site backup), and the data on your primary drive will be copied over to the new HDD automatically. Some software people are not fans of this method, and I certainly don't recommend doing this often, but as a one time thing (or maybe once a year), it can make things much simpler.

Minor side note on RAID that I meant to mention in my first post:

Be very careful with them, and say it with me:

RAID itself is NOT a backup.

RAID setups are designed to keep a system running through a drive failure, not backing up data. (With some configurations more worried with system performance.)

Simple disks with manual* copies tend to be a little more friendly for archival reasons, because they are slightly more resistant to data corruption, and often far easier to recover from errors. Minor corruptions in a RAID system can nuke the whole volume, and put your data in a state that becomes very hard to recover. While on the other hand, the chances of a complete failure of two or three simple drive volumes that have been sitting on a shelf for a few years is unlikely.

Simple volumes are also far easier to work with on a technical level. If your primary field isn't IT, and you don't have a very close friend who is in IT who you can count on to help you out, then I strongly suggest staying away from using RAID as part of your critical archive.

*Manual copies including scripts that automatically copy the data to multiple locations.


Also, if you are using RAID in your main computer, and believe this is protecting you: It isn't. RAID provides NO Protection against data deletion and over write. It will happily write over data due to command errors, let you delete all your critical data by accident with a smile on its face, and doesn't bat an eye if a virus infects your system.

Backups and Archives occur Off your main computer.

I disagree that you have to make hard decisions regarding erasing many of your images now - storage is very cheap, and gets cheaper probably faster than your rate of accumulating data. Just keep it reasonably organized.

Stay away from optical storage - DVD, BlueRay etc. They don't last more than a few years, and almost impossible to maintain. RAID is also not a solution. Things fail (like RAID cards0, and you might end up loosing data. (It happened in my organization a few times.) Multiple hard drives is much better approach, especially if one is offsite, and run on different platforms (Windows and Linux e.g.) Ideally, regularly do bit-to-bit comparison between archives (using hashes e.g.), to detect failing hard drives early (SMART technology is apparently not too smart after all). And don't do mirror - do full backups, as mirrors will not save data when there was data corruption, and will not help in case of accidental deleting data.

Here is the detailed description of my setup:

https://photography-on-the.net …p?p=14107332&po​stcount=37

pulsar123 wrote in post #14268957
I disagree that you have to make hard decisions regarding erasing many of your images now - storage is very cheap, and gets cheaper probably faster than your rate of accumulating data. Just keep it reasonably organized.

Stay away from optical storage - DVD, BlueRay etc. They don't last more than a few years, and almost impossible to maintain. RAID is also not a solution. Things fail (like RAID cards0, and you might end up loosing data. (It happened in my organization a few times.) Multiple hard drives is much better approach, especially if one is offsite, and run on different platforms (Windows and Linux e.g.) Ideally, regularly do bit-to-bit comparison between archives (using hashes e.g.), to detect failing hard drives early (SMART technology is apparently not too smart after all). And don't do mirror - do full backups, as mirrors will not save data when there was data corruption, and will not help in case of accidental deleting data.

Here is the detailed description of my setup:

https://photography-on-the.net …p?p=14107332&po​stcount=37

very good advice. Agreed and thanks for sharing.

I understand doing a quick pass through any set of images uploaded to delete the blank frames, completely out of focus and missed shots... but beyond that, storage is way too cheap to justify deleting most of the shots you won't be using. and you might get rid of something useful in the process. With 10,000+ images, you'd have a large number even if you reduced it to only the "keepers". If this number justifies storing on external hard drives, it's not a significant difference in price or time to just back up everything. Storage is super cheap now and a 2TB drive holds lots and lots and lots of pictures

Using the term "archive" is a bit misleading, what the OP is looking for is off-site backup. There's been some good advice around backing up your data mentioned already.

The requirements for archiving are different and hard drives aren't the solution for long-term storage.

Also don't forget that future post processing may be able to turn some of those currently
undesireable shots into good ones, with techniques like the anti-blur technology Adobe is
currently working on. And yest, storage costs are consantly dropping, so there is no
pressing need to eliminate hack and slash your way thru your archives. I know as an
amateur that takes most of his pictures while traveling, I still go back and look at photos
from trips many many years ago, and the still bring back many good memories. Even the
fuzzy and out of focus ones.

--Evan