Archiving RAW Files

I am seeking info on what others do to archive RAW files. Going from 8MP to 10MP and now 15MP with the 50D has present space complications. I have been saving on multiple drives but am interested in using DVD's also. My problem is my computer (or possibly just the DVD drive) doesn't seem to like RAW's. The last several attempts have given error codes and not all files were valid. The drive seems to be working OK-plays CD and DVD disks, burns CDs.

Any ideas?

Burning DVDs is slow and labor intensive. You can get a 1TB external hard drive for less than $100. That works out to be slightly more expensive that DVDs if you buy them on sale in quantities of 100, but you will spend a lot less of your time doing the copying.

I use the Adobe DNG converter to convert all my RAW files, then copy both the RAW and DNG to a 750GB external drive, than make a second copy on a 1TB external drive, and leave the originals on my internal drive. Haven't figured out yet if that is the best strategy (or even a good one), but at least I have multiple copies of all images and, I hope, am somewhat future-proofed.

Hard disks are cheap. Put a 2TB drive in your PC, and get a 2TB drive to keep offsite as a backup. All drives fail, the only question is when. I keep three copies, one CR2, one DNG, one jpeg, spread across different disks and locations. If I edit an image I keep the PSD too.

DVDs are expensive, burning is slow and manual, and the reliability is poor.

Storage is cheap. Get larger drives.

Try reading through this:

http://www.dpbestflow.​org/links/39

Kirk

I outgrew and stopped using DVDs for storage a while ago. They are a real hassle to burn and store.

Big hard drives have come down dramatically in price, so there's little reason not to go with them instead of DVDs.

The first thing I'd look into is either replacing smaller drive(s) with bigger, and/or adding a second (or third or whatever) hard drive to your computer, assuming it's a desktop. Most have room for at least one additional drive. Some have room for many more.

My primary desktop has a 1TB drive for softwares, general purpose and a partition for Photoshop to use as a scratch disk. There's a second 1TB data drive used strictly for recent image storage.

If yours is a fairly recent computer, with SATA connectivity, it's very simple to install a drive and get it set up. And you can get 1 or 2TB drives pretty inexpensively. I recommend using "enterprise class" hard drives, in particular. They are a little more expensive, but typically have five year warranties and are designed with large, server applications in mind, where they run 24/7/365. They consume less power and are better built to give long, long service life. Both Western Digital and Seagate offer these types of drives. I'm sure some others do, too.

On location I use a laptop with a 750GB external drive, which attaches with USB or eSATA. It's convenient, but not quite as fast to write to and retrieve from when using USB connectivity, compared to an internal drive.

Now for backup current work and archiving previous years' images, I currently use three Netgear NAS servers on a simple, wired network. Each contains four 1TB drives, but using RAID X they only provide for 2.7TB storage space per server. But it's worth it since each server also backs itself up. I can replace any individual disk in the server if it crashes, and the server will automatically rebuild the new disk without any data loss.

I'm going to be adding a hot swap tray to my desk top, so that I can use 1 and 2TB drives for incremental backups to be stored off site. An off site, secondary backup might be important if I ever have a disaster wipe out stuff on site, such as a fire or flood or theft.

Chances are the "corrupted" files are fine, but you just have way too much on the hard disk. Once you get about 90% full, things slow down and errors start to happen. Move stuff off the disk and see if that helps. Hopefully the images are fine. You might need to run an image rescue/repair software on them... such as Lexar and Sandisk sometimes provide.

Another common cause of RAW file corruption is when transferring them. I had issues with a memory card reader causing occasional corrupted files when the reader was attached to a USB hub. It works flawlessly when connected directly to the computer, though.

I've gotten more ruthless about editing, as camera resolution has increased and image file sizes have grown and grown. Currently using a pair of 7D and 5DII, so I fill up drives and NAS servers pretty fast!

Is there any loss in dynamic range, wb, color when converting CR2/NEF to DNG?

I just tried some and they're ~19% smaller file sizes (this was NEF to DNG)

I would be concerned about putting all your marbles in the machine. Hard drive are mechanical and will wear out and they do not necessarily give you notice. I would never consider putting all my pics on either a main or backup drive in my computer without a separate copy somewhere else. My preference is a separate hard drive housing with a eSATA interface so I can quickly copy the files over to this drive and then remove and power down that drive from the system. I do not erase the pics from the CF cards until I have two distinct separate copies !) one for editing and 2) one stored externally to my computer. I feel comfortable with that policy.

Anyone here use dual layer blu-ray discs for archival? The price is down to about $9-$10/disc, not too bad for 50GB of storage. I'm curious how long it takes to burn that much data.

the thing with pressed disks is that they can go bad even in a good environment while hard drives are sealed (and faster)

ok, this can sound difficult... but here is my .02

i use the non-commercial version of Logmein Hamachi. ----"which is free"

and create a network from my Home PC, to my Workstation @ work.

Share your pictures folders, on both machines...

then use SyncToy from Micro$oft ---"which is free too", and it will assign a "master folder" and the other an EXACT mirror of the other...

---------------

or you could just buy one of these...

Network Storage Unit

and just slap two drives in it, set it to mirror the disks and let it manage the whole thing... it also has an email alert system to let you know if a drive is failing.

danpass wrote in post #11306686
Is there any loss in dynamic range, wb, color when converting CR2/NEF to DNG?

I just tried some and they're ~19% smaller file sizes (this was NEF to DNG)

It's the exact same data with lossless compression. It's smaller as a computer has more time and CPU power than the chip inside the camera, which compresses but quickly. So DNG = CR2 + metadata.

juiceman72 wrote in post #11306804
Anyone here use dual layer blu-ray discs for archival? The price is down to about $9-$10/disc, not too bad for 50GB of storage. I'm curious how long it takes to burn that much data.

Takes about 3 hours. PITA, but I go from camera to 1TB USB drive, then build of batches of 50GB to burn at a time. The real pain is catching up to what you've already shot, and burning the existing images. I had 400GB to burn before I was fully caught up. Time consuming, even if you have a dedicated PC for the burning.

I love the idea of keeping multiple large USB drives around, even 2TB drives are getting reasonably cheap. Problem is, I can't realistically expect that drive to be 100% functional 5-10 years from now. I can burn a disc once, sleeve it up and store it away, and expect it will be there (and readable) even 15-20+ years from now.

Depends if you're interested in long-term storage, or short-medium term storage.

Make 2 backups of your files to hard disk.
Copy 1, should be an external drive connected to your computer, either daily or weekly depending on how much work you do. This drive you can leave connected so your backups are more convenient.
Copy 2, should be an external drive, either monthly or weekly, and that drive should be disconnected and put in a closet or safe deposit box...
Those 2 copies are in addition to the originals you have on your computer.

I don't agree with having external disks permanently connected. A virus or stupid user can delete them. It also means another power supply plugged in, which I don't like. You'd be better off with an internal mirror, which has no power supply and is much much faster on sata.

I have one offsite backup of my working files, which I consider secure enough. Once i'm done with a customer file it's archived to a disk which sits in a drawer, and is also copied to my offsite archive.

In case anyone's wondering yes that means I have two offsite disks - mirror of my active drives, and a mirror of my archive drive. Both are eSata. The archive mirror is DNG/jpeg, the archive drive at home is CR2/jpeg. Just hedging my bets really.