Backups – A change in Workflow

[Dans_D60]

Requirements for safe and secure storage for images have grown even beyond my expectations. Currently I need over 100 gig for image storage alone. Last weekend I decided a change was necessary to meet my workflow needs:

- Long term safe storage for images
- Short term easy accessible safe storage for images
- Redundant storage for both long term and short term storage.

So here’s my latest technical solution:

Primary workstation:
- Windows XP PRO, 2 GHz Pent IV, 1 Gig memory, on UPS
- 160 gig internal disk drive
- Two (2) identical 60 Gig external USB 2.0 disk drives on separate UPS
- DVD/RW +R, -R
Network server and firewall internet access
- Windows 2K Pro, 800 MHz Pent III, 320 Meg memory on UPS
- Maxtor ATA RAID 1 controller
- Two (2) Maxtor 200 Gig disk drives

The Primary workstation was purchased from Gateway about a year ago. I added the 1 gig of memory and the SONY DVD RW drive. Last weekend I added the two 60 Gig external USB disk drives. Good deal from CpmpUSA. The USB disk drives are a net $49 each after three rebate offers. Good for one week only: http://www.compusa.com/products/product_info.asp?product_code=306325&pfp=BROWSE

Workflow:
- Breeze Downloader Pro reads CF cards using a 2.0 USB card reader
- Saves RAW images in a new automatically date created directory on the first USB external disk drive
- When Download Pro exits, I have a VB script that will automatically copy the new folder to the second USB external drive
- I now have a redundant mirror image of the RAW files. Proceed to C1, Photoshop, InDesign, etc. using the first USB external drive
- Nightly, Retrospect Backup software executes and proceeds to copy any new files from USB drive one to USB drive two
- Periodically (once a month) I purge older directories from the external drives to DVDs and to the Win 2K server in the garage. The server has RAID 1 so the copying is automatically on mirrored drives

Appears to be an absurdly complex and expensive solution to image storage. Actually it’s not so bad. The Win 2K server was a hand-me-down and costs me very little to maintain. It’s slow by today’s standards but only used for internet firewall access and mass storage. Using Downloader Pro will copy in a one-click operation. On average I download over 500 images from one photo session so this one click operation is great. The key are these inexpensive ($49) external disk drives. They are external and on their own UPS that is isolated from any computer glitches or power issues. I did use an external storage service on the internet for a while to save disk images, but the time and cost made this prohibitive.

Any other ideas or suggestions on backups are welcome. It was easy when my storage needs were less than a few gig, but now with thousands of images … the rules have changed!

Dan
http://www.pettusphoto.com

[RichardtheSane]

My solution that I will be implementing shortly is a little different.
I plan on getting 3x 160Gb serial ATA hard drives and setting up a raid 3 array, which would give me 320Gb of storage and also provide parity chhecking so should a drive fail then the data can be rebuilt onto the other drives.
This would be purly for data storage, and the operating system would run on my current Ultra ATA raid array - 5 hard disks in total, but all internal to my machine.
I do like the idea of your vbscrips that copies the files across after downloading - I may try to do something like that.
I also have a DVD-RW drive which I am in the process of backing up my images to.
:)

[iwatkins]

Do you both keep every image you take ?

I on average keep maybe three or four shots out of thirty taken, all the rest get deleted. Of course, sometimes I keep none, others times I'll keep nearly all of them.

I'll do all my post processing, touching up etc. and then keep the original RAW and the resultant, post processed TIFF. These get placed into a directory on my machine for keeps. Once a day this directory is mirrored onto a DVD-RW via a batch job.

I'm only only my second DVD-RW.

Cheers

Ian

[Dans_D60]

iwatkins wrote:
Do you both keep every image you take ?

I on average keep maybe three or four shots out of thirty taken, all the rest get deleted. Of course, sometimes I keep none, others times I'll keep nearly all of them.

I'll do all my post processing, touching up etc. and then keep the original RAW and the resultant, post processed TIFF. These get placed into a directory on my machine for keeps. Once a day this directory is mirrored onto a DVD-RW via a batch job.

I'm only only my second DVD-RW.

Cheers

Ian

Ian:

I keep every image just like I do with film or slides. I recognize that a few are obviously trash. Nevertheless, I find myself still incorporating slides from 30+ years ago that I thought would never make use of at the time. It’s actually an easier workflow to save everything then “tag” and sort selected images for post processing.

Dan
http://www.pettusphoto.com

[OviV]

I am strictly an amateur but would hate to lose my images. I am currently storing them on a file server with RAID. I was thinking of archiving them to CDRs. Would this be cosidered safe? Should I burn two copies of the CDRs for additional security?

Regards,
ovi

[robertwgross]

Currently there has been discussion about the inferior nature of some brands and types of CDR disks. Apparently some go bad after a while, although I have never had that problem. I have not heard any similar discussion about DVD disks.

When I have RAW files in CF cards, I copy those from the reader to one hard disk directory. Then I use the utility to convert them into TIF files at another directory.

Whenever the size of the first directory exceeds 800MB or so, I copy one CDR of them and set that aside, then erase the hard disk RAW files that are done. Blank CDR disks cost about twenty cents.

Meanwhile, I have the converted TIF files that have to be worked over, edited, scrapped, whatever. Eventually I have a bunch in both 48-bit and 24-bit versions. I copy all of those from the first hard disk to a second hard disk. Then, on the first hard disk, I delete the (big) 48-bit versions of the files.

Then when I have 4 GB of those 24-bit TIFs, I copy them out to a DVD+RW disk and set that aside. Blanks cost several bucks each.

After careful scrutiny, I choose the marketable ones and copy them to other places. So, I end up with RAW one place, TIF in two places, and then TIF and sometimes JPEG in other places. I've never had any complete loss of an image file.

---Bob Gross---

[MrPogo]

I keep absolutely everything I take, unless there's something badly wrong with it (completely out of focus, whatever I was trying to photograph was moving too fast and was half out of the frame by the time I pressed the shutter etc), as you never know when you could wish you'd kept the shot.

[Rob Larsen]

Following is the workflow I'm using. It definitely has room for improvement, but it is working for my amateur shooting volume.

1.) Download from CF in card reader with Downloader Pro. Automatically create directory called DOWNLOADDATE-JOBID in my "originals" filesystem (\ORG). Filename is DATETAKEN-PIC#.CRW.

2.) Job automatically runs and creates duplicate directory structure under \PROJECTS on second internal HDD and creates the following new subdirectories:

\PROJECTS\yyyymmdd-jobID\ORG
\PROJECTS\yyyymmdd-jobID\WIP (work in progress)
\PROJECTS\yyyymmdd-jobID\Final
\PROJECTS\yyyymmdd-jobID\Web

3.) Start C1 to screen pictures. Selected are converted to TIF and saved in the \ORG directory.

4.) Do PS edits if necessary and save with layers in \WIP and completed flattened images in \FINAL. Sometimes like to go back to \WIP and tweek in alternative ways and save different version in \FINAL. Reduced web-bound versions of selected images are stored in \WEB. TIFs are also moved to \Final when PS editing is not required (much more often now with C1 DSLR!)

Backup:
1.) Perform periodic Full backups to DVD-RW on both "originals" and "projects" filesystems.

2.) Perform scheduled nightly incremental backups until backup consumes greater than one DVD capacity -- then perform new full backup and start incremental set over.

3.) Annually (or other schedule depending on storage constraints or my assessment of how much "I'd really hate to lose these" ), perform 2 full backups to DVD-R and store one copy at my in-laws house. Move all backed up files to my 200GB 3rd HDD into an "archive" filesystem. Will purge these files as needed for storage. Nice to keep the old files handy until necessary to delete.


Future plans are to implement RAID as another form of security. I like the CompUSA price on those external hard drives and you now have me thinking about what benefits integrating those into my workflow can bring me. Besides speeding up my incremental backup process, they could prove nice as general portable storage. Thanks for the suggestion!

And, I always keep all original files (except gross errors) and only convert those RAW images I intend to use.

[Leigh]

I currently back-up to DVD-RAM, double sided 9.4 GB disks, with an Iomega Super DVD Writer (all DVD formats), and Iomega's Automatic Backup software.

If / when I choose to go for larger storage, it would likely be with Maxtor's 300 GB, USB-2, External Hard Drive, which includes Maxtor's Backup system, that operates with a "push" on a button-- on the unit.
( The Maxtor's costs about $300)

Leigh