How do you all store and guarantee the safety of your in-process and archived images?

[FlipsidE]

*Let me start this all off by saying that this may get fairly technical, and I apologize if it gets too techie for some to understand.*

I have been thinking about this very heavily as of late. I have started my migration to RAW versus JPG and very much enjoy it. I really think I'm going to continue in that direction, but only time will tell. Either way, after coming home from a photography workshop this past weekend with 1.6GB worth of photos, I am starting to see the need for LARGE amounts of storage.

Now I am not professional, so I don't have the same pressure that a lot of professional photographers have of making sure that images don't go anywhere.

Currently, I have a 120GB HDD on my File Server. This is where I store all of my photos. Now, 120GB seems like a lot of room until you realize that a single day woth of shooting renders 1.6GB of RAW files. Then, you have room needed for JPG's and possibly TIFF's for printing. So far, my folder containing workshop RAW, JPG, and TIFF photos is up to 2.28GB. Now, 120GB starts to look very small when you consider it might only hold 40 or so days/shoots worth of data (once you take away space needed for the Operating System as well as other files I store on my server).

So, let's say I up my storage and buy one of those nice, new, fast 250GB SATA drives. That's great. Now, I have over twice the amount of space which should last me for a while. But, once you get past a certain point, archiving all of that data gets to be a bit of a pain. Double Layer, 8.4GB DVD's make that a bit easier, but not much as it would still require backup, catalog, and storage of at least 18 DVD's for 150GB worth of images.

Now, if you have that process and ability to keep up with 18 DVD's, let's say, hypothetically, you have downloaded all your images from a shoot, lost track of how many you had cleaned up since the last archive, and are just about ready to burn a DVD. But, then your drive crashes. All that work is lost. Even if you have a backup of all the RAW data, all the time spent cleaning up and readying the photos for printing is now completely lost. So, how do you all prepare yourself for this situation? Are there any of you out there running something like dual 250GB HDD's in RAID 5? Would you recommend this?

I guess I should ask...with all the information I listed above, how do you make sure you don't lose any photos at all? Even the ones that may not be great but you might want to revisit at some later time?

Thanks

FlipsidE

[chris.bailey]

All I can do is say what I do.

I keep my pics on a seperate 250GB SATA Hard drive that backs up to a LaCie 500GB external hard drive every day, this also holds my system backups. I use drive Image for the backups.

I keep my pics in folders by year and then month with subfolders for RAW Inbox, JPG Inbox, Processed and then with sub folders under this for specific shoots so

Jan 2005
- RAW Inbox
- Studio 15th
- Studio 25th
- Ketts Park 28th

etc etc

At the end of the month I will make a backup onto DVD of that months work and that goes into my firesafe at work.

My 250gb work drive is now getting fairly full so I may end up going up to 2 x 250gb SATA's non Raid and 2 x 500gb externals as backups. Ill then just leave up till 2004 on the first one and then start again.

I would not go to RAID as the fault tolerance is more easily provided by backing up to other hard drives.

What I do though is prune my RAW folders so your 1.6gb will probably be halved when you get critical of what are keepers and what are not, probably less.

I reckon my system is pretty safe as I have at least two copies and normally three of everything I do and it is hardly time intensive. Big hard drives are now dead cheap and the new external firewire 400 drives pretty quick.

[FlipsidE]

Wow, Chris. Excellent advice! Thanks!

FlipsidE

[steven]

I take my raw files and write them to DVD+/-Rs twice and keep on off site.
This is in addition to the copy that resides on harddrive. So I have 3 copies.
There is an issue of having to check the DVDs to make sure they still can be read at a later date but this is the price of digital. (almost like storing slides;)

[Longwatcher]

My solution is similiar.
First after loading to HD from card; at end of the session I make an archive copy to DVD+R, I used to use CD, but started getting carried away. Then after that is accomplished I edit my pictures on the computer and when done I make another DVD to include the edited pictures. Then sometime later I delete the files from my computer. At this point I have 2 copies.

At some future point as I am doing now with CDs to DVD+R, I will copy the data to the new format. I still keep the original CD or now DVD as another backup.

As of now I am not keeping additional copies off site, but then I have not been doing sessions that were critical if I lost the set to anybody but myself. Although I seem to remember one exception where I kept the archive DVD in my car until the job was done.

My problem is in a single session I usually take 6GB of data and have one where I took over 30GB. a 250GB drive gets filled up fast that way, especially if you add video as well.

Just my experience,

[lost]

Ok I have little experience in photography but TONS of experience in Data Backup and storage.

Raid

First let me point out that in order to run RAID 5 you would need 3 Drives of similar size, not 2 as you stated. You could use Raid 1 (Mirroring) with 2 drives. Raid 5 provides a slight advantage in speed over Mirroring and gives you a higher percentage of usable disk space. Unless your mobo supports raid I would suggest an add-in Raid card which you could get fairly cheap. I would not suggest using Software Raid (OS provided). I could go into a LOT of detail about RAID but I think this should get you started.

Backups

Here you have a lot of options, from External Hard Drives to DVD to Tape. DVD seem to be the most cost effective options these days but requires you to be there to swap out Disks.

STATIC DATA : With Static data like your RAW files and finished PS pictures DVD is perfect. You can backup once with out the fear of not having your most current copy in your archives.

Dynamic DATA : With dynamic date (EX. Photoshop settings, Word & Excel files, OS files) a DAILY backup procedure is the best Idea. This is something that needs to be automated and not require your attention unless of course you need to restore files. An External Hard Drive would probably be the best bet. There are several different types of software to do this for you. (Use Google) Keep in mind how far back you would want to restore from. I would suggest a scheme that would allow you to do a Full Backup once a week and daily incremental backups.

I would suggest storing Static Data in a seperate location or partition to keep from taking up space in your Dynamic backups.

If you want to become paranoid you would also want to keep duplicates of above in an off-site location. There are also some websites offering secure file sorage for backup purposes but I have no experience with those.

This comes from a Net Admin with years of Experience in Data Backup. Please let me know if I am missing something particular to Digital Photos.

[CyberDyneSystems]

I've given this considerable thought,. after having a Hard drive fgo down and loosing several years of data (fortunately NOT my pictures!) I tend to take this all very seriously.

The trouble is we tend to be LAZY
So a combination of totally automated backup is needed in addition to hands on CD-DVD burning.

Therefore ,. I use a Three Pronged approach

I. STAGE ONE: RAID 1
There was a time when speed was of utmost importance to me and all my drives were striped RAID... for speed. Now my data is too important to me.

Step one is to get all your images onto a different drive, or at the very least a seperate partition , then all your programs and stuff are on the C: drive, and all your data is safe and sound elsewhere on D: or E: ...
...now WHEN Windows takes a dump,. you have no fear if you want to format C: or recover from a "Drive Image" you only mess with C: and all your data is safe and sound on the untouched E: drive. You shoulod do this no matter what direction you go,.

YOUR DATA SHOULD NEVER BE ON THE C: DRIVE!

I suggest a matched pair of Hard drives for your images and data ONLY. Use these drives in RAID-1 "mirrored" array. I use a Pair of 160GB drives .. (which I am very soon going to either out grow,. or I am going to have to move some data off line.)

RAID-1 means two drives have the exact same data written to them simultaeneously. If you use Hardware RAID (recomended) there is virtually no perfromance hit.

RAID-1 is not a cure all though, and should not be assumed as a perfect solution on it's own. It's biggest advantage is protection from a real life hard drive mechanical failure. If one drive bites the dust,. the other drive should be fine with all the data in tact.

RAID 1's biggest advantage? You don't ever have to think about it,.. it is allways happening and allways giving you a redundant copy of every file you make.

II. STAGE TWO: AUTOMATED EXTERNAL BACKUP
Fun with Mirror Folder
The second part involves an external Hard drive that is not powered on all the time,. and automated software. The external drive is unimportant for the specifics other than it is best to have a drive larger than the drive you are copying from.

I keep my external drive "off" and turn it on every few days for an over night back up,. I don't have to initiate this back up,. it is automated,. but I do like to keep this back up drive turned off to keep it safe most of the time.

The software may be something that comes with your external drive, but I HIGHLY recomend you all look into a program called "Mirror Folder"

Mirror Folder is a combination back up program and software RAID program that will function with internal, external and network drives.

It allows you to chose individual folders in your directory structure to be treated in different ways.
You can for instance choose to have your "My Documents" folder "mirrored" to an external drive in "Real Time" (ie: software RAID) while ignoring all the other crap on that particular hard drive. You can have it Mirror this data to several locations at once. At the same time you can have it mirror all your images in your "Graphics" folder to an external drive,. either real time,. or as I do it,. every 4 days, etc.

It will also archive only incremental changes as they are made.

You can choose a network drive as well,. for instance,. here at work the "file server" is really little more than a glorified back up solution,. all the individual PC's have Mirror folder running on them and all the data is copied to the "server" twice daily. The server is RAID-1 and has the only DVD-Burner in the office. A cheap solution that works well.

Mirror Folder goes a loooong way to making back ups from drive to drive totally hassle free and has enough options and flexibility that you can esily configure it to do exactly what you want it to do and ONLY what you want it to do.

III. STAGE THREE: Hands On DVD-Burning
Face it,. you have to do it. All of the above will offer some peice of mind for those of us who are too busy with other things to do this religously,.. but we can not let on line hard drives be our only repository for our hard earned data. Get a fast DVD burner and do it at least once a month. Make two copies,. check them,. and then get some copies to work our your sisters house.. anywhere off site that will be safe.


Lastly,. for a directory of my images I do pretty much Chris does,.. I date all the folders,. and or name the specific shoot,.
1-12-05 Scheherazade

Usually a "shoot" is at least one DVD,..(or as many as five!!) ...but if no big shoots occur I can often fit a months worth of "plinking" onto a DVD.

[lost]

There are a few things that need to be stated that have been skipped.

First, The reason RAID is not enough is that if the Raid Controller or MOBO (Depending on which is doing your raid for you) dies you will be up a creek. With some high end boards you can get around this but doubtful for most consumer models. Also other factors that might affect both drives(power, environmental)

Second, The Mirror Folder software does not protect the OS at ALL. If you have an external hard drive you probably have some software for this if not, google can point you to all the options you could want. Stick to some of the better name brands.

Third, You should not be backing up static(non-changing) files with you normal routine. This only makes the process longer and take up more space. So a good practice would be to keep your finished work in a seperate place from projects you are working on.

Fourth, FlipsidE if you are using a File server the mirror software would help you getting the software to the server for backup and speed up opening files.

Fifth, The only burning I do is my raw files immediately after transfer to my computer and My Finished work once a week.

I have more to say but it will have to wait until later tonight sorry.

[FlipsidE]

CDS,

Man, I gotta say, you think almost exactly like I do. I just wonder, honestly, how many other people have gone to such lengths as you have (and hopefully I will) to make sure their data doesn't go away on them. For the most part, a daily DVD backup will work. But, what happens if you are in the middle of editing a group of photos, and your drive just dies on you? Well, however many hours of work that was is now completely gone. Your RAW data is still in tact (be in on a card or cards or on a pre-edit DVD), but you've lost something just as valuable, IMO...your time. Time has got to be one of the most important things to me, and IMO, my time (even my free time) is worth money to me. So, if I lose a few hours of photo editing, I lose money too. Think of it as lowering your billable rate if you will.

I'm nowhere near professional level yet...nowhere near. But, I must say, photos, word documents, music, etc...you name it. All of it takes time and sometimes actual money to replace. And, IMO, it's worth the time and money upfront to set up some kind of backup scheme like you have.

I really appreciate you writing this, CDS. I know it took a good bit of time to go into that much depth, but it really is great advice! I will most definitely keep all this in mind as I decide on my backup scheme!

Thanks again!

FlipsidE

[FlipsidE]

lost,

I agree that there are many things that can go wrong with a RAID setup. And, I know it's only a way to really guarantee more availability and not guarantee that your data is still around. The only reason I really plan to implement RAID is so that I can feel more comfortable while editing 150 images in a sitting knowing that if one disk just bites the dust, the other one will take over and keep right on going storing all data and all changes that have been made as well as new changes that are made after the drive crash. It's only for those few hours that I would count on RAID to save me. If the mobo was to die, though, then you're in a lot deeper trouble that something RAID could easily pull you out of.

I think I would prefer to use an external file sever or disk/disk array to backup my files fairly often. That would be a tad more stable, but a tad less safe than RAID overall. Still, having almost all your data on three disks versus one gives a much better chance to salvage something if something does go wrong.

I must say, I am so impressed with all the great info all you guys are sharing in this thread. This is far more than what I had hoped for! Thanks so much!

FlipsidE

[FlipsidE]

On another note along the same subject, I think RAID 1 would come in very handy for me. Why? Well, I love photography. I enjoy setting up for it. I enjoy working with models and meeting new people. I enjoy shooting. I enjoy the complexity that goes into this hobby. But, I absolutely loathe post processing.

For some, post processing is their favorite part. And, that is great. There are some people out there that have some amazing skill in photoshop, and it is obvious that they truly enjoy it by how much time it took to come up with end results. For me, I don't know. Maybe it's the fact that I just haven't been trained at all in photoshop. Maybe it's the fact that I honestly don't know how to make the right tweak here or there to get that photo looking just right. Or, maybe, in the end, post processing just isn't for me. Whatever it is, I just cannot stand post processing. So, I would prefer to spend absolutely as little time as is possible to get the post processing part done.

If I'm 90% of the way through post processing 150 photos for personal use, and my disk dies on me w/out any kind of RAID, I would almost guarantee that these photos either wouldn't ever see the light of day, or it would be a LONG while before I sat down and tried to edit them again. That or I would just get prints made w/out edits and live with any mistakes/blemishes might be there.

So, IMO, of all these backup techniques listed...RAID is the one I'm most interested in. The 3 hours during an intense post processing session would be the part where I'd depend on RAID to keep me safe. A backup taken before and after a session like this is easy enough. But, to lose three hours of time because I didn't take precautions to make sure I had at least one disk available is something I couldn't forgive myself for for some time.

FlipsidE

[Avalonthas]

Well I usually have about 1.8 to 2.5 gigabytes of shots after a shoot, and then its just usually under 4 gigs once I am done with the jpegs and tiff files. So after all the post processing is done, I back all the stuff on a single DVD then stick it on my shelf. Each day gets one DVD even if there is space left. I do this for easy storage and easy retrieval, and since DVD's and cases are so cheap these days, it only costs 50 cents per backup DVD, which is quite acceptable (And usually I slip a 4x6 or two in the covers to give me an idea of what the images are like inside, which i print on my canon i9900).

Then I make a secondary backup and send it to one of my two 500gig hardrives which are strictly for storage. They each costed me 150 bucks from a friend (i got no idea where he got em from/nor am i obliged to ask lol). Right now I have used about 120 gigs/1000gigs since i went Digital so I probably have enough storage to last me 1.5 to two years. As for my actual computer I have a single 74gig Western Digital Raptor drive which I use primarily because its rated at 10,000 RPMS so it super fast. Once I double backup my stuff, i delete it off my main HDD for future stuff. Easy

[cactusclay]

Wow, you guys must really have a lot of keepers, to need so much storage. 98% of my stuff gets tossed in the name of mediocraty.

[CyberDyneSystems]

I wasn't expecting a point counterpoint on this,. :) :)

but I think I need to jump in where Lost left off on some of these points.

1. To the best of my knowledge,. no RAID 1 array is reliant on the host controller for data integrity (unike RAID 0 or RAID 5) ie: if the Mobo or host controller dies on a RAID 1 set up,. attaching one of the drives to any controller of the same interface,.(ATA, SATA etc..) will give you access to your data. Perhaps I am unclear what Lost's point was,. but if this was it,. then I am quite sure he is mistaken.

However,. I did make it clear in my initial post regarding the USE of RAID 1 that it serves only one purpose and that is hopefully protection from a single hard drive failing mechanically. It can not protect your data from many problems that can occur, (virus corruption, user error etc.. in these cases for example both copies will be corrupted at the same time) ..so it is not to be trusted alone. Thus,. it is only one of my three pronged approach. However,. it is certainly not to be dismissed,. like any tool it serves it's use.

2. Mirror folder and OS recovery: Mirror folder CAN be set to mirror entire drives,. including the C: drive,. and the OS.. if you so chose. But I agree there are far better ways to protect the OS,. (I prefer drive Image as well) than using a real time solution, which is what Miror folder is best at.

So no,. I would not recomend Mirror Folder for an OS back up. I was quite clear in my original post that the methods described were applicable to protecting DATA and that this was my only concern in this thread. The first point I made involved seperating ones data from the OS and apps to make this task easier and more controllable. I did not go into any of the steps that I take or would recomend for preserving a clean OS install,. as it has no bearing in this thread. Prior to using Mirror Folder (in fact still to this day) I used simple home written batch files for all of my data back ups. Mirror Folder is the first tool I have found that lets me chose with such complete flexibility how and what I back up to a point where my own batch files are no longer needed.

3: All of this is selectable with most any back up software.

Four and Five,. I agree with you completely :)

The arguments about whether RAID of any type is of use to "the averadge user" seem to go on all over the net in computer forums,. I don't see a point in revisiting it much here other than to say that with the controllers now installed on most Mobos,. and/or very affordable solutions available,. and 160GB hard drives selling for Under $100.00 regularly,. it is now a very inexpensive option to add one more level of "peice of mind" to your beloved data.

[Dante King]

Wow. I take the low tech approach. I take my raw files and copy them to DVD with a thumbnail page of its contents. The DVD and Page are put in a binder for storage and further reference. I have about 500 gigs of stoarage on my PC but it gets eaten up real quick. This way I can save money by not buying raids and more HD space and save up for more camera equiptment!!! :lol:

[lost]

Sorry CDS my intention was not to contradict you but to try to help Flip find the right solution for his situation. Anyway discussion always (usually) leads to a better understanding for all parties. I hope Flip was able to find a solution that he feels confortable with and wont keep him constantly busy doing Backups:D


I think Flip ended up with more "advise" than he ever expected. :lol:

[CyberDyneSystems]

Too true!

Actually,. despite all our heavy tech in this thread,. this one and a few others recently popping up in the forum regarding "handling and cataloguing" our precious data files is leading me to think that we made need a new informational "sticky" thread dedicated to the subject and all it entails from this hardware stuff,. all the way to software for cataloging and organizing etc...

What do you think?

[lost]

I think a sticky might be in order. Most people have no idea how fragile their files are, and how important the discussed safeguards are.


BTW.. My coment about Raid controller failure was (supposed to be) in reference to Raid 5.