a serious long-term data storage

[jevidon]

So here's the situation. I have multiple external HD's that have multiple copies of my pictures, but I'm getting tired of unplugging and re-plugging in my various USB drives. What I'd like to do is have a dedicated RAID system and ideally have that be separate from my main CPU. Is there a stand-alone RAID storage system that's out there on the market, or do I need to set up a new box just for storage?

[Wilt]

You should be made aware that it is possible for a single point of failure to destroy data on ALL DRIVES contained within the RAID...you simply need to lose the cooling fan on the cabinet that houses all the drives. It HAS HAPPENED to someone, and they lost all their data in the RAID. Only the fact that the data was also backed up off site saved his bacon!

[SimonG]

... or do I need to set up a new box just for storage?
IMHO, that's your best option. As Wilt pointed out, RAID is not a backup solution, so you'd still be facing your current problem. With a separate backup box, you could maintain a mirror of all your local files relatively easily. There are RAID boxes on the market, but most of them are fairly cramped internally, with poor cooling, and that's a bad thing with large amounts of data on the line.

Another option would be to set up your current drives so that you do not need to un-plug them all the time. A fairly inexpensive USB 2 controller card and a power bar would mean all you have to do is flip switches on the drives themselves. Alternatively, you could look into the Saturn Drive Cartridge Solution (http://www.shopaddonics.com/items.asp?CartId={7C9D48EVERESTA4-596B-4E30-A706-26B7784A5FFB}&Cc=SDCS&iTpStatus=1&Tp=) by Addonics, which is essentially a hot swap tray for external USB drives.

[tommykjensen]

Check out these RAID solutions

http://www.stardom.com.tw/web/pro-sohoraid.htm

I have a SR3600.

And previous posters say. You still need a backup. I have tried losing 2 drives running in mirror raid at once. Fortunately I still had backup. And I regularly backs up my SR3600 box.

[225719]

uhh... A raid 10 is very secure, best back up in the world as far as I know. I'm running a 4 drive raid 0 of raptors and never had any problems and my raid is 4 times likly to fail then a normal raid.

[SimonG]

uhh... A raid 10 is very secure, best back up in the world as far as I know. ...
First of all, a redundant storage solution is not a backup... anyone that would treat it as such deserves to lose all of their data. Keep in mind that even with RAID10 there's typically a single point of failure that can bring the whole array crashing down: the controller card itself (nevermind a faulty PSU that starts taking drives down with it, a virus, user error, etc).
... I'm running a 4 drive raid 0 of raptors and never had any problems and my raid is 4 times likly to fail then a normal raid.
I think that you mean AID0... methinks you should knock on wood before something happens eh. :razz:

[225719]

well, theres no such thing as a permanant storage solution.. cd's lose there dye, same for dvd's... Raid 1 and above are the most secure. And while a psu can fry a rail and posibly kill the drive (s) whats the ods of that? About the same a a meteor striking your house and destroying your dvd archive... Raid cards them selfs can to a degree destroy a raid but it could be rebuilt.

What would you recomend to use? Solid state?

[tommykjensen]

well, theres no such thing as a permanant storage solution.. cd's lose there dye, same for dvd's... Raid 1 and above are the most secure. And while a psu can fry a rail and posibly kill the drive (s) whats the ods of that?

Well it has happened to me. I have lost 2 harddisks that were running in mirror. They died both at the same time don't know exactly what caused it but it happened so don't say it won't happen. I guess I should watch out for meteors.

[Jon]

well, theres no such thing as a permanant storage solution.. cd's lose there dye, same for dvd's... Raid 1 and above are the most secure. And while a psu can fry a rail and posibly kill the drive (s) whats the ods of that? About the same a a meteor striking your house and destroying your dvd archive... Raid cards them selfs can to a degree destroy a raid but it could be rebuilt.

What would you recomend to use? Solid state?Raid only protects you against the drives failing. It's worthless if you accidentally delete a file (or worse, a directory), get hit by a virus/worm, or if the computer is stolen or destroyed. That's where the off-site backup comes into play.

[SimonG]

well, theres no such thing as a permanant storage solution..
Precisely. That's why it's important to have a robust storage solution in addition to a tested backup program that includes offsite storage of some sort. Combining the various strengths while minimizing the weaknesses of the various media available is crucial. If one is going to be serious about it, that's what's required. Otherwise, you are playing the odds game, which is fine also, so long as you understand the risks and are willing to deal with bumps along the way.
... Raid cards them selfs can to a degree destroy a raid but it could be rebuilt.
Well, if your card tanks, and you do not have / are not able to obtain a replacement card of the same make, etc. there's no guarantee that you'll be able to access anything. That's because there's no standardized method of setting up the parity in a RAID5 volume, for instance; each manufacturer will have a slightly different implementation. That's a pretty severe single point of failure if you ask me.

[wilflee]

RAIDing makes sense for data safety. However, hard drives are hardly optimal long term storage medium as hard drives' main strength is high speed random access. There are too many moving mechanical parts inside to make it suitable for long term storage. And the exposed disc platters will also corrode over time leading to data loss.

For serious long term storage, consider using RAIDed tape drives. You can use 2 tape drives for mirroring or 3 tape drives for RAID 5. Dedicated hardware controllers will perform the RAIDing for you so you can backup at full speed and tape changers can help you manage tape sets that increase storage capacity into to the thousands of terabytes range.

But to reduce costs for long term storage, even a simple tape drive will be more durable than hard drive. You can do manual mirroring by simply backing up data twice. And as your storage needs grow, just get more tapes.

[jevidon]

RAIDing makes sense for data safety. However, hard drives are hardly optimal long term storage medium as hard drives' main strength is high speed random access. There are too many moving mechanical parts inside to make it suitable for long term storage. And the exposed disc platters will also corrode over time leading to data loss.

For serious long term storage, consider using RAIDed tape drives. You can use 2 tape drives for mirroring or 3 tape drives for RAID 5. Dedicated hardware controllers will perform the RAIDing for you so you can backup at full speed and tape changers can help you manage tape sets that increase storage capacity into to the thousands of terabytes range.

But to reduce costs for long term storage, even a simple tape drive will be more durable than hard drive. You can do manual mirroring by simply backing up data twice. And as your storage needs grow, just get more tapes.

sounds like the most logical idea. any tape drives you recommend?

[Bakewell]

RAIDing makes sense for data safety. However, hard drives are hardly optimal long term storage medium as hard drives' main strength is high speed random access. There are too many moving mechanical parts inside to make it suitable for long term storage. And the exposed disc platters will also corrode over time leading to data loss.

For serious long term storage, consider using RAIDed tape drives. You can use 2 tape drives for mirroring or 3 tape drives for RAID 5. Dedicated hardware controllers will perform the RAIDing for you so you can backup at full speed and tape changers can help you manage tape sets that increase storage capacity into to the thousands of terabytes range.

But to reduce costs for long term storage, even a simple tape drive will be more durable than hard drive. You can do manual mirroring by simply backing up data twice. And as your storage needs grow, just get more tapes.

Tapes are very slow and do deteriorate over time. The cost for these thousands of terabytes? Not practical except for the largest of corporations. If cost isn't a factor than a dedicated line to an offsite realtime Storage Area Network (SAN) may be the best bet(just kidding). I'm not convinced that CDs and especially DVDs aren't the best answer for the average joe. The article that stated CDs only last 5 years was pure hogwash. DVDs, properly handled, should last anywhere from 25 to 100 years and possibly more. Just remember to transfer to the latest technology before it (in this case DVD players) becomes obsolete.

[Wilt]

Tapes are very slow and do deteriorate over time. ... The article that stated CDs only last 5 years was pure hogwash. DVDs, properly handled, should last anywhere from 25 to 100 years and possibly more. Just remember to transfer to the latest technology before it (in this case DVD players) becomes obsolete.

1. Magnetic media is the ONLY form of digital storage proven to have lasted 50+ years.

2. About those 25 year CD...tell THAT to all the people who have lost data recorded to CD-R media (PhD theses, etc.) and to those who own all the home recorded music CD's that are no longer playable!

[GilesGuthrie]

Tape is expensive, no question. It is also the best form of long-term archiving.

Pros should really consider an Ultrium2 drive. It will back up 200GB (ignore the compression, because you're already compressing your image files). Archive off the data in 200GB(ish) chunks, and send the tapes offsite, preferably to a different town. You're perfectly safe sending the tapes in the mail in a padded bag. Ask a remote family member to store them somewhere for you in case you ever need them.

You can also protect yourself from local hard drive failure by using a NAS (Network-Attached Storage) device. These can be obtained quite cheaply, and located in a different part of your house, either using wired or wireless networking to transmit the data. Most of them come with backup/synchronisation software. It will also protect you from limited local physical disturbance.

But the cost of protecting data is all about the volume and the degree of security. How much protection do you need?

[Raymate]

RAID is good, do you have Mac or PC?

I have a 500 GB raid in my G4. In a Mac it's easy to create a mirror RAID, it's built into OS10

Also I archive to DVD burning stuff twice onto two different brands of blanks.

If you want off site backup, FireWire HDs from Lacie are cheap and small enough for someone to stick in a cupboard out the way.

[Raymate]

Well it has happened to me. I have lost 2 harddisks that were running in mirror. They died both at the same time don't know exactly what caused it but it happened so don't say it won't happen. I guess I should watch out for meteors.

That's very very very bad luck, I have instaled hundred of RAID and never seen a issue yet, seen single drives die in the RAID set but never two... ask someone else to pick your lottery number :)

[tommykjensen]

ask someone else to pick your lottery number :)

Haha You could be right. I never win more than a few dollars :lol:

[Raphael Emond]

You can also protect yourself from local hard drive failure by using a NAS (Network-Attached Storage) device. These can be obtained quite cheaply, and located in a different part of your house, either using wired or wireless networking to transmit the data. Most of them come with backup/synchronisation software. It will also protect you from limited local physical disturbance.

That's the way a friend work, and I found it to be pratical. More than me using 2 External HDD like a mirror and plug/deplug.

[tbrasington]

I don't like RAID too much it has failed on me before but I managed to recover the data. Rather stick to seperate hard drives in my pc and externals.

[wilflee]

sounds like the most logical idea. any tape drives you recommend?
Depends on your budget. You can get Sony AIT-2 autoloaders systems for about $500. AIT-3 autloaders systems for about $1000. The new Ultrium systems starts at around $2000.

If you're really tight on budget, older DLT systems can be had for less than $200. But the capacity is lower so you'll have to switch tapes manually and the backup speeds aren't that great (about 100 megs per minute).

[jevidon]

Depends on your budget. You can get Sony AIT-2 autoloaders systems for about $500. AIT-3 autloaders systems for about $1000. The new Ultrium systems starts at around $2000.

If you're really tight on budget, older DLT systems can be had for less than $200. But the capacity is lower so you'll have to switch tapes manually and the backup speeds aren't that great (about 100 megs per minute).

Thanks wilfee and to all the rest of you guys. This info will be very useful when I can finally afford a good backup system. Unfortunately I just did a spreadsheet of the cost of my camera gear (all purchased in the last 2 months) and it's painful.

[cbock]

to do it *right*, pick up the dam book (http://www.thedambook.com/)!

[jevidon]

to do it *right*, pick up the dam book (http://www.thedambook.com/)!

wow, that seems like it's got a lot of info. I'm sure most of it is pretty self-explanatory, but I might check this out. That bridge plugin looks mighty interesting too....thanks.

[woffles]

Nothings perfect. Magnetic media story.

Large company had tape backup system. Would do backups, verified fine, placed in storage. Tried recovery and they were all bad. This repeated with different tapes and tape drives. Called in the big boys, (Sun).

Sun engineer noticed sound behind wall of tape storage facility. Asked as to the origin. Was told that it was an elevator shaft behind the wall. Big electric motor going up and down daily, degaussing all their backup tapes.

Combo of DVD backups and magnetic media work for me. Have external hard drive that I turn on, do backup, then turn back off. Also have DVD burner to backup important pictures. No one good solution out there.

I would avoid Maxtor drives if I were you.

[Raphael Emond]

If I were you, I will do 2 backups...

One in a magnetic form.. i.e.: External HDD like 300Gb for fast security. Only pluging it when in use.

One in a DVD form. Kept away from your home and done on a month basis.

Plus having a copy of your file on your home computer. It's unlikely that the 3 media would fail at the same time...

For myself, I have 3 backups, one in the main machine and 2 external HDD only pluged my I do the backup. One of the disk is always outside of my home in case of fire/flood

I don't want to spend 2k on tape backup systems. I never had a HDD crash. With good airflow and good brand drives, they can last long. But safety is always good with things important.

You can't find 100% reliable backup system, But 95%+ is obtainable when working with the right tools.

Hope it helps!

[foty89]

1. Magnetic media is the ONLY form of digital storage proven to have lasted 50+ years.

2. About those 25 year CD...tell THAT to all the people who have lost data recorded to CD-R media (PhD theses, etc.) and to those who own all the home recorded music CD's that are no longer playable! If you buy garbage, you get garbage. There are well tested, long life CD's and DVD's. But to get them, you have to PAY for them. This also goes along with having multiple backup media and methods and then transfering to new methods and media as time passes.

[tommykjensen]

If you buy garbage, you get garbage. There are well tested, long life CD's and DVD's. But to get them, you have to PAY for them. This also goes along with having multiple backup media and methods and then transfering to new methods and media as time passes.

When I have created the backup DVD's I copy them to a secondary pc that also has two HD's this is to ensure the DVD's can be read. On this secondary pc the 2 HD's are in hotswap bays and while I copy the DVD's only one of the harddisks is mounted. When I have finished copying the DVD's I insert the second HD's and copy the first hd to the second ie a manual mirror. Then I shutdown the pc and remove the second hd. I only use this pc for this purpose. So in effect I have 5 copies of all my photos.

[neil_g]

That's very very very bad luck, I have instaled hundred of RAID and never seen a issue yet, seen single drives die in the RAID set but never two... ask someone else to pick your lottery number :)

ive had an array controller trash the data on RAID 1 and 5 drives it was running. luckily it was a server and had a tape backup.

i think for the average user a couple of NAS disks set-up to sync and then keep one in a fireproof box perhaps? imho tape is too expensive for a "home user", yes you get what you pay for but its still not fool-proof, easy to corrupt, can be quite slow (depending on application) and you'll need a spare SCSI.

there is also the option of backing up to hosts on the internet, fotki for example offer quite cheap no limits photo hosting. just make sure you have lots of bandwidth. this is currently my preferred method, all the photos i have taken are also on one of fotki's webservers. FTP them up, sit back (at work usually lol) and relax.

its all a game of pro's and con's based on application and budget.

[foty89]

I also have copies of all my pictures stored online, as well as on multiple hard drives and on CD's.

[SimonG]

... There are well tested, long life CD's and DVD's. ...
I think that the point is that there are no 25 year old verified discs that prove that the accelerated testing that has been done by the manufacturers is 100% accurate. You can believe their claims if you want, but I for one am not one to fall for marketing.

[foty89]

I think that the point is that there are no 25 year old verified discs that prove that the accelerated testing that has been done by the manufacturers is 100% accurate. You can believe their claims if you want, but I for one am not one to fall for marketing.

This is true, however saying that there are no such discs when there are ones that have been developed and tested, though not for the actual time, is not true. As with everthing, you are going to want to have more than ones means of backup. You also can't compare wallyword cheapo discs to the archivial ones out there, they might be called the same thing and look much alike, but they are not the same.

[Doom1701e]

I use a Maxtor Shared Storage 300GB drive which is connected through the router onto the network, and connected to the Maxtor Shared Storage Drive are 2 200GB Maxtor Personnal Storage HDD via USB into the MSS Drive so there is no need for plugging and unplugging. So I have all my RAWs on my computer, backed up to the MSS 300GB network drive, and backed up to one of the two 200GB personnal drives. Not too expensive and a little redundant but I feel I've got good backup. Not including the periodic DVD backups too.

[odvdveer]

I have 2 external usb drives, one is attache to the network with a linksys nslu2 (makes it a nas), and I copy to a dvd that is stored in a fireproof save in a different house. However save the raid or other backup systems are, if your house is burned down you have a major problem.

Olga

[jevidon]

I use a Maxtor Shared Storage 300GB drive which is connected through the router onto the network, and connected to the Maxtor Shared Storage Drive are 2 200GB Maxtor Personnal Storage HDD via USB into the MSS Drive so there is no need for plugging and unplugging. So I have all my RAWs on my computer, backed up to the MSS 300GB network drive, and backed up to one of the two 200GB personnal drives. Not too expensive and a little redundant but I feel I've got good backup. Not including the periodic DVD backups too.
I like this setup you have here, but I have had nothing but bad experience with Maxtor drives in the past. I have come to only investing in Seagate and occasionally Western Digital. If only I had more time to get this backup system going properly...

any recommendations on backup automation software (i.e. automatically distributes files to multiple backup locations)?

[neil_g]

if youre any good with DOS (and youre on PC) then knock up a BAT file that copies files to several locations with a new modified date, thats all i do. its simple and free.

[jevidon]

good call neil. but curious, why a new modified date?

[neil_g]

saves copying all files everytime..

XCOPY with a /E and /D switch. i do not suppress the overwrite confirmation just in case i wish to keep originals etc.

[jevidon]

got it. thanks for the tip.

[CyberDyneSystems]

Hard drives can do pretty well to fit the bill IF you use a lot of them in staggered back ups to prevent the issues of virus, deletion etc.

RAID installed on the PC (1,5 or another one for redundancy) and external drives that are NOT allways one, and use incremental timed back ups etc.. means that a lighting strike or other disaster won't take out ALL your data, as the idea would be to never have them all running. A pair of individual RAID1 arrays, one internal, one external, would be really hard to beat as well.

Still, I'd allways recomend and practice the use of burning DVD's as well.

[Bakewell]

Of course we’re all using a UPS (uninterruptible power supply) aren’t we? That takes care of lightning strikes, brown outs, and power loses

[Doom1701e]

I like this setup you have here, but I have had nothing but bad experience with Maxtor drives in the past. I have come to only investing in Seagate and occasionally Western Digital. If only I had more time to get this backup system going properly...

any recommendations on backup automation software (i.e. automatically distributes files to multiple backup locations)?

Seagate bought Maxtor a few months ago, if that helps any. :) The Maxtor Shared Storage drive comes with an automated backup program. You just mark the folders you want backed up and it will check them all for new/modified files every time a backup is run. You can schedule regular backups or do like i do and just back up after every shoot. Only takes a few minutes (I am only using 77GB right now). The Maxtor software can also keep up to I think 10-15 archival copies if you want to use this feature. So if you modify the original file and run a backup, the previously backed up copy is kept but the newly modified copy is also backed up. I dont use it but I am sure someone could get some use out of it.

[Jon]

I think that the point is that there are no 25 year old verified discs that prove that the accelerated testing that has been done by the manufacturers is 100% accurate. You can believe their claims if you want, but I for one am not one to fall for marketing.Know of any hard drives where the technology in use has been tested for 25 years continuous? Or tapes/tape drives? By the time you've conducted a real-time age test, it;'s too late to be usable because the components you were testing are no longer in production.

[SimonG]

Magnetic hard drives have been around a long time. They may be getting faster and smaller, and contain much higher data densities these days, but the underlying technology has been going strong for decades now. Personally, I have had experience with some systems in the 15-20 year old range that are still working today, and I am sure that there are others who have had similar experiences. Incidentally, I've never seen a hard drive manfacturer claim that their devices are suitable for long term archival storage as I have seen the CD / DVD manufacturers do, so perhaps it's a moot point anyways.

That said, that wasn't really my point. I was trying to caution people against relying solely on their CDs or DVDs as a long term backup, particularly where they will not be checked every now and then. If you believe the manufacturer's claims, you should be able to burn these discs and come back to them in 25+ years with all of the data in tact... and that's not something that I recommend testing with data of any value. Like others have said, it's not wise to rely on any one backup / storage medium.

[Jon]

The "underlying tehnology" is changing constantly too. Platters are now being made of glass, the mag stratum is changing (which allows the higher densities), head composition is changing. Anyone know how well the latest mag. oxides hold up to shock, whether the signals will wander and fade as we've seen in older floppies, or whether we'll see (as we have in mag tapes) flaking of the substrate from the new heads?

[CyberDyneSystems]

Of course we’re all using a UPS (uninterruptible power supply) aren’t we? That takes care of lightning strikes, brown outs, and power loses

Hopefully,.

...the first physical Hard Drive failure i ever had happend on my previous PC while it was connected to an APC UPS. The UPS simply did not react fast enough, and the whole works was killed (brown out) Power supply, Mobo and hard drive (one of three) died. Two managed to survive.. they were attached to a SCSI controller as opposed to the Mobo controller,. might explain there survival.

[SimonG]

Jon, you make a good point. I edited my last post to include this point, but you replied at about the same time, so I'll say it again: I've never seen a HDD manufacturer make claims like the current CD / DVD manufacturers are making regarding the archival abilities of the format, so I am not sure that your comparison to this technology is completely valid. I don't have tons of experience with tape, mainly because it's priced way out of my reach as a home user, so I will not comment on it.

I sincerely hope that the claims are correct... that would be a good deal of data security purchased for very little. I saw a claim today for CDs with an expected life span of 300 years (Mitsui Gold Archival Quality CD-R). Do you think that these claims are reasonable? I'm sure that they've done testing of some kind, but what did they assume in doing it?

[wilflee]

Tip for tape users. Get add'l sets of tapes and rotate the tapes so you'll have multiple sets of backups. I have a few hundred gigs of data now. Burning to DVD is out of the question as it will take over 100 DVDs and weeks to burn. But tapes hold a few hundred gigs each so it makes sense.

Seagate has a good article on tape rotation scheme --> http://www.seagate.com/products/tapesales/backup/A2g1.html

You can also RAID the tape drives. Data is written to 3 tape drives simultaneously. If one of the tape dies, you can rebuild the data from 2 remaining tapes.

[tommykjensen]

You can also RAID the tape drives. Data is written to 3 tape drives simultaneously. If one of the tape dies, you can rebuild the data from 2 remaining tapes.

That sounds interesting but isn't such a solution really expensive. I think just the single tape drives with a capacity higher than 10-20 gb is really expensive.

[CyberDyneSystems]

Too bad we couldn't just write them to all these old useless blank VHS tapes I have kicking around :lol:

[tommykjensen]

Too bad we couldn't just write them to all these old useless blank VHS tapes I have kicking around :lol:

Haha I remember for Amiga somebody had developed a backup system that in fact did use a videorecorder to store the backup on.

[winkie]

my brother has been looking at these: http://www.buffalotech.com/products/category-detail.php?categoryid=19

I personally would still back-up the data some where else, but these allow you to have network access to your data.

[Nortelbert]

if youre any good with DOS (and youre on PC) then knock up a BAT file that copies files to several locations with a new modified date, thats all i do. its simple and free.

Heh... I do that too; back up only new files every night at 1 AM to two different locations. That gives me two backups and one original. Can't see all three failing unless my house burns down. In that case I'd be more worried about my kids than my photos.

[Titus213]

Fascinating thread. My photos go to the HD, then to an external HD for backup. Then I burn a DVD for off-site storage (son's house) and another for on-site secure (safe). I figure the technology will change before the DVDs go bad and then they will have to be converted. But I am accululating a substantial stack of DVDs....

[odvdveer]

Heh... I do that too; back up only new files every night at 1 AM to two different locations. That gives me two backups and one original. Can't see all three failing unless my house burns down. In that case I'd be more worried about my kids than my photos.There are small fireproof saves. One of these will protect your backups and valuable papers on-site against fire and buglary.

Olga

[gkuenning]

Wow, lots of misinformation in this thread. So let me start with some credentials. I'm a storage systems researcher with published papers to my credit. It's part of my living, so I make it a point to keep up with the state of the art in the field. I'm also responsible for supervising my institution's computer facility, including dealing with the backup problem. That's why I have gray hair. :)

I'll start by saying that the people who think tape is best are dead wrong. Tapes are inherently inferior to disks from a reliability standpoint. Why? Two reasons: first, they're exposed to the environment. That means they pick up dust and stuff. (I hope all photographers are appropriately terrified of dust!) Second, tapes wind on spools, layer by layer. That means (a) the layers can stick to each other, and (b) you get print-through. Ask an old audiophile about the latter if you don't know what it is. For every failure mode that disks have, tapes have two. They've been used for decades only because of price, which is no longer a valid justification, and everybody who does tape backups has a story about lost data.

Disks have their own problems, for sure. Hard-drive manufacturers specify an MTBF (mean time between failures) assuming that you replace the drive every 2-5 years. They also make other assumptions; for instance, they generally assume that the drive is used relatively regularly. Many people are concerned that a drive that remains powered off for several years might have the heads stick to the platter (despite the fact that they're lubricated) and fail when it is started up. But they're still a safer bet than tape.

CDs and DVDs? In the long term, you're not going to be reading them on the drive you used to write them. Even if the medium doesn't deteriorate, you may have alignment problems. You can protect against that pretty well by double-checking the readability on a different drive as soon as you burn the thing.

Then there's the whole question of "bit rot", and the fact that what gets written to the storage medium isn't necessarily what you sent to it. Big sites HAVE to worry about this stuff.

And don't forget physical catastrophes. Your house might burn down. If Katrina comes your way, or an earthquake hits, it might not even be enough to have a backup in a bank vault.

That's the bad news. The good news is twofold. First, most of us can get away with playing the roulette wheel. The odds are GREATLY in your favor if you are a small user. So one option is to roll the dice, and be willing to lose stuff if life decides to screw you. I've got a lot of 35mm negatives and prints. No backups. If a fire hits, I'll lose 'em. And I'll sweep up the ashes, gather my family around me, and go on with life.

You can do the same with your RAID. While RAID isn't backup, and it can fail in many ways, the odds are hugely in your failure. So it might be best to just make the bet.

If eating the loss isn't an option, then you can do what some of the other people here do. Burn those DVDs and CDs, and put 'em in a bank vault, preferably at least 100 miles away. After two years, check them for readability and copy them to fresh media. Keep checking them every two years, and when they start to go flaky or the technology changes, make new copies from the first set of copies. You might lose some stuff, but most of it will survive.

Or you can use a network mirroring service. They typically have multiple installations in different places, so they can survive natural disasters. The good ones also take backups so that an intruder can't destroy everything. Some of them got their start providing backups for banks, so they're GOOD at reliability. But it's not necessarily going to be cheap.

Incidentally, if you do lose data and really want to get it back, keep the failed device. There are services that can do wonders in data recovery, using things as exotic as electron microscopes. They're not cheap, but for some of us, it might be worth it.

[Raymate]

Wow, lots of misinformation in this thread. So let me start with some credentials. I'm a storage systems researcher with published papers to my credit. It's part of my living, so I make it a point to keep up with the state of the art in the field. I'm also responsible for supervising my institution's computer facility, including dealing with the backup problem. That's why I have gray hair. :)

I'll start by saying that the people who think tape is best are dead wrong. Tapes are inherently inferior to disks from a reliability standpoint. Why? Two reasons: first, they're exposed to the environment. That means they pick up dust and stuff. (I hope all photographers are appropriately terrified of dust!) Second, tapes wind on spools, layer by layer. That means (a) the layers can stick to each other, and (b) you get print-through. Ask an old audiophile about the latter if you don't know what it is. For every failure mode that disks have, tapes have two. They've been used for decades only because of price, which is no longer a valid justification, and everybody who does tape backups has a story about lost data.

Disks have their own problems, for sure. Hard-drive manufacturers specify an MTBF (mean time between failures) assuming that you replace the drive every 2-5 years. They also make other assumptions; for instance, they generally assume that the drive is used relatively regularly. Many people are concerned that a drive that remains powered off for several years might have the heads stick to the platter (despite the fact that they're lubricated) and fail when it is started up. But they're still a safer bet than tape.

CDs and DVDs? In the long term, you're not going to be reading them on the drive you used to write them. Even if the medium doesn't deteriorate, you may have alignment problems. You can protect against that pretty well by double-checking the readability on a different drive as soon as you burn the thing.

Then there's the whole question of "bit rot", and the fact that what gets written to the storage medium isn't necessarily what you sent to it. Big sites HAVE to worry about this stuff.

And don't forget physical catastrophes. Your house might burn down. If Katrina comes your way, or an earthquake hits, it might not even be enough to have a backup in a bank vault.

That's the bad news. The good news is twofold. First, most of us can get away with playing the roulette wheel. The odds are GREATLY in your favor if you are a small user. So one option is to roll the dice, and be willing to lose stuff if life decides to screw you. I've got a lot of 35mm negatives and prints. No backups. If a fire hits, I'll lose 'em. And I'll sweep up the ashes, gather my family around me, and go on with life.

You can do the same with your RAID. While RAID isn't backup, and it can fail in many ways, the odds are hugely in your failure. So it might be best to just make the bet.

If eating the loss isn't an option, then you can do what some of the other people here do. Burn those DVDs and CDs, and put 'em in a bank vault, preferably at least 100 miles away. After two years, check them for readability and copy them to fresh media. Keep checking them every two years, and when they start to go flaky or the technology changes, make new copies from the first set of copies. You might lose some stuff, but most of it will survive.

Or you can use a network mirroring service. They typically have multiple installations in different places, so they can survive natural disasters. The good ones also take backups so that an intruder can't destroy everything. Some of them got their start providing backups for banks, so they're GOOD at reliability. But it's not necessarily going to be cheap.

Incidentally, if you do lose data and really want to get it back, keep the failed device. There are services that can do wonders in data recovery, using things as exotic as electron microscopes. They're not cheap, but for some of us, it might be worth it.


Being an Apple Engineer for almost nine years I agree with all that is said, you just need to move your data to new formats every few years. For the now RAIDs are a good gamble for every day use and working spaces. But long terms just copy old backups to new formats and spread them around the family.

I have done this myself I have CDR from over ten years ago that still work but I now have the content on DVDR and I'm sure they will in turn be copied to Blue Ray or any other storage the future holds.

Hardware changes, media changes just copy stuff every few years and you should be good.

The poster is correct never use tape based system.

In my line of work I see customers who have lost drives and data almost weekly (we have many customers) but as drive size get bigger so does the risk. Nothing is 100% but some stuff works better than others.

I still have some betamax video tapes that still play today, I also have VHS recording from 20 years ago and they play great today, but at the time I used high grade media costing a small fortune, but perhaps it paid of, they still play. I also have LaserDiscs from the late 70's that still play.

Spread the images around and in time if you cant open that file we just live in hope your family memeber can.

R

[jevidon]

Gkuenning, thank you so much. Your post has clarified everything I was hoping to learn from starting this thread. However, I'm interested to hear your opinion on Blueray and HD-DVD technology. Are these formats being designed to sustain a longer shelf life and is it worth investing in it now? And if not, is there any draw back to using dual-layer DVD's or are the single-layer more reliable?

Thanks again and looking forward to your post.

P.S.:
While RAID isn't backup, and it can fail in many ways, the odds are hugely in your failure.
The only mistake I found in your post is quite amusing. Almost a Freudian slip... :)

[remo]

My backup solution is:

1. Copy from camera to local Hard Drive.
2. Copy from local Hard drive to backup drive (non-RAID).
3. Copy from local drive to DVD's and as soon as possible take them to off-site (parents house, in-laws, children)

[225719]

Hopefully,.

...the first physical Hard Drive failure i ever had happend on my previous PC while it was connected to an APC UPS. The UPS simply did not react fast enough, and the whole works was killed (brown out) Power supply, Mobo and hard drive (one of three) died. Two managed to survive.. they were attached to a SCSI controller as opposed to the Mobo controller,. might explain there survival.



Well, Thats because your PSU was junk :lol: My pc took a massive serge as well but my PSU (PC power and cooling) took the blow and isolated it from the rest of the system like its suposed to. After all you get what you pay for now adays...

[Raphael Emond]

I'm interested to hear your opinion on Blueray and HD-DVD technology. Are these formats being designed to sustain a longer shelf life and is it worth investing in it now?

If it's like DVD, the first batch to appear on the market is to be avoided.
More with the burnables discs. Since it's a new technology, wait a least a year to
mature. Even today, DVD-R discs can be real crap if you don't pick the good brands.
I've seen some media, after the burn, which is almost unreadable with a good scan.

The technology should be good. But let wait a little for the dust to settle on the
"format war"

[claudermilk]

Thanks for the post, gkuenning. Basically backs up what CyberDyne says & what I've been thinking.

My own system is currently: Main online storage is RAID0+1. Extended is removable HDD. Backup is another removable HDD. Secondary backup to DVD. I will add a second backup HDD to rotate to the office, so one onsite with the PC getting fresh backups, one offsite just in case & rotating regularly. I can add extra HDDs as needed with this and easily maintain redundancy.

Tapes are way too expensive for what you get & having seen the (un)reliability of them over the years, I don't trust them. Now seeing the unreliability of optical media (right now running IsoBuster on less than 2 year old DVD recovering data) that's not a viable option either. Seems like there's nothing better than the good old hard drive yet.

[gkuenning]

Gkuenning, thank you so much. Your post has clarified everything I was hoping to learn from starting this thread. However, I'm interested to hear your opinion on Blueray and HD-DVD technology. Are these formats being designed to sustain a longer shelf life and is it worth investing in it now? And if not, is there any draw back to using dual-layer DVD's or are the single-layer more reliable?
Jorlando, I'm going to have to disappoint you on this one because I'm completely ignorant on those subjects. I've deliberately avoided following the Blu-Ray versus HD-DVD wars; hard disks are more my interest and it's more efficient to wait for one format to win before I bother to learn about either. :D As to single- versus double-layer, I haven't had occasion to look into that question. Sorry.


The only mistake I found in your post is quite amusing. Almost a Freudian slip... :)
LOL! I guess I should have used the preview button. I promise I'll do so this time.

[hemuni]

Have you considered a NAS storage solution like Synology's Diskstation:
http://www.synology.com/enu/products/DS106serise/index.php

It handles all my backups with the possiblity of restoring files in upto 20 levels (i even have a usbdisc connected for backing up the backup).

Its also a full FTP server, webserver with PHP+MYSQL, photoserver, fileserver, printerserver and more.

Supports gigabit lan directly and wifi wia router.

I just love the photoserver. All i have to do is copy some photos into the photosection and the station generates the html and php based galleries automaticly. Great for showing pics to impatient family members :)

I higly recomend it. Its been running for little over a year without any breakdowns its superstable.

[jevidon]

thanks hemuni. My only concern is the limitation of not being able to perform a RAID array, but I do like all of the automated things it can do and would easily fit inside my desk at work where I am hooked up directly to a University gigabit ethernet connection ;)

[Transportithere]

I do remember when 8 track was the hottest thing, since the bottle opener.
How about 45s and LP records? Remember the 386, 486. Did you ever dream of a Gigabyte? I agree needing to back up every few years, will be necessary.

[winkie]

thanks hemuni. My only concern is the limitation of not being able to perform a RAID array, but I do like all of the automated things it can do and would easily fit inside my desk at work where I am hooked up directly to a University gigabit ethernet connection ;)

their CS-406 has 4 SATA drives, and can be raided...
http://www.synology.com/enu/products/CS406series/index.php

thanks hemuni... excellent suggestion

edit... I just realized this is just an enclosure, and does not come with the hard drives.

[jevidon]

enclosure is fine with me - i'd rather have the choice of what HD to put in there as opposed to some generic crap. thanks for the tip winkie.

[Ferdinand]

In short, we are DOOMED.

I am just gonna live my life to the fullest and take lots of pictures :)

[jevidon]

i think i've got a pretty fool-proof system.

While in the field:
Shoot photos (duh)
If possible, back up on to Hyperdrive and leave photos on CF cards for redundancy

Once back at home:
Download photos from CF cards to two separate external HD's... Sort, tag, throw away and catalog photos from main set. Once 4GB's of photos collected, burn to DVD and make copy of photos to the main and secondary HD. After that is completed, delete previous backup download.

So far I have felt quite confident about the longevity of this system. Yes it does require twice the space, but in the process of going back through my shots and archiving them properly I have thrown away 5000+ photos that were useless.... It was definately worth the hours of tedious work.

[Transportithere]

To my limited access to technology and knowledge; DVD is my best resource. I have heard that DVD will deterorate with time. The best I can say is, "oh well". I tried. I have mailed a collection to the Library of Congress for registration.
From what I was informed CD/DVD is prefered to hard copy.
I am open eared to other techniques. I had HDs gererate General Failures.
Hence the tail of; Who is General Failure? Why is he reading my hard drive!

[rhys]

1. No system is foolproof.
2. Not every photo taken before digital survived.

How many Daguerrotypes were simply melted down for their silver content?

How many offices have burned down complete with stored negatives?

What about negatives that are redundant? I gather many Civil War news plates were collected up and used as glass in greenhouses after the newsworthiness of the Civil War receeded.

[jevidon]

I have mailed a collection to the Library of Congress for registration. From what I was informed CD/DVD is prefered to hard copy.What is the process to send a collection to the Library of Congress? Do you need to pay a separate fee for each photo on the CD/DVD or is it one fee for the whole thing?

[Chrisedge]

Also, don't forget to generate Md5 files to verfy the files you do backup onto discs.

I do that everytime.

[jevidon]

Also, don't forget to generate Md5 files to verfy the files you do backup onto discs.

I do that everytime.I never even thought of that. what program do you use to create the checksums?

[StealthLude]

I got 2000 Gigs of server storage. I backup from camera cards to server. Im still protected from hard drive faluir since Im using a $500 3-ware RAID card in Raid 5 configuration. I do however have a VPN setup to my home, where I have to backing up the picture folder file to help loss in times of fire, theift, or something else out of my control.

[Chrisedge]

I never even thought of that. what program do you use to create the checksums?

md5summer

Google it and it comes up.

I have had to piece together backups that I THOUGHT were complete, then finding out a few pictures were screwed up. Now I just verify them and I know they are OK on the DVD at least. Who knows how long they will last...