RAID expandability

[narlus]

I'm long overdue for a better strategy than a desktop full of external HDs, and figure that some sort of an eSATA server would likely be the best way to go about doing this. here's what i've got in mind, tell me the weaknesses in my strategy, or other suggestions:

i would like to keep using an internal drive for 'work in progress' activities, mainly as i figure it will be faster.

once i've worked on a set, i will typically have 20-50 shots that i liked well enough to work on, and also generated full-res PSD and JPGs. there will likely be ~100-300 shots in RAW that would be kept for whatever purpose in the future. the 'worked on' photos would be assigned stars via Bridge.

Once the set is complete, i'd move that folder to my RAID box. ideally, i would also to know if there's a way to do remote backups of just the starred items (RAW and PSD files).

if one chooses a RAID box of ~2TB, what are the options to increase capacity when i fill it to 2TB? would i have to pull all the drives out, shelf 'em, and reload new ones?

any help is appreciated! thanks in advance.

btw, i am looking at this one:
http://buffalotech.pricegrabber.com/search_getprod.php?vcode=PGBFT&partnum=LSQ40TLR5

[tim]

Raid only protects against drive failure, it's not all that helpful for most photographers IMHO. Instead I suggest internal drives, with an external drive as backup. If you need more space buy a bigger drive, migrate data over, and put the old drive in a drawer, or install a bigger data drive.

My PC has:
- C drive for OS and software (400GB drive, 20GB used)
- D drive for personal data, general data, personal photos, media (1TB, 200GB free)
- E drive for commercial photos (1TB)
- F drive for swap/scratch/cache

I have two 1.5TB drives as backups to my internal drives, kept offsite, backed up with robocopy.

[basroil]

Raid only protects against drive failure.

Not even that actually, just downtime.

Basically, you really do want to just get multiple drives, they are cheap. I just dropped off one of my 500gb drives filled with photos (has less than 1gb free) at a place halfway across the country, and i have two copies of every photo, one in an external esata drive, the other in an internal sata one. Fairly fast transfers. In fact, averaging about 45mb/s right now as I'm copying over the drive, which is plenty fast considering it's a 14mb raw and a 2kb xmp (very small files mean very poor disk performance)

EDIT: Forgot that the first set of files I was backing up was all files under 1mb, with 60% about 500kb and 20% about 2kb. Now after an hour of backing up, stabilized at about 55mb/s. When it gets to video, should jump up a bit more, maybe 60-65mb/s

[tim]

I just dropped off one of my 500gb drives filled with photos (has less than 1gb free) at a place halfway across the country

That must make refreshing your backups quite difficult, but will protect you when a house sized meteor wipes out your city. Of course you'll be dead so you won't care, but the data will be safe! :p

[basroil]

That must make refreshing your backups quite difficult, but will protect you when a house sized meteor wipes out your city. Of course you'll be dead so you won't care, but the data will be safe! :p

Exactly ;)

Though backups can be easy or even easier. I just drag and drop, and skip files that haven't been touched recently. But windows 7 also has some fairly good backup tools, but I like being able to keep the main files on the internal and then backing up to the external, that way I can also have those files available for my laptop if needed.

[Silvertree]

I haven't worked with Bridge so I can't comment on the software work flow. I can say the IT Director and a few others had Drobos they were testing. Everyone really liked them so you might take a look.

[narlus]

Not even that actually, just downtime.

Basically, you really do want to just get multiple drives, they are cheap. I just dropped off one of my 500gb drives filled with photos (has less than 1gb free) at a place halfway across the country, and i have two copies of every photo, one in an external esata drive, the other in an internal sata one. Fairly fast transfers. In fact, averaging about 45mb/s right now as I'm copying over the drive, which is plenty fast considering it's a 14mb raw and a 2kb xmp (very small files mean very poor disk performance)

EDIT: Forgot that the first set of files I was backing up was all files under 1mb, with 60% about 500kb and 20% about 2kb. Now after an hour of backing up, stabilized at about 55mb/s. When it gets to video, should jump up a bit more, maybe 60-65mb/s

i have four disk bays in my PC...should i get an external box and then mirror the internals to the externals for backup? RAID is not worth it? i am very confused about the best strategy.

the only thing I am clear on is that I need to do something :D

[René Damkot]

How about a drive dock?
http://photography-on-the.net/forum/showthread.php?p=8746527#post8746527
I'm planning to get one, as soon as finances allow.

Then just dupe everything, and store them in a safe off site.

[blackhawk]

Use one of these or other models for fast swap outs.
http://www.mwave.com/mwave/ViewProductGrid.asp?PID=INTENCLOSURE-KINGWIN&DPDID=
http://www.mwave.com/mwave/specHR/images/AA70870a.jpg

[narlus]

How about a drive dock?
http://photography-on-the.net/forum/showthread.php?p=8746527#post8746527
I'm planning to get one, as soon as finances allow.

Then just dupe everything, and store them in a safe off site.

that actually looks really good. super cheap too, $30?

http://www.newertech.com/products/voyagers2.php

[narlus]

yeah i think i am gonna do this as my archiving strategy. i just gotta figure out my mobo/eSATA details, then i'll buy a bunch of internal disks and start copying.

[basroil]

yeah i think i am gonna do this as my archiving strategy. i just gotta figure out my mobo/eSATA details, then i'll buy a bunch of internal disks and start copying.
If you don't have esata and don't mind a esata cable coming out the front of your computer, antec has hot swap bay with a passthough sata to esata connection.

[IUnknown]

This is what I ordered last weekend,
http://www.newegg.com/Product/Product.aspx?Item=N82E16822108028&nm_mc=TEMC-RMA-Approvel&cm_mmc=TEMC-RMA-Approvel-_-Content-_-text-_-

NAS Server's just hook up to your network. There are 4 bay drives available. I'm still learning about this technology, haven't really thought about what the upgrade options are going this route. Drobo's are great in that as technology improves you just add the larger capacity drives as you get them, and they don't have to match up in any way (but its an expensive option). I also use these with my media center so that I can watch movies without my computer being on. I'd like to look into solid state drives and how much those improve performance in lightroom, etc.

http://www.drobo.com/

[René Damkot]

On a slightly related note:

Purely for off-site backup: What about tape?

Seems there's still development:
http://www.engadget.com/2010/01/23/ibm-and-fujifilm-develop-35tb-magnetic-tape-cartridges-unveil-i/

[blackhawk]

On a slightly related note:

Purely for off-site backup: What about tape?

Seems there's still development:
http://www.engadget.com/2010/01/23/ibm-and-fujifilm-develop-35tb-magnetic-tape-cartridges-unveil-i/


Tape is still more robust than any other common data storage medium, and that dismays me! Think how silly easy it is now to destroy most of the digital memories we depend on and cherish in a one EMI pulse.

[basroil]

Tape is still more robust than any other common data storage medium, and that dismays me! Think how silly easy it is now to destroy most of the digital memories we depend on and cherish in a one EMI pulse.

Tape is still the form of modern media that is most susceptible to EM pulses and magnets though. Also is more susceptible to scratches and dust degrading the quality, and it's linear density is the absolute lowest (so less backups/cubic foot). Do we have to mention the fact it's a linear format which means random read/write times are the lowest thing next to punchcards (and sometimes even that's faster). I'll take a RAID 6 system with a half a dozen drives over tape any day ;)

[René Damkot]

Tape is still the form of modern media that is most susceptible to EM pulses and magnets though.
Didn't know that...
Then again, that wouldn't matter much if it's kept in a safe, right?
Also is more susceptible to scratches and dust degrading the quality,
Is that so? I thought it was one of the most dependable (like 50+ years, from reading I've done)
and it's linear density is the absolute lowest (so less backups/cubic foot).
If I look at that article, 35TB is hardly "low density"
No HDD comes close AFAIK.
Do we have to mention the fact it's a linear format which means random read/write times are the lowest thing next to punchcards
Well, obviously. But that's no issue for offsite backup only: If you'll need the data on those tapes, it's because your local backup died. So you'll need all of it.


Just curious, since I need something for off site backup.
Either a HDD in a dock, or a tape. I don't care which, as long as it's dependable (over many years without checking) and preferably cheap.

[basroil]

Didn't know that...
Then again, that wouldn't matter much if it's kept in a safe, right?
But then again, same with flash and hdds
Is that so? I thought it was one of the most dependable (like 50+ years, from reading I've done)
If properly kept, even most dvds are rated for 50+ years, and my 7 year old dvds are still ticking and they were 2.4x disks from the time that 1/3 disks was DOA, so archival ones may actually last three decades. Most tapes are rated for 15-30 years rather than 50+.
If I look at that article, 35TB is hardly "low density"
No HDD comes close AFAIK.
Prototype is far from a product. That said, they do have an 8tb one, which is a far cry from the 80gb I remember messing with a few years back
Well, obviously. But that's no issue for offsite backup only: If you'll need the data on those tapes, it's because your local backup died. So you'll need all of it.
Yup, and if you need all of it fast, the newer tapes (just found this out myself) get read speeds faster than many SSD (180MB/s). But they also cost in the 10k+ range.

Just curious, since I need something for off site backup.
Either a HDD in a dock, or a tape. I don't care which, as long as it's dependable (over many years without checking) and preferably cheap.

Tape just isn't practical or cost effective for regular users. Maybe if you had dozens of TB that you have to back up, but the initial cost is just too high for most people. For large businesses, hell, it's amazing. But why pay $500 for your first 500gb when you can buy five drives twice the size? Chances are at least one of those five will last longer than three decades if mostly unused.

[blackhawk]

Tape is still the form of modern media that is most susceptible to EM pulses and magnets though. Also is more susceptible to scratches and dust degrading the quality, and it's linear density is the absolute lowest (so less backups/cubic foot). Do we have to mention the fact it's a linear format which means random read/write times are the lowest thing next to punchcards (and sometimes even that's faster). I'll take a RAID 6 system with a half a dozen drives over tape any day ;)

Even a near strike lightning induced EM pulse is more than enough to completely destroy the whole computer including hard drives, grounded case and all... let alone a EMP bomb.

HDs can easily have their interfaced fried rendering the data inaccessible without a recovery service. Not so with a tape drive.
It's simple to fix a break in the tape is easy to repair, a hhd is not.
It takes time for a virus to overwrite a tape drive... and has a tape no firmware.

"keep it simple, stupid"

[basroil]

Even a near strike lightning induced EM pulse is more than enough to completely destroy the whole computer including hard drives, grounded case and all... let alone a EMP bomb.

HDs can easily have their interfaced fried rendering the data inaccessible without a recovery service. Not so with a tape drive.
It's simple to fix a break in the tape is easy to repair, a hhd is not.
It takes time for a virus to overwrite a tape drive... and has a tape no firmware.

"keep it simple, stupid"

Wait, what? How does any of this relate to backups for an off-site offline system?

[blackhawk]

Didn't know that...
Then again, that wouldn't matter much if it's kept in a safe, right?


It must be earth grounded and a completely sealed Faraday cage.


Is that so? I thought it was one of the most dependable (like 50+ years, from reading I've done)

It can last longer than 10 years, but not a good idea as just like hard drive the magnetic image starts degrading as soon as it is made.


Just curious, since I need something for off site backup.
Either a HDD in a dock, or a tape. I don't care which, as long as it's dependable (over many years without checking) and preferably cheap.

Best to use a removable hd enclosure mounted in the computer case such as the ones Kingwin makes.

While tape offers many advantages for businesses, a set of at least two hard drive based images store off site will do and is easier. The drives should be spun up and refreshed or used at least every 3 months to prevent possible data lose.
A third copy on DVDs isn't a bad idea.

Data on the computer is not safe. A fire or near lightning strike can easily wipe the hard drives.