Approve the Cookies
This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and our Privacy Policy.
OK
Index  •   • New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Guest
New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Register to forums    Log in

 
FORUMS Canon Cameras, Lenses & Accessories Canon Accessories 
Thread started 27 May 2014 (Tuesday) 15:04
Search threadPrev/next
sponsored links
(this ad will go away when you log in as a registered member)

My storage/backup problem and fix... need input

 
Talley
Talley Whacker
Avatar
11,087 posts
Gallery: 46 photos
Likes: 2773
Joined Dec 2011
Location: Houston
     
May 27, 2014 15:04 |  #1

So my current setup is an internal 3TB drive and an external 3TB drive which is setup to backup the internal. The problem is it's getting full. currently 270GB left and it won't take too much longer to fill it up. probably by end of summer. The video camera has accelerated the consumption of GBs.

Long term solution would be to bin anything not needed. Compress the video files I want to keep and free up space but this would only free up maybe a few hundred gigs and I need something to last me a few more years.

This is what I'm thinking of doing and would like your input.

I purchased a 5TB external. Backup everything to it. take the external 3 and raid that together with the internal for 6TB. I basically double my space internally and have the 5TB on the external side. Was also thinking of adding another 5TB external that I could keep at a safety deposit box and back it up every other month type thing.

Any thoughts on this and any input would be appreciated.


A7rIII | A7III | 12-24 F4 | 16-35 GM | 28-75 2.8 | 100-400 GM | 12mm 2.8 Fisheye | 35mm 2.8 | 85mm 1.8 | 35A | 85A | 200mm L F2 IS | MC-11
My Gear Archive

  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)
Diamond-Lord
Junior Member
29 posts
Joined May 2014
     
May 28, 2014 13:01 |  #2

All depends on your long-term storage needs, and which files will be on long-term storage.

Long term storage on a spinning disk really doesnt make sense. You're better off renting an LTO-6 tape drive and backing up long-term files on LTO-6 tape cartridge that costs around $40 to hold 2.5 TB compressed.

However, the idea of using a 5TB external HDD to back up a 6TB RAID array makes sense --- assuming that 1 TB of that RAID is holding OS and other non-essential files. I have that current setup and it works well for immediate nightly backup of my main RAID (I put the HDD backup in my gun safe which has a 2000F fire rating and too heavy to blow away in a tornado). However, I also have around 10TB of AVI files that are 15 years old and store them on LTO tape in a bank security deposit box since hard disks dont have the reliability over decades that tape has. The Library of Congress stores all their archival material on LTO-6 Ultrium tape, so that should indicate what the bonafide experts think about long term storage of digital files.




  
  LOG IN TO REPLY
pwm2
"Sorry for being a noob"
Avatar
8,626 posts
Likes: 3
Joined May 2007
Location: Sweden
     
May 28, 2014 13:30 |  #3

1) Raiding two 3TB disks to get 6TB? That means RAID-0, i.e. striping. That's the absolutely worst thing you could ever do unless your only goal in life was to try to create a higher transfer rate. If one of the disks fails, all surface area fails. Don't spend a single more second thinking this thought.

2) Compressing video? The captured video is already compressed. So lossless compression doesn't give much. And lossy compression would mean loss of quality.

3) Stop talking about backup and start thinking about archiving. Backup is what you do of a live file system where files are expected to change, or added/moved/deleted. But the majority of your files aren't expected to be living, i.e. changing. So get two cheap 2TB or 3TB disks. If you have a stationary computer, make one an internal disk and one an external.

Identify directories with files you aren't expected to change. Make one copy of the files to one of the new disks. The other copy to the second disk. Then you can release that disk space from the "working set", which means you have less data to constantly backup.

4) With one internal 3TB disk and one external 3TB disk, it doesn't sound like you are running any actual backup software. Most probably, you are just making one mirror copy. A real backup uses version control, which mean that if one file changes, then the backup disk will store both a copy of the file before the change, and one copy of the file after the change. This means that you can go back in time and recover in case you accidentally damage a file but doesn't notice it directly when it happened. A mirror just means you quickly get multiple disks containing the same, destroyed, file. And if the mirroring software supports delete, then a delete on the main disk gets mirrored on the external disk.


5DMk2 + BG-E6 | 40D + BG-E2N | 350D + BG-E3 + RC-1 | Elan 7E | Minolta Dimage 7U | (Gear thread)
10-22 | 16-35/2.8 L II | 20-35 | 24-105 L IS | 28-135 IS | 40/2.8 | 50/1.8 II | 70-200/2.8 L IS | 100/2.8 L IS | 100-400 L IS | Sigma 18-200DC
Speedlite 420EZ | Speedlite 580EX | EF 1.4x II | EF 2x II

  
  LOG IN TO REPLY
morph2_7
Goldmember
1,112 posts
Joined Sep 2012
Location: Los Angeles
     
May 28, 2014 13:48 |  #4

Talley wrote in post #16934050 (external link)
take the external 3 and raid that together with the internal for 6TB. I basically double my space internally and have the 5TB on the external side.

Any thoughts on this and any input would be appreciated.

Sounds like software RAID. You don't gain much other than doubling the capacity. Not worth the risk of losing everything if one of the 2 (3TB) HDs fails. Btw, I've never done software RAID with 1 internal + 1 external drives. Not even sure if that's possible.




  
  LOG IN TO REPLY
pwm2
"Sorry for being a noob"
Avatar
8,626 posts
Likes: 3
Joined May 2007
Location: Sweden
     
May 28, 2014 13:49 |  #5

Diamond-Lord wrote in post #16936125 (external link)
All depends on your long-term storage needs, and which files will be on long-term storage.

Long term storage on a spinning disk really doesnt make sense. You're better off renting an LTO-6 tape drive and backing up long-term files on LTO-6 tape cartridge that costs around $40 to hold 2.5 TB compressed.

Note that backup to tape is very seldom a good idea any more. The tape drives are extremely expensive in relation to how storage they allow. 2.5TB compressed is an irrelevant figure because the on-the-fly compression used is very simple and JPEG, MOV, MP3 and other media formats are normally already using a much more effective compression - so the tape drive either has to write without compression or will instead increase the file sizes when doing the compression.

If you are buying a web hotel from some company, then they are almost certainly not using any tape backup system but instead using one or more RAID:ed machines for the backup. Spinning disks just happens to have a very affordable price compared to the total cost of a large-capacity tape solution. So I can buy 10 4TB disks for the price of one larger tape drive. The tape drive data could be classified as "near online", which maybe 40 seconds access time. And requiring that someone remembers to replace that tape cartridge before the next full backup is run. The 10 4TB disks could instead handle 32TB of online RAID-6 data with two parity disks. Be used for two mirrored RAID-6 with 12TB free space each. And no cartridge to forget.

And another thing: The cartridges are a good way to "lock in" data. When that very expensive tape drive dies, the tape access is lost. Maybe there are newer generation newer tape drive to buy that happens to support at least read-only access of the currently used tapes. Or maybe it suddenly gets time to in a panic scan ebay for some used equipment. With hard disks, people normally just move the data to newer disks every 3-4 years, as the disk sizes increases.

This is the big reason why people are so much interested in NAS solutions or small home servers. And the NAS or home server solution normally also handles other tasks, like running web pages, mail server or allowing access to media when out traveling.


5DMk2 + BG-E6 | 40D + BG-E2N | 350D + BG-E3 + RC-1 | Elan 7E | Minolta Dimage 7U | (Gear thread)
10-22 | 16-35/2.8 L II | 20-35 | 24-105 L IS | 28-135 IS | 40/2.8 | 50/1.8 II | 70-200/2.8 L IS | 100/2.8 L IS | 100-400 L IS | Sigma 18-200DC
Speedlite 420EZ | Speedlite 580EX | EF 1.4x II | EF 2x II

  
  LOG IN TO REPLY
pwm2
"Sorry for being a noob"
Avatar
8,626 posts
Likes: 3
Joined May 2007
Location: Sweden
     
May 28, 2014 13:53 |  #6

morph2_7 wrote in post #16936238 (external link)
Sounds like software RAID. You don't gain much other than doubling the capacity. Not worth the risk of losing everything if one of the 2 (3TB) HDs fails. Btw, I've never done software RAID with 1 internal + 1 external drives. Not even sure if that's possible.

Software or hardware RAID is irrelevant here. Hardware RAID just means there is a dedicated RAID controller that computes the parity information and that optionally contain a battery-backed cache memory. The issue here is that RAID-0 should be read as RAID-times-zero. Anything times zero is zero. And RAID-0 fails to fulfill the "R" of RAID (Redundant Array of Inexpensive Disks) since there are no redundancy. Just a much increased failure risk since any disk lost in a RAID-0 gives a total loss.


5DMk2 + BG-E6 | 40D + BG-E2N | 350D + BG-E3 + RC-1 | Elan 7E | Minolta Dimage 7U | (Gear thread)
10-22 | 16-35/2.8 L II | 20-35 | 24-105 L IS | 28-135 IS | 40/2.8 | 50/1.8 II | 70-200/2.8 L IS | 100/2.8 L IS | 100-400 L IS | Sigma 18-200DC
Speedlite 420EZ | Speedlite 580EX | EF 1.4x II | EF 2x II

  
  LOG IN TO REPLY
rdwalton
Member
Avatar
153 posts
Likes: 2
Joined Dec 2009
Location: Dallas, TX
     
May 28, 2014 14:00 as a reply to  @ pwm2's post |  #7

I have multiple drives in my computer that I use for redundancy, as well as an external drive. I use Acronis to backup my entire system and send copies to those drives. In addition, I use cloud storage just in case there's some sort of natural disaster or burglary. Cloud storage is fairly cheap these days, so I use that as well. Oh and I utilize the free TB of space that you get free at Flckr.


Reginald D. Walton | GEAR | Smugmugexternal link

  
  LOG IN TO REPLY
gnome ­ chompski
Goldmember
1,252 posts
Gallery: 1 photo
Likes: 136
Joined Jun 2013
Location: oakland, ca
     
May 28, 2014 14:09 |  #8

I am much more low tech than many here, but i just cull all nonsense and try to approach my photography as if I were still shooting film. Example: 24 or 36 exposures so choose carefully. While I am not as successful at restrain most of the time, I do maintain a rigorous policy of deleting garbage shots before I do any processing.

I do keep a redundant back up at work, that I bring home every now and then to back up my working external drive to.

Basically, I cannot help LOL.


Tumblr (external link)
Flickr (external link)
Instagram (external link)

  
  LOG IN TO REPLY
morph2_7
Goldmember
1,112 posts
Joined Sep 2012
Location: Los Angeles
     
May 28, 2014 14:50 |  #9

pwm2 wrote in post #16936246 (external link)
Software or hardware RAID is irrelevant here. Hardware RAID just means there is a dedicated RAID controller that computes the parity information and that optionally contain a battery-backed cache memory.

It is relevant. Software RAID taxes CPU power while hardware RAID uses its own CPU. You may gain a little bit in data access at the cost of CPU time. As I said, OP wouldn't gain much with software RAID while doubling the risk of data loss.

pwm2 wrote in post #16936246 (external link)
The issue here is that RAID-0 should be read as RAID-times-zero. Anything times zero is zero. And RAID-0 fails to fulfill the "R" of RAID (Redundant Array of Inexpensive Disks) since there are no redundancy. Just a much increased failure risk since any disk lost in a RAID-0 gives a total loss.

I fully understand what striping is.




  
  LOG IN TO REPLY
pwm2
"Sorry for being a noob"
Avatar
8,626 posts
Likes: 3
Joined May 2007
Location: Sweden
     
May 28, 2014 15:37 |  #10

morph2_7 wrote in post #16936385 (external link)
It is relevant. Software RAID taxes CPU power while hardware RAID uses its own CPU. You may gain a little bit in data access at the cost of CPU time. As I said, OP wouldn't gain much with software RAID while doubling the risk of data loss.

So what does it matter if software RAID taxes CPU power? The thread has not been about issues with high CPU load during disk accesses.

Next thing is that you can't really hardware-accelerate any two-disk RAID-0. There are no extra operations involved, so the load in a 1TB transfer is the load from sending or receiving the data between memory and disk. If you move to RAID-5 or RAID-6, then a hardware RAID controller can off-load the computation of parity and can split a single read/write stream into the n+1 or n+2 file streams to the individual disks.

The issue here is still not software RAID, but that the OP wanted to RAID-0 two 3TB disks to get 6TB which means that the OP wanted to run striping. A concept that was designed to increase transfer rates at a time when disks had very low transfer rates. But a concept that has a large number of users learn the disadvantages first when it's time to look for the backup because a single-disk failure has just trashed all data.

I fully understand what striping is.

So why then involve software/hardware RAID? Striping the disks is still as much a failure if using a dedicated hardware RAID card. Striping in a workstation might possibly be a good idea as a work disk for video editing. But today's disks manages quite good transfer rates without striping and today we have SSD as an alternative choice.


5DMk2 + BG-E6 | 40D + BG-E2N | 350D + BG-E3 + RC-1 | Elan 7E | Minolta Dimage 7U | (Gear thread)
10-22 | 16-35/2.8 L II | 20-35 | 24-105 L IS | 28-135 IS | 40/2.8 | 50/1.8 II | 70-200/2.8 L IS | 100/2.8 L IS | 100-400 L IS | Sigma 18-200DC
Speedlite 420EZ | Speedlite 580EX | EF 1.4x II | EF 2x II

  
  LOG IN TO REPLY
Diamond-Lord
Junior Member
29 posts
Joined May 2014
     
May 28, 2014 17:56 |  #11

pwm2 wrote in post #16936241 (external link)
Note that backup to tape is very seldom a good idea any more. The tape drives are extremely expensive in relation to how storage they allow. 2.5TB compressed is an irrelevant figure because the on-the-fly compression used is very simple and JPEG, MOV, MP3 and other media formats are normally already using a much more effective compression - so the tape drive either has to write without compression or will instead increase the file sizes when doing the compression.

Backup tape is STILL a very good option, but it depends on total storage needs and accessibility. For overall TB-to-dollar cost ratio, nothing can beat LTO-6 tape storage. For my 10TB of video files, I've only spent $100 on two LTO-6 Ultrium cartridges and the cheapest hard disk storage for that same 10TB would cost at least $400. So please tell me again how hard disk storage is more economical than LTO tape?

However, if the OP needs near-online access to his stored files then LTO is probably not the best solution. The LTO drives are indeed expensive (LTO-6 units start around $2,000). So I agree that for the OP only needing to store up to 5 TB of files, then LTO would not be an economical solution, although renting an LTO-6 drive for a couple days is a workaround that most people can afford.

If anybody out there can provide a means to reliably long-term archive 6 TB on a media that costs less than $50, I'd love to hear about it. LTO tape will soon be able to store 10TB per cartridge for the same price (within 1-2 yrs). Remember, this is for very long term storage needs only. Some of us actually need that.




  
  LOG IN TO REPLY
pwm2
"Sorry for being a noob"
Avatar
8,626 posts
Likes: 3
Joined May 2007
Location: Sweden
     
May 28, 2014 18:30 |  #12

Diamond-Lord wrote in post #16936759 (external link)
Backup tape is STILL a very good option, but it depends on total storage needs and accessibility. For overall TB-to-dollar cost ratio, nothing can beat LTO-6 tape storage. For my 10TB of video files, I've only spent $150 on LTO-6 cartridges and the cheapest hard disk storage for that same 10TB would cost at least $400. So please tell me again how hard disk storage is more economical than LTO tape?

Price of cartridges alone is irrelevant since they can't be used without a drive. And the drive quickly gets out-of-date. And the user needs a backup plan if the drive fails. And the user needs a backup plan if the main storage fails since it isn't feasible to directly mount and use the tapes.

For low backup volumes - 10TB is a small volume - the drive is too expensive. For large backup volumes, you do want an autoloader too which further pushes the fixed cost. And with a large volume to backup, you suddenly realize you want more than one drive.

Small or large, it's still hard to scale away the hardware costs, making the media cost the important piece. And that still doesn't cover the human costs of changing tapes in drives or autoloaders. Let's settle for 30min/week and $50. That's $2500/year for switching tapes.

Want off-site storage? Either extra time + travel costs to move the tapes off-site. Or extra drive+auto-loader+regular visit to have a tape system off-site. An off-site disk solution can be happy with electricity+network+co​oling and a visit every 1-2 years.

However, if the OP needs near-online access to his stored files then LTO is probably not the best solution. The LTO drives are indeed expensive (LTO-6 units start around $2,000).

If anybody out there can provide a means to long-term archive 2.5TB on a media that costs less than $40, I'd love to hear about it. LTO tape will soon be able to store 5TB per cartridge for the same price (within 1-2 yrs).

But the media cost is still mostly irrelevant since the total cost tends to be too high.

Home users normally go for disk-based solutions.
Companies normally go for disk-based solutions.
Web-service companies normally go for disk-based solutions.

There just might be one or more reasons why disk-based solutions are so popular even when tape media is so cheap.


5DMk2 + BG-E6 | 40D + BG-E2N | 350D + BG-E3 + RC-1 | Elan 7E | Minolta Dimage 7U | (Gear thread)
10-22 | 16-35/2.8 L II | 20-35 | 24-105 L IS | 28-135 IS | 40/2.8 | 50/1.8 II | 70-200/2.8 L IS | 100/2.8 L IS | 100-400 L IS | Sigma 18-200DC
Speedlite 420EZ | Speedlite 580EX | EF 1.4x II | EF 2x II

  
  LOG IN TO REPLY
PLLphotography
with the TF
Avatar
5,208 posts
Gallery: 6 photos
Likes: 1024
Joined Apr 2009
Location: VA
     
May 28, 2014 18:31 |  #13

I pay like $50 a year for crashplan. Unlimited cloud storage.


Phillip - phillipwardphotography​.com (external link) | Instagram (external link) | Donate to POTN

  
  LOG IN TO REPLY
Diamond-Lord
Junior Member
29 posts
Joined May 2014
     
May 28, 2014 18:49 |  #14

pwm2 wrote in post #16936832 (external link)
Price of cartridges alone is irrelevant since they can't be used without a drive. And the drive quickly gets out-of-date. And the user needs a backup plan if the drive fails. And the user needs a backup plan if the main storage fails since it isn't feasible to directly mount and use the tapes.

All valid points, but there again what are the exact long-term storage needs of the OP?

Having the tape drive go obsolete is irrelevant if you rent a unit to offload big files from hard disk to tape for a one-time job, like I did in 2009. If the main hard drive fails? Go out and buy a new HDD for cheaper cost like anyone else would.

For specific long-term storage where backed up files don't need to be accessed often or even at all, LTO-6 is still the best solution for individual consumers or enterprise. I have 10TB+ of video files from the 90's when I owned a small videography business and I haven't accessed those files in a few years, so I need the best solution for long-term storage. LTO-6 is by far the least expensive solution for me, not 10TB worth of spinning disks that arent rated as reliable as tape for 30+ years of storage.




  
  LOG IN TO REPLY
On ­ the ­ loose
Junior Member
21 posts
Joined Jan 2014
     
May 28, 2014 19:08 |  #15

Now that disk drives are so cheap it makes sense to copy files to a second hard drive. I keep a drive dedicated as working photographs, ie any that I am likely to need. All on a 4T drive. I copy these files to a second drive on the system and files I might want remote access to live on a server drive accessable from the internet via password, but under my control. Othere images live on an external drive connected only when required and so likely to last longer than a drive always on with the computer. I also copy all my images onto 500mb external drives to archive them. These drives are stored away from my house. These archive drives are kept in a sealed plastic box with some silica gel to remove the possibility of corrosion or damage from high humidity. I put jepg files and raw files on separate disks, so at least one copy should survive. I also keep other important files on such drives.

Compared to the cost of equipment the cost of properly storing images is very small.




  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)

3,955 views & 0 likes for this thread
My storage/backup problem and fix... need input
FORUMS Canon Cameras, Lenses & Accessories Canon Accessories 
AAA
x 1600
y 1600

Jump to forum...   •  Rules   •  Index   •  New posts   •  RTAT   •  'Best of'   •  Gallery   •  Gear   •  Reviews   •  Member list   •  Polls   •  Image rules   •  Search   •  Password reset

Not a member yet?
Register to forums
Registered members may log in to forums and access all the features: full search, image upload, follow forums, own gear list and ratings, likes, more forums, private messaging, thread follow, notifications, own gallery, all settings, view hosted photos, own reviews, see more and do more... and all is free. Don't be a stranger - register now and start posting!


COOKIES DISCLAIMER: This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and to our privacy policy.
Privacy policy and cookie usage info.


POWERED BY AMASS forum software 2.1forum software
version 2.1 /
code and design
by Pekka Saarinen ©
for photography-on-the.net

Latest registered member is sdc123
716 guests, 375 members online
Simultaneous users record so far is 15144, that happened on Nov 22, 2018

Photography-on-the.net Digital Photography Forums is the website for photographers and all who love great photos, camera and post processing techniques, gear talk, discussion and sharing. Professionals, hobbyists, newbies and those who don't even own a camera -- all are welcome regardless of skill, favourite brand, gear, gender or age. Registering and usage is free.