Approve the Cookies
This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and our Privacy Policy.
OK
Index  •   • New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Guest
New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Register to forums    Log in

 
FORUMS General Gear Talk Computers 
Thread started 12 Sep 2011 (Monday) 21:03
Search threadPrev/next
sponsored links
(this ad will go away when you log in as a registered member)

Server based storage and back-up solution help

 
Mike ­ Photo
Senior Member
Avatar
531 posts
Gallery: 1 photo
Likes: 39
Joined Apr 2006
Location: Boston, Ma
     
Sep 12, 2011 21:03 |  #1

Hello all,

I'm looking for thoughts and ideas for small scale server back-up solutions for a client of mine. I have some experience with setting up back-up solutions but this client doesnt have the budget for what ive done in the past. Before i bring my ideas to the table I thought id see what other people have tried and may or may not have had success with.

What id like to achieve is a centralized server that stores his entire catalog of images and is possibly expandable in the future as it grows. Something like 6-8Tbs in a Raid 5 array with some sort of mirrored back-up for off site. This way each computer could have access to all files and be backed up to the server and the server would have redundancy.

I was also thinking he could then use his current external HDD system as local back-ups for each computer. So essentially each computer would have a external mirror and be backed up to the server.(for non image based files or working edit files)

So please any thoughts or suggestions would be great.


Instagram (external link)
MIKE PHOTO

  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)
tim
Light Bringer
Avatar
51,009 posts
Likes: 369
Joined Nov 2004
Location: Wellington, New Zealand
     
Sep 12, 2011 21:13 |  #2

You need to tell us more about the client, what they do, how many PCs, data volumes, etc.

Working with images on a server is significantly slower than working with local images. Using the server as backup is better than as a working drive.


Professional wedding photographer, solution architect and general technical guy with multiple Amazon Web Services certifications.
Read all my FAQs (wedding, printing, lighting, books, etc)

  
  LOG IN TO REPLY
Mike ­ Photo
THREAD ­ STARTER
Senior Member
Avatar
531 posts
Gallery: 1 photo
Likes: 39
Joined Apr 2006
Location: Boston, Ma
     
Sep 12, 2011 22:25 |  #3

Thanks tim.
Essentially the server would be an archive accessible by all computers no one would be working off the server itself. 4-5 computers could search the servers catalogs for an image if needed but the one editing computer would pull images off the server to its own hdd and backup system. all new content would be copied to the server and edited on the PP computer, then the edits would be eventually copied to the server as they were finished.

So what im saying is the server itself would only be accessed for archived data which would then be pulled off for use or further editing. and each computer station would have its own external back up Drive.

The reason for the system is the photographer has a massive amount of old work which is scattered among dozens of HDDs and every time we need to look for an image we have to go plugin drives and search for them. I want to centralize the work for searching. Also he has millions of slides that will slowly over time be picked through and digitized for stock work.

most the computers will only have office type work backed up so figure 3-4 small hdds would be backed up on the server(just the files not the whole system). the editing station might not need to be backed up since it will have a mirrored external drive and the work would be backed up to the server archive as its finished.

Hope thats more info for you.


Instagram (external link)
MIKE PHOTO

  
  LOG IN TO REPLY
tim
Light Bringer
Avatar
51,009 posts
Likes: 369
Joined Nov 2004
Location: Wellington, New Zealand
     
Sep 12, 2011 22:49 |  #4

I'd be aiming to get rid of all external disks. Have one moderate powered server, nothing too fancy, hooked up to a massive disk array and gigabit ethernet or better. Copy all the data to the server, and put the old drives in a storage room.

All computers should back up to the server, not external disks. External disks fail.

That massive file server will have to be backed up with tape, there's not really another practical option.

This won't be particularly cheap, but shouldn't be too expensive either. An i3/i5/AMD computer, some kind of enclosure for disks, a raid card, and many, many disks. Add a tape drive and some tapes.

Backups and organisation of the work will be important.


Professional wedding photographer, solution architect and general technical guy with multiple Amazon Web Services certifications.
Read all my FAQs (wedding, printing, lighting, books, etc)

  
  LOG IN TO REPLY
Mike ­ Photo
THREAD ­ STARTER
Senior Member
Avatar
531 posts
Gallery: 1 photo
Likes: 39
Joined Apr 2006
Location: Boston, Ma
     
Sep 13, 2011 01:49 as a reply to  @ tim's post |  #5

I was thinking of building a small i3/i5 server with a hot swappable bay case, It should be more then powerful for this purpose. I dont have much experience with raid cards can you recommend any affordable ones?

Other then tape what ways could the server be backed up? could you back it up with a Nas server like a 5 bay raid enclosure?

Also do you think 2 Nas servers mirrored could be an alternate solution? one could be moved off site and backed up once a week.


Instagram (external link)
MIKE PHOTO

  
  LOG IN TO REPLY
tim
Light Bringer
Avatar
51,009 posts
Likes: 369
Joined Nov 2004
Location: Wellington, New Zealand
     
Sep 13, 2011 02:16 |  #6

I think AMD might be a better choice for this. They're cheaper and perform fine.

I don't know anything about specific RAID products. I don't think you want hot swappable anything, I think you want a big tower case with fans and plenty of space for drives. You want it to be easy for people who don't understand IT.

Moving something that heavy isn't practical. Tape backup is easy. Again I can't recommend a specific product, just the idea.

This is bordering on needing an experienced storage designer to help with it.


Professional wedding photographer, solution architect and general technical guy with multiple Amazon Web Services certifications.
Read all my FAQs (wedding, printing, lighting, books, etc)

  
  LOG IN TO REPLY
Moppie
Moderator
Avatar
15,045 posts
Gallery: 19 photos
Best ofs: 1
Likes: 343
Joined Sep 2004
Location: Akarana, Aotearoa. (Kiwiland)
     
Sep 13, 2011 14:13 |  #7

tim wrote in post #13093918 (external link)
That massive file server will have to be backed up with tape, there's not really another practical option.


Have you seen the cost of Tape based back up systems, and how un-reliable they are?

A couple of 2TB drives is faster to back up to, and more reliable.


flickr (external link)

Have you Calibrated your Monkey lately?

Now more than ever we need to be a community, working together and for each other, as photographers, as lovers of photography and as members of POTN.

  
  LOG IN TO REPLY
tim
Light Bringer
Avatar
51,009 posts
Likes: 369
Joined Nov 2004
Location: Wellington, New Zealand
     
Sep 13, 2011 16:20 |  #8

Moppie wrote in post #13097288 (external link)
Have you seen the cost of Tape based back up systems, and how un-reliable they are?

A couple of 2TB drives is faster to back up to, and more reliable.

I don't know how much they cost, but huge enterprises back up to tape, so they must be reasonably reliable. A quick look around suggests $1000 - $10,000, so they are a bit expensive.

Backing up an 8TB (and growing) storage system to 2TB disks is problematic and is asking for trouble for a business that isn't IT savvy. How do they split the files across the disks? What if one disk gets dropped? How do they update the backups? How often will they update the backups given they have to bring a pile of 3-6 disks in?

So tape is expensive, and disk is bordering on impractical. I'm not sure what the best solution is.


Professional wedding photographer, solution architect and general technical guy with multiple Amazon Web Services certifications.
Read all my FAQs (wedding, printing, lighting, books, etc)

  
  LOG IN TO REPLY
tekkie
Goldmember
Avatar
2,621 posts
Likes: 1
Joined Dec 2006
Location: Clarkston, MI
     
Sep 13, 2011 17:39 |  #9

Moppie wrote in post #13097288 (external link)
Have you seen the cost of Tape based back up systems, and how un-reliable they are?

A couple of 2TB drives is faster to back up to, and more reliable.

definitely incorrect tape drives are very reliable, in the tens of thousands of tape backups we have gone through I remember one tape being bad

expensive yes

there is no reliable / cheap solution for 8TB of data that I am aware of other than backing up to the cloud but thats going to take forever unless they have some crazy bandwidth

the real solution is a LTO 3 / 4/ 5 tape library but clearly that is out of the question


Canon 1DMKII, 7D, 5DMKII, 1D MKII
Canon 500L, 100-400L
, 70-200 2.8L, 17-40L, 24-105L, 24mm 2.8,50mm 1.4, 85mm 1.8, 135mm 2.8 SF, 100mm Macro
Canon 430 & 580 flashes

  
  LOG IN TO REPLY
Mike ­ Photo
THREAD ­ STARTER
Senior Member
Avatar
531 posts
Gallery: 1 photo
Likes: 39
Joined Apr 2006
Location: Boston, Ma
     
Sep 13, 2011 22:00 as a reply to  @ tekkie's post |  #10

Thanks for all the tips guys. I have used tape systems in the past for business applications but I'm aware of the initial set up costs and they are out of my budget.

I met with them today to toss around some ideas and Im meeting with a dell rep. next week to see what there thoughts are.

After talking with them today I think I might set up some sort of 5 bay Nas server with 2tb drives. they already have the entire digital archive backed up on external hdds so Ill just transfer all of it to the nas. Then either buy a second nas that can be located in the adjacent house or copy all new data once a week to a 2tb docked drive and move it with the other externals off site.

Its not ideal but its cheap will meet their current needs of having a central archive thats redundant and backed up off site. this way I can back up all the computers to the Nas server.

Any recommendations on backup software for backing up several comps to one server?

Thanks for all the help guys.


Instagram (external link)
MIKE PHOTO

  
  LOG IN TO REPLY
rcfury
Senior Member
Avatar
916 posts
Likes: 1
Joined Dec 2007
Location: Manassas, VA
     
Sep 13, 2011 22:34 |  #11

W2k8 R2 server has a pretty good backup server. You could use the standard windows client backup thats preinstalled to backup the data. Bedsides that you got the business class backup solutions like netbackup or backupexec which requires licences per CPU. Im not too sure what the cost of licences is for 5 computers though.
For my home network i use windows backup server to backup my PCS. This server is hooked up to my 5TB array for all storage. Which is then mirrored to another 5TB array on another server so i have multiple copies of everything.


~Nathan
Gear: Canon 1Dm2, 10D, Canon 17-40L, 70-200 2.8L EF-50 1.4, and the Canon 550EX flash
-----
♦ Check out my FLICKR  (external link) (external link)

  
  LOG IN TO REPLY
Maverick7687
Member
41 posts
Likes: 1
Joined Dec 2006
     
Sep 13, 2011 22:42 as a reply to  @ Mike Photo's post |  #12

Alternatively, you could build a second, very low power, server and store it off-site with an internet connection. Initially copy all the files to 'server2' while on-site, then use an ssh tunnel and sync software of your choice to run nightly/weekly/bi-weekly.
No moving equipment from location to location, which is very likely to be neglected.


Canon 40D | Canon 70-300mm USM IS | Canon EF 50mm 1.8 | Tamron SP 28-75mm 2.8

  
  LOG IN TO REPLY
tim
Light Bringer
Avatar
51,009 posts
Likes: 369
Joined Nov 2004
Location: Wellington, New Zealand
     
Sep 13, 2011 23:05 |  #13

Copying images over the internet is still impractical because of speeds and data limits, unless you have a business class connection with unlimited data. Given an ethernet cable means the backup location is too close to survive a fire, hard drives seem to be the only way to do that backup.

Do you need an automated way to get data from the clients to the server? If it's sets of images then it'll be done on an ad hoc basis anyway. A user should download the images, work on them, then upload the xmp/dng/psd or whatever files back up to the server. It's not automated, but it's practical. I'm not even sure you could automate it.

Backing up the server is different. You have to decide if you'll want to mirror it, or treat the hard drives like big tapes and do differential backups. I personally prefer more of a mirror, but the trick would be dividing data up between the different backup drives. You could potentially do it by year, like I do, but i'm on a smaller scale.


Professional wedding photographer, solution architect and general technical guy with multiple Amazon Web Services certifications.
Read all my FAQs (wedding, printing, lighting, books, etc)

  
  LOG IN TO REPLY
Maverick7687
Member
41 posts
Likes: 1
Joined Dec 2006
     
Sep 13, 2011 23:15 |  #14

That's true, if you are transferring a huge batch at a time it could take a while, hence the suggestion to set it to work overnight. Although if your connection has a low bandwidth cap, you're absolutely correct.


Canon 40D | Canon 70-300mm USM IS | Canon EF 50mm 1.8 | Tamron SP 28-75mm 2.8

  
  LOG IN TO REPLY
tim
Light Bringer
Avatar
51,009 posts
Likes: 369
Joined Nov 2004
Location: Wellington, New Zealand
     
Sep 14, 2011 00:26 |  #15

A 5Mbps upload rate means 2.2GB upload per hour, assuming you hit the limit, which isn't guaranteed. My home connection is 2Mbps which is 800MB/hour, 10Mb means 4.4GB uploaded per hour. Assuming a wedding is 25GB (if you use a 5DII it might be 50GB) that means one 25GB wedding would take around 11 hours to back up over a 5Mbps connection. If you want to upload/backup a catalog of 100 weddings that will take 45 days.

Online backups really aren't that practical yet.


Professional wedding photographer, solution architect and general technical guy with multiple Amazon Web Services certifications.
Read all my FAQs (wedding, printing, lighting, books, etc)

  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)

2,905 views & 0 likes for this thread
Server based storage and back-up solution help
FORUMS General Gear Talk Computers 
AAA
x 1600
y 1600

Jump to forum...   •  Rules   •  Index   •  New posts   •  RTAT   •  'Best of'   •  Gallery   •  Gear   •  Reviews   •  Member list   •  Polls   •  Image rules   •  Search   •  Password reset

Not a member yet?
Register to forums
Registered members may log in to forums and access all the features: full search, image upload, follow forums, own gear list and ratings, likes, more forums, private messaging, thread follow, notifications, own gallery, all settings, view hosted photos, own reviews, see more and do more... and all is free. Don't be a stranger - register now and start posting!


COOKIES DISCLAIMER: This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and to our privacy policy.
Privacy policy and cookie usage info.


POWERED BY AMASS forum software 2.1forum software
version 2.1 /
code and design
by Pekka Saarinen ©
for photography-on-the.net

Latest registered member is rush1981
626 guests, 351 members online
Simultaneous users record so far is 15144, that happened on Nov 22, 2018

Photography-on-the.net Digital Photography Forums is the website for photographers and all who love great photos, camera and post processing techniques, gear talk, discussion and sharing. Professionals, hobbyists, newbies and those who don't even own a camera -- all are welcome regardless of skill, favourite brand, gear, gender or age. Registering and usage is free.