Drowning in data; what to do with it all?

Flores wrote in post #10411213
I'm going to chime in with a vote that you look into some kind of network based storage.

if your brave, you can build it yourself, but I'm not familiar with your level of expertise.

for under $1000 you should be able to put something together with a few T of storage, in a RAID configuration (faster, because you have multiple read/write heads on the virtual 'drive', bigger storage space, and usually can be configd so that if a drive fails, you lose no data), a gig-e interface to your network, and no more messing about with plugs again.

of course, this type of system will give you other issues.

is there any particular reason your going with externals in the first place?

its just backup storage of customer files. i try and keep my primary HDD free for whatever we're working on.

we're both pretty good with computers; just don't have the kind of time i used to to look up solutions and tweak all the settings and drivers. every minute i mess with figuring this out is a minute i can't work on something that we're being paid for.

http://www.amazon.com …et-Attached/dp/B001FAE64K

the 8T version might be more than you need... ever. but the smaller system might be what your looking for.

there is going to be some initial setup involved with this, it's not 100% plug in play, as it's basically a stand alone server dedicated to storage, but once you get it fired up and configured, you should be able to just push/pull backups off it, and just let it run. Since it's also a content server, if you have a PS3 or apple TV or directv with ethernet on it, you can use your television as a slideshow display off it as well, without having to do anything but select the directory you want to 'play'.

I built one similar to this a few years back, and I have a several thousand pictures on it, and I archive my VHS and DVD collection onto it. beats having to look for a disc when I want to watch something.

I agree. Some of these NAS boxes are fairly decent. You can have RAID1 or whatever - better than single external drives which could fail. They usually let you plug in a usb device to dump data to for portable backup (not fast, but it doesnt require your pc so you can just let it run).

well, i have the drives communicating, but really slower than i wanted.. goin around 20-22mb/sec this could take weeks. I'm half tempted to just pull all the hard drives out and stick em in the tower and just make one huge raid 0, organize all the files, then dump them onto 2 identical drives, remove and re-enclosure them.

i like the server idea, but its not much faster than what i've got now (about twice as fast theoretically, but... why can't they have usb 3.0 stuff readily available... i can't wait from some reliable plug and play with decent speed behind it.

dwarfcow wrote in post #10411429
well, i have the drives communicating, but really slower than i wanted.. goin around 20-22mb/sec this could take weeks. I'm half tempted to just pull all the hard drives out and stick em in the tower and just make one huge raid 0, organize all the files, then dump them onto 2 identical drives, remove and re-enclosure them.

i like the server idea, but its not much faster than what i've got now (about twice as fast theoretically, but... why can't they have usb 3.0 stuff readily available... i can't wait from some reliable plug and play with decent speed behind it.

Well, eSata is probably the fastest your realistically going to be able to get on an external. even with some ultrafast USB, the DRIVE itself is still 'only' 3Gbs transfer rate. I use that on my home made storage system to transfer from a 1T portable drive, because USB is too fricken slow when you want to move a lot of data in a hurry.

The only way to go any faster is to use the SATA RAID built into your motherboard. at which point, if you dont want to the drives IN your machine (external storage). your left with Gig-E as the transport mechanism. which is slower than SATA, but quicker than USB.

if your having performance issues, it may be an outdated or mangled driver for the motherboard. if it's an intel board, hit intel's website and look at updating the drivers (i'm ass-uming windows). You may also want to check that you don't have some auto-indexing system turned on for external drives, as that is going to kill you until you disable it or it completes it's indexing

dwarfcow wrote in post #10411429
well, i have the drives communicating, but really slower than i wanted.. goin around 20-22mb/sec this could take weeks. I'm half tempted to just pull all the hard drives out and stick em in the tower and just make one huge raid 0, organize all the files, then dump them onto 2 identical drives, remove and re-enclosure them.

i like the server idea, but its not much faster than what i've got now (about twice as fast theoretically, but... why can't they have usb 3.0 stuff readily available... i can't wait from some reliable plug and play with decent speed behind it.

USB2 is specified to 480mbps of bandwidth. (57.2MB/s) Taking into account usb transfer overheads realistically puts the transfer rate down to about 25-30MB/s. This is also for only one device attached to that channel. Add in the file transfer overheads and 20-22MB/s seems about right.

Transfer rates also slow down when copying multiple small files versus one large one.

The hard drives themselves can easily saturate the bandwidth of USB2. The sequential read rates measure eg. Samsung spinpoint F1 1TB drive measures from 118MB/s to 56MB/s depending on how far the read heads are reading from the center of the spindle.

Your best bet here for now is to do exactly as you said. Connect the drives directly to the sata bus on the motherboard. Sata 2 has a 3Gbp/s bandwith. Easily enough to cover the read rates of the hard drives of today.

If you can get the ESata working you will be better off as well without having to pull enclosures apart.

No other external USB2 drive solution will do any better transfer speed wise they will all hit the USB2 bandwidth bottleneck.

If you want to stick with the USB theme. Grab some drives, usb3 external enclosures and a usb3 card for your pc. USB3 should easily cover hitting the max read and write speeds the drives can push through it.

Otherwise go the networked file server as suggested.

I had the same aggrivation as you did with usb2 external drives.

I now run a seperate fileserver over a gigabit network. It does a bit more than storage but for our purpose here it runs a Hardware Raid 5 volume for my photos. I back up this photo repository onto a 1.5TB drive using a sata hotswap caddy installed on my fileserver. I do my backups when I add new photos onto the raid volume. I keep the backup drive tray in a safe place when not in use.

I get transfer speeds of around 100MB/s between my pc and server, the limit of the gigabit network. Lightroom runs as if I have the files directly on my pc.

Cost me a bit especially for the hardware raid cards, drives and 5in3 drive bays but I am happy. I have plenty of room for expansion later down the line, high transfer speeds and a backup of my files.

2 tb's are going down A LOT in price. (saw one 2tb for $80 yesterday), I suggest hanging in there until slickdeals.net comes up with a killer deal and then just buy 2x2tb. Fill up one drive with all your data and backup everything onto the second.
Not sure if you'd save more if waiting for one fast drive to use in the pc and one as backup, up to you.

Western Digital drives are top notch. Seagates failure rates of the + 1 gb drives still keeps me away. The WD Mybooks did have an issue with the controller cards in the unit, not the drive but the enclosure.

I am running a Mac set up, I have 2 - 1.5tb drives in one enclosure using firewire 800. First is my Time Machine back up, second is my photo storage. This entire enclosure is then backed up to my Windows Home Server that has 5 TBs of space. I also use WHS as a storage drive for misc files that are not photo related. Works well for me, is cheap and productive. Get any old PC box, install WHS on it, load it up with drives and your good to go.

BTW: Here's an awesome video on photo/video data capture, storage and backup workflow. Maybe overkill for your own data needs, but a very worthwhile watch regardless of the scale of your data:

http://www.youtube.com …Y&feature=playe​r_embedded

alright, well... i think i got some of my external issues somewhat resolved.

Turns out all the new externals were formatted as NTFS with Archiving turned on; i formatted them all as exFAT, and turned off archiving (NTFS wasn't recommended by alot of people for external use) the new drives are now transferring at around 22-24 as oppose to 12mb/sec it is still slow transferring from one drive to another (around 14-16mb/sec) but i doubt i will have to do that much once i get this junk organized and consolidated.

I took my eSata WDbook and hooked it up to our laptop, and it plugged n played just fine; which, while relieving just annoys me even more for some reason (stupid desktop #@!$); I'm not sure if the heating issue will be resolved, I've got a test copy of 200gb going on to it, wanna see if it hangs up while xfering. however it's not that much faster than usb 3 (its going around 35mb/sec; so much for 3.0gbits/sec [375mb/sec]) and i can't imagine the HDD, even at 5400rpm's would cap at 280mbits/sec transfer rate (and i think it was advertised as being a 7200rpm WD drive.)

dwarfcow wrote in post #10408810
and i have not been able for the life of me to get the eSata to work (rated at 3gbits/sec). The motherboard recognizes the drive on eSata, but i can't get windows to acknowledge its existence; either in explorer or in disk manager.

If you can see it with the motherboard then I assume you know how to get into the BIOS?
If so check if the hard drives are set in ACHI mode, or IDE compatibility mode.
If its the later you will see eSATA connected drives in BIOS, but may have trouble seeing them in Windows and you won't be able to hot swap.

If it is set to IDE compatibility then only a reinstall can solve it as Windows has to install a different set of drivers.

As for your storage problems, it sounds like your not really making the best use of your current Hard drives and need to expand.

The fastest way to access photos will always be on an internal drive, but you need to keep externals for back up.

I run 4 drives in my computer and 2 externals.
A 500gb and 320gb hold the OS, programs, music, movies, games etc.
The 320gb drive also gets used as a photoshop scratch disc.
I then use two 1TB drives for my photos and LR catalog.
One is my working drive, it's were photos get downloaded too off the camera and where I load them from for editing then save back to.
Its broken into different folders and the structure is matched in LR.
The second drive is an internal back up. i.e. A quick and easy to access back if something happens to the other drive or I stuff up a photo. It is an exact copy of the working drive.

The two external drives are also 1TB each and both exact copies of the working drive as well. They are updated using an eSATA docking station.

When the 1TB drives get full I will simply replace them with 2TB drives and carry on. By the time they get full 4TB should be available, and after that, we'll it will be time for a new computer anyway, and who knows, we might using quantum physics to store data in a different reality.


This works very well for me, and maximizes use of the computers resources by spreading reading and writing across different drives.

Install HotSwap
http://mt-naka.com/hotswap/index​_enu.htm

And it should help with working with eSATA drives.

Basically it will "Scan for Hardware Changes" which will in turn help the OS recognize that you now have a new "Internal" SATA HDD and it should work just fine. Then when you are done you should "unmount" it before unplugging or swapping it.

Oh and like someone else suggested. You need to get an eSATA HDD dock and invest in bare internal HDD. You can store them in anti-static storage boxes like these:
http://www.amazon.com …TA/ref=dp_cp_ob​_e_title_3

You can get 3 of them for like $15 from eBay.

Off course you should use a good back up s/w (I use MS SyncToy coz its free) and think about mirrored back up with off site storage.

dwarfcow wrote in post #10419436
alright, well... i think i got some of my external issues somewhat resolved.

Turns out all the new externals were formatted as NTFS with Archiving turned on; i formatted them all as exFAT, and turned off archiving (NTFS wasn't recommended by alot of people for external use) the new drives are now transferring at around 22-24 as oppose to 12mb/sec it is still slow transferring from one drive to another (around 14-16mb/sec) but i doubt i will have to do that much once i get this junk organized and consolidated.

I took my eSata WDbook and hooked it up to our laptop, and it plugged n played just fine; which, while relieving just annoys me even more for some reason (stupid desktop #@!$); I'm not sure if the heating issue will be resolved, I've got a test copy of 200gb going on to it, wanna see if it hangs up while xfering. however it's not that much faster than usb 3 (its going around 35mb/sec; so much for 3.0gbits/sec [375mb/sec]) and i can't imagine the HDD, even at 5400rpm's would cap at 280mbits/sec transfer rate (and i think it was advertised as being a 7200rpm WD drive.)

This,

http://www.newegg.com …rt-_-59-105-563-_-Product

sir would solve all of your problems. the initial investment may seem high, but the returns are worth it. I have one with one 500gb HDD and two 1TB HDD drives that mirror my images across all three drives. I also store my music on it so I can access it from anywhere in the house. I also have a ton of movies on it that I can stream to my PS3. There's also options to set it up for remote access over the web. It does automated backups and will even work with time machine.

Definitely one of the best photography related purchases I've ever made.

Got an old PC laying around? Build an UnRAID server!

http://www.lime-technology.com/