Hard drive upgrade issues (be careful)

[Crypto]

I use two hard drives for my back up (same files on both), one internal and one external. Well, they were becoming full and I decided to upgrade to larger 250GB drives. So I decided to put the new drives in a USB 2.0 enclosure and drag the entire contents of the old drive to the new drive. I simply highlighted all the files on the old drive and drug them to the new drives.
The other day I needed a file from my backup. I tried to copy it over and I kept getting a CRC (Cyclic Redundancy Check) error. Which normally means it is a corupt file. I ran the disk error tool and cleared up the error, tried to open the file, it said it was corrupt. At this point I became worried about my images, so...I decided to look over my images to see if there were any problems. BTW - I shoot all RAW and I was viewing them using RSE. I noticed the thumbnails on several images were all screwed up, sure enough, I had corrupt images too. So I went to my second drive to get the image I wanted, I got it, but I noticed more images were corrupt.
Luckily, I still had the old drives in a box. So I could pull all the good images off the old drive and replace the corrupt files. I WAS VERY LUCKY!
These RAW images actually opened with NO problems using the "Microsoft RAW Image Thumbnail Viewer". But if I drug them into PSPX or RSE or DPP, they were corrupt. Also, I found a couple JPGs that were corrupt, they opened using the microsoft viewer, but were corrupt when opening them using PSPX.
I know this is long, but I wanted to share with everyone that you can't always count on windows to transfer large files using this process, and PLEASE hang on to the original drive until you are sure your files are ok!!!!

[PacAce]

So, did you ever figure out why your files on the new drive got corrupted? And did you say that the files in the internal drive got corrupted, too? That's what I gathered since you said you had to go back to the old, replaced drive to restore your files from.

[Crypto]

So, did you ever figure out why your files on the new drive got corrupted? And did you say that the files in the internal drive got corrupted, too? That's what I gathered since you said you had to go back to the old, replaced drive to restore your files from.

No, The only thing I can figure is it was caused by selecting such a large amount of data at one time to transfer.
Yes, the files were corrupt on the internal drive too. The internal drive had the files transfered via USB enclosure prior to putting it in the PC on the secondary IDE. So both drives had the files transfered using a USB enclosure.

[gjl711]

When backing up files avoid drag and drop as D&D = toss and hope in that you write the file to the new location and hope all went well. A network glitch, bad spot on the drive, or just a random bit change can ruin a file. A much better method is use one of a myriad of backup programs and enable file verification. It takes much longer as each block is re-read and compared to the original for data integrity, but at least then you know you have a good copy of the original. If using Windows, DOS has a flag you can set as well. I believe it is something like “copy /v ‘files to copy’ ‘to location’”

[Mathiau]

^^ good advice, if your hard on time, do it when you go to bed.

cant really say it was windows, it could of been the drive, or a bad USB port. could be anything.

this is why it is good to use a RAID 1 as a storage drive + DVD or tape backups if your pictures are that important, one drive goes bad, you got the other.

ideally, if you can delete data - it is not a backup solution.

i did experience something like this myself, i had a 74g SCSI drive and was moving my pics over to a IDE drive, about 15 JPEG's became corrupted, colors were weird and off, turned out it was the RAID controller card i was using.

It is also good to do a copy, and not a cut, this way you can go back and check to make sure everything was copied over fine.

[DavidW]

It sounds like some kind of controller issue - many of the controllers in USB enclosures aren't what they should be. I've had similar issues with some memory card adapters and some flaky SD cards. The big transfers aren't the problem - they just work the controller hard and reveal the underlying flakiness. I always work new storage setups hard (especially external hard disks or memory cards of any sort) before relying on it in an attempt to reveal any underlying flakiness.

The RAW Image Thumbnailer can be misleading, as it looks as the thumbnail in the RAW file, not the RAW data itself.


One of the neat features of Beyond Compare (http://www.scootersoftware.com) is that you can compare files between two locations. Ordinarily it just compares metadata, but there's a feature for binary or CRC comparisons. It's a very powerful piece of software for Windows users.



David

[neil_g]

drag and drop of large amounts of data in windows shouldnt have any issues, ive done it many times on GB's worth of data with no problems.

sounds like a hardware to software error of some description..

[gjl711]

I agree that data corruption as pervasive as the one described is most likely a failure of hardware but that could have happened during the Drag and Drop copy. D&D is a very convenient was to move data and in most applications it is just fine, but moving data always incurs errors. Depending on media, it can be anywhere from 1 bit in 10 meg from floppies to 1 bit in 10 gig for FTPs. But bits fail, no transmission is perfect. Most of the time the bit will be buried in image data thus it will be subtle and unseen but on occasion it can hit a file header thus corrupting the file. My point was that if the data is important, use a method that validates data integrity and recopies if necessary. In most cases it is a simple change of procedure or tool. Backups are funny things, kind of like house insurance. Most people will never have to make a claim their whole lives, but finding out that your insurance is flawed the day after the tornado blew away your house is not a pleasant shock.

So, as a former systems administrator I have the following advice:

Back up often. If you just came back from a critical shoot, back it up. You’re not going to be able to go back and re-capture those pics.
Use a method that verifies data integrity thus assuring that your backup is a flawless copy of the original.
Use two backup methods, short term and long term. A removable HD is a great short term backup as is a spare machine. And DVDs make great long term storage.

[Zepher]

I agree that drag and drop shouldn't be an issue, as I do it all the time moving thousands of files amongst all my drives to organize my stuff.
The other day I moved around 300gigs of files between 6 of my drives, all drag and drop.

[Crypto]

I agree that drag and drop shouldn't be an issue, as I do it all the time moving thousands of files amongst all my drives to organize my stuff.
The other day I moved around 300gigs of files between 6 of my drives, all drag and drop.

Did you move all three hundred gigs with one C&D action? WOW

[Zepher]

Did you move all three hundred gigs with one C&D action? WOW

No, it was a total of 300gigs. most at one time was about 65,000 files totalling 20gigs.
I had one folder with 50,000 files in it, took about 5 or so minutes for the files fo show up in the window after opening the folder.

[Crypto]

If you transfered large files within that 20 gigs, I would check them just to be sure. I never thought twice about this before and I've been doing it for a long time, but now I'm a little skeptical.
I first noticed the problem when I tried to pull a 200 MB program file off my drive.
It could be all hardware, but who knows.

[peacock]

Forgive me if I am asking people to restate the obvious , but what is the best method of copying the data from one HD to another , I have 200gb stored on a 250gb hard drive and just bought 2x 400gb drives to store/ backup my files on with room for future expansion.
again sorry if t sounds daft .

[Jesper]

In Windows you could do it with xcopy:

1. Open a command prompt by selecting Start / Run..., type in "cmd" and click OK.
2. Use a command like this to copy the entire contents of drive D: to drive E:

xcopy d:\*.* e: /E /V

The /E means: copy subdirectories, even empty ones, and /V means verify the copied files. Type "xcopy /?" to see a list and description of all the available options.

When I make backups on CD or DVD, I always enable the "Verify Data" option in my burning program. Occasionally (maybe one in twenty?) I have a bad CD or DVD and the verification fails.

[peacock]

Many thanks for the reply , how would / might this benefit me more than simply copying and pasting from one drive to another ?

[Mathiau]

because windows will verify the written data and that it si good, also because you are copying the data, instead of cutting it, if a file is bad you can go back and try to copy it again with out worry of losing the file(s)

[JimAskew]

Hi,

I use Acronis to back up my files...see the link below:

http://www.acronis.com/homecomputing/products/?source=google&ad=acronis&s_scid=acronis|544035737&gclid=CIj-2viDyYcCFQU8UAodJHO2Gw

I have a "mirror" external drive of 250G that I use plus two more external HD that iIused for redundancy ofr critical files.


Every Sunday at 0200 hours Acronis "wakes up" and backs up my C drive tothe redundant drive.

Acronis gives me two capabilities: it mirrors my entire C drive so I can restore it completely should it fail and it creates a bootable CD for me so I can boot from the CD Drive and run from the backup until I can replace a "crashed" HD.

All this helps me to sleep better at night :)

[Mathiau]

For sure, amazing how far along back-up software has come these days.

[gjl711]

I use the backup utility that comes along with windows. It is not the most elegant thing but covers all the bases. It works well across a network and has the ability to verify data as it is written. Secondly, on the backup machine, I restore the data thus confirming that the backup file is good. Lastly, for long term backups, I write out the backup file to a DVD again with verify.

[peacock]

interesting stuff thanks:)

[Mathiau]

I use the backup utility that comes along with windows. It is not the most elegant thing but covers all the bases. It works well across a network and has the ability to verify data as it is written. Secondly, on the backup machine, I restore the data thus confirming that the backup file is good. Lastly, for long term backups, I write out the backup file to a DVD again with verify.


ntbackup - works great, and if you team it with a veritas suite program, you got some serious back up power going on! use it for our works exchange server.

[gjl711]

ntbackup - works great, and if you team it with a veritas suite program, you got some serious back up power going on! use it for our works exchange server.

I have been using ntbackup for years without flaw on both my, and my wife’s machine. I have never had to use it, knock on wood, but I have on my wife’s machine as she had a hard drive fail. Everything recovered flawlessly and easily. I will look into the veritas suite though, thnax.

[Crypto]

don't these backup programs create proprietary files that require their software to open up the files? I tried a couple and hated that about them. Why not just back up the actual file in its original format? I like knowing my files can be read on any computer without the software. Especially if my PC dies. Just my opinion

[gjl711]

don't these backup programs create proprietary files that require their software to open up the files? I tried a couple and hated that about them. Why not just back up the actual file in its original format? I like knowing my files can be read on any computer without the software. Especially if my PC dies. Just my opinion

That is why I have stuck with the standard backup program that comes with windows.

Secondly, I do uncompress (restore) the files on my backup machine giving me access to the individual files and having the compressed file to write out on dvd.
Secondly, weekly I do incremental backups saving only the new files, or the files that have changed. Ntbackup does this automatically. Lastly, I have this scheduled so there is nothing for me to do. Every Friday I turn on both machines and at 2 am it backs itself up and does some cleanup as well. Saturday morning I turn off the backup machine for the week.


Once setup this scheme pretty much takes care of itself. About once a quarter I write out the latest full backup to dvd and store it in my desk at work.

[Mathiau]

at the same time, it kind of sucks we have to worry so much about backing up our data, think the day will ever come where harddrives, or storage media wont fail? or at least wont fail with out giving us notice.

Smart Monitoring seems to be a useless technology now a days.

[gjl711]

Well, that can be done today using either a RAID1 or RAID0+1 array. This greatly reduces the chances of data loss but is not a 100% guarantee. Here in the Midwest when the thunderstorms come a rumbling through, a nice 10,000,000 to 100,000,000 volt spike at 1000 to 300,000 amps at the local transformer does wonders to electronics. I guess the message should be, if you can’t afford to loose it, back it up.

[FlyingPete]

In Windows you could do it with xcopy:

1. Open a command prompt by selecting Start / Run..., type in "cmd" and click OK.
2. Use a command like this to copy the entire contents of drive D: to drive E:

xcopy d:\*.* e: /E /V

The /E means: copy subdirectories, even empty ones, and /V means verify the copied files. Type "xcopy /?" to see a list and description of all the available options.

When I make backups on CD or DVD, I always enable the "Verify Data" option in my burning program. Occasionally (maybe one in twenty?) I have a bad CD or DVD and the verification fails.

Another excellent command line tool is robocopy (Robust Copy) it came (comes?) with the Windows resource pack, and is readily availalbe on the web for free download.

It has data verfication capablilties and can "mirror" sources and destinations, I use it with some scripts for all my photo backups. Command line syntax is simular to xcopy but with more features.

[Zepher]

If you transfered large files within that 20 gigs, I would check them just to be sure. I never thought twice about this before and I've been doing it for a long time, but now I'm a little skeptical.
I first noticed the problem when I tried to pull a 200 MB program file off my drive.
It could be all hardware, but who knows.

I routinely transer 700-1400meg files between my drives all the time.
mainly movies and tv shows and never had one problem with drag and drop ( I use the "Move Here" option.

I did have one issue with one hard drive a few years back that was going bad.
portions of images would be currupted and mp3 files would have skips and crackles in them.

[Mathiau]

if you can’t afford to loose it, back it up.


Words to live by! and this doesnt just mean to a DVD or something either :)