Image Storage and Management

[JCalvert]

I am interested in hearing what people are using for storing and managing their images.

I am using Photoshop CS ( what's there to say, its great) and also ACDSee which is really terrific at sorting them.

Do people generally keep the images on hard drives, or move them off onto CDs / DVDs. If so, how do you know where they are ( i.e. which CDs)?

What size drives are people using and does anyone use RAID 1 or 5?

Do you keep RAW, TIFF and THMs in the same directories, probably not.

Cheers
jc

[robertwgross]

JC, I think you are going to hear all sorts of varying opinions here.

I transfer RAW files from my CF cards to a hard disk directory called "Canon Receiving". Once there, I can run them through the converter to make TIF files at "Canon Converted". Once the RAW files have piled up to be one CD-R full, I burn them to a CD-R. Then the CD-R gets a label with the first filename to the last filename, and the date. Then I go erase the RAW files from the hard disk.

Once the converted TIF files are viewed, I can throw out the bad ones. The good ones can be refined, if necessary, for rotation, brightness, with cropping, or whatever. Then stored on the same hard disk and also a different hard disk. Once in a while, when I get a really stunner, I will copy it to a different directory with a marketable name as opposed to a Canon-type filename.

Periodically, I also make short-term backups of entire disks onto DVD, and it takes a pile of them.

It is possible for me to have an image in only one place, and that is typically only the CF card. Once into my computer, it finds a home in two or more places, and sometimes four or five, depending on whether it is a temporary thing or a permanent save-for-posterity Ansel-grade image.

The only digital image files that I give away for free are low-resolution web-type files. I keep the good ones for printing.

In one year, I have had to go back and resurrect a RAW file from the CD-R archives only once. I knew approximately what date the shot had been made, and I just sorted through a tall stack of CD-R disks until I found the correct date.

---Bob Gross---

[Longwatcher]

For me,
The first thing I do after I move the images to computer is create a CDR or DVD+R (depending on number of images in set) using the original RAW files. I label them "RAW archive" and with the subject matter, date taken, file numbers, and copyright logo. Then I store them by date.

After that I convert/edit as necessary. When I have completed that step I write the resulting edited images again to CDR or DVD+R depending on total file size. Usually this is a CDR since I only edit a small percentage of the whole. The edited images usually get stored as full size ready to print .tif and reduced size .jpeg for viewing. I label the discs as "Edited Images" with subject matter, date and copyright logo. I then store with Raw archive disks. If a paid job then I make a second backup copy of all discs and store seperately.

I then make any copies and prints for customers and/or models. Finally if for my on-line portfolio, I upload to web server.

Although I have not had a problem yet, it is my intention when I need to to migrate the discs to another format. While I don't use the 25 cent discs, I don't buy the really expensive ones either. Currently I trust "Imatation" brand CDR and "Memorex" brand DVD+R as being reliable, but inexpensive.

My logic to the whole process is, the RAW format will be useable at least as long as I have a Windows XP OS, since I have the software on disc. It is smaller the .tif on a per image basis and contains the original information as recorded. If I will lose it in the future, I can load and convert each disc to which ever format will be the most efficient at the time of conversion and then store it on the current storage format. The edited images are the only really important ones though and both TIF and JPEG are standard enough they should be good for another 10-15 years at least, when the technology will be radically different.

Just my opinion and what I do,

[James01]

All I do is make a folder called Photos and sub folders of the event name with the processed JPG's or Tiffs in them.

Under the Photos folder I make another subfolder called Digital Negatives and create subfolders of the RAW files in there with the same event name as the corresponding JPG or Tiff folder.

I immediately burn them to a Multi-session DVDRW until I have a full DVD. When the DVD is full I make 2 copies on Plain DVDR's and keep one offsite (like a safe deposit box or work).

I also use a program called Where-is-it to catalog the CD’s. This takes a few minutes to read the disc’s and create a catalog of thumbs and file/folder structure. Comes in very handy when looking for a picture.

Just as a note I also keep them on my hard drive after they are burned for quick access… Storage is cheap but unreliable (hence why I burn them to DVD for archiving)


I'm not a pro and some would say putting CD’s in a safe deposit box is a bit much, but would be absolutely devastated if I lost pictures of my family that could not be replaced (like kids first steps kind of thing).

[billhercus]

I am just working out what to do as well. It does make sense to me to go down the USB 2.0 route to a hard drive.

I happen to have a spare 40GB drive and a portable housing for it can be had, even in rip-off UK for 30 pounds.

This is fast and cheap storage indeed!

[James01]

billhercus wrote:
I am just working out what to do as well. It does make sense to me to go down the USB 2.0 route to a hard drive.

I happen to have a spare 40GB drive and a portable housing for it can be had, even in rip-off UK for 30 pounds.

This is fast and cheap storage indeed!


I just fear using hard drives without some sort of redundancy (raid 1 / 5 /10 /50) . To easy for a drive to crash and say bye bye to photos.

[psk4363]

Thanks JC, for posting your questions. Having recently upgraded to the 10D I have been looking at alternative methods for image storage.

Santa arrived a little early, this week in fact, with a LaCie DVDRW drive so it's been very useful to read about how the others in his forum store their images, particularly those with DVDRW drives.

Cheers,
Barry

[billhercus]

"I just fear using hard drives without some sort of redundancy (raid 1 / 5 /10 /50) . To easy for a drive to crash and say bye bye to photos."


For pure storage data the likelyhood of a crash is very, very remote indeed. Furthermore, free software is available to recover files from practically any situation.

CD storage has its risks too .....

[James01]

billhercus wrote:
"I just fear using hard drives without some sort of redundancy (raid 1 / 5 /10 /50) . To easy for a drive to crash and say bye bye to photos."


For pure storage data the likelyhood of a crash is very, very remote indeed. Furthermore, free software is available to recover files from practically any situation.

CD storage has its risks too .....




You can discuss this with out Disaster recovery team... Ive replaced 100's of single bad hard drive. Doesn't seem remote to me. Once the heads slap the paltters or the motor gives out the only way is to send it off and have the palttres removed for data recovery. Big $$$


Oh yea on the cd storage...

Yes its risky as well but easier to make 5 copies of a DVD or CD than a hard drive.

[Jesper]

I always shoot RAW with my 10D and I keep my CRW and THM files in a directory structure sorted by date. I copy them from the CF card with a card reader, just using Windows Explorer. I have a subdirectory for each month, with subdirectories for each day:

...\EOS10D\200311\2003_11_20
...\EOS10D\200311\2003_11_21
...etc...
...\EOS10D\200312\2003_12_01
...\EOS10D\200312\2003_12_02

Occasionally I backup this directory structure to DVD. I don't remove the photos from my harddisk after backing them up - the harddisk is so large that I won't have a problem of running out of space very soon.

I use CaptureOne DSLR LE to navigate through the RAW files. For editing, I have a work directory that also has subdirectories for the date:

...\work\2003_12_03
...\work\2003_12_04

When I want to edit images, I convert the images using CaptureOne and write them to the work directory of the editing day. From there I use Photoshop Elements and other editing software. I don't backup TIFF or JPEG files, only the original CRW and THM files.

I just have a single IDE harddisk - no RAID or whatever.

[robertwgross]

billhercus wrote:
For pure storage data the likelyhood of a crash is very, very remote indeed. Furthermore, free software is available to recover files from practically any situation.


One year ago I had purchased a new HP computer with 80GB disk. Approximately 70 days later, the disk crapped out, and it was not the fault of the disk controller or anything else. I called HP, and they gave me some options:
(1) Return the entire computer to the retailer or to a repair depot, let them have their way with it, and then hope that you get it all returned in one piece 30 days later.
(2) Replace the failed disk on my own, at my own expense.

Obviously, (1) was unsatisfactory because of the wait. (2) cost me money, but it all happened within 12 hours.

I've still got a nice 80GB disk here, if anybody would like it. It was only used for 70 days! It looks like new.

---Bob Gross---

[maderito]

After my first 1000 images on my 10D, I decided to get a serious image database program that would allow me to file and find images in a logical, coherent fashion. I opted for IMATCH ( http://www.photools.com/ ), a highly flexible image cataloguing software that works seamlessly with images on a hard drive and removable media. It is at times maddeningly quirky and not always intuitive, but once you’ve mastered the tools you need, it’s definitely feels worth the effort.

Among its MANY capabilities, IMATCH allows you to assign categories to your images – as many as you like. Thus a photo of my daughter might be assigned to her own category, as well as “family”, “favorites”, “vacation-2003”, etc. I can then create selections (i.e. sets) of photos based on logical combinations of categories. Once the selections are created, they can be submitted for further processing (e.g. conversion from TIFF to JPEGS) using scripting facilities of the program.

IMATCH is flexible and powerful. It requires a certain degree of dedication to organization and filing your images. The beauty, for me, is that the physical location of your images doesn’t matter. In fact, I simply keep all my original images stored on the hard disk in file folders of 100 or 250 images (depending on whether they are RAW or JPEG). Processed files (usually TIFFs) I save in folders labeled with Year_Month.

I have a general work folder for use before processing with IMATCH. I download images from the CF card to the work folder. After discarding unwanted images and processing the rest in the work folder, I open IMATCH and assign categories to all images in the work folder. Then, while still in IMATCH, I move the images to their final destination on the hard disk, creating new folders as needed. IMATCH keeps track of the physical location of the files. The work folder is now empty and is ready for the next batch of images. I can now visualize my updated image collection in a virtual file system based on categories in a manner that mimics a Windows file directory.

IMATCH has way too many features that I’ll never use. But its core features are so essential and I can’t imagine not using it. My only fear is that the product will no longer be supported one day.

IMATCH costs $49.95 US. This is not an advertisement. So if you bite on this product, remember that I have said up front that this program has a steep learning curve. Many complain that it is way too complex for day-to-day needs. Someone once said that truly complex problems often yield to simple solutions that are always wrong. IMATCH is fundamentally a complex software designed for the not so simple and maybe complex problem of image management and storage.

[excessnoise]

My 2 cents worth. Yes Raid 5 (various names -redundant array of inexpensive disks) would be the ticket but you have to have minimum 3 disks, 2 for data, 1 for parity. You then need a server (raid controller) so for home use might not be practical.

The way I organize is Year folder, under that month folder and then project/event name.
-->2003
|-->11_03
|-->project name i.e. (Hawaii vacation) ,etc.
That way If I know what time frame, I can read in English the actual event folders.

[hmhm]

I have two PCs at home with big-ass hard drives. One PC "syncs" itself to the other PC every night, so I have two copies of every file. If a hard drive dies (just a matter of time), I have everything on the backup hard drive.

This would be "better" if:
- the backup PC was in a remote location. There are plenty of scenarios that take out both PCs (theft, fire, flood, etc.)
- a new file had a checksum stored with it, and these checksums were periodically recalculated online to detect corruption

I'm yet to find a good way to organize images, I tend to just keep them sorted into directories by date.
-harry

[arthurb]

james01 wrote:
I just fear using hard drives without some sort of redundancy (raid 1 / 5 /10 /50) . To easy for a drive to crash and say bye bye to photos.



As a computer proessional I also insist on using Raid technology for storing important data. When set up properly, you can have drives fail and have no loss of data.

The problem is that you can't go to BestBuy or Circuit City and tell the sales droid you want a machine with Raid. The technology is not generally used on desk top machines, it is more common on servers.

An easier and cheaper alternative for the average person is a second hard drive. 99% of machines sold today have the physical space and cables already in place to add a second hard drive. With the holiday season sales starting, hard drives are fairly inexpensive.

Once the second hard drive is in place, you then make sure to copy your photos to directories on both drives.
The nice thing is that copying doesn't degrade the images, like opening and saving.

Unless you have very bad luck this should protect you from a "normal" hard drive failure. This will not protect you from failure due to the computer falling off a table or a major lightning strike

There should still be a cd or dvd backup. The only problem with dvd is that some players won't read all formats so if you have a major disaster, you may have trouble getting the right reader to access your files.

I like the advice of the fellow who talked about 5 cd-dvd copies, and the safe deposit box is great for unreplaceable photos. (If your house is destroyed, you may still be able to get your photos.) Another idea is if they are family photos, send copies of the backups to family members.

cds and dvds are cheap compared to the priceless value of your photos.

Good Luck with whatever you choose.

[Canuck]

jcalvert wrote:
I am interested in hearing what people are using for storing and managing their images.

I am using Photoshop CS ( what's there to say, its great) and also ACDSee which is really terrific at sorting them.

Do people generally keep the images on hard drives, or move them off onto CDs / DVDs. If so, how do you know where they are ( i.e. which CDs)?

What size drives are people using and does anyone use RAID 1 or 5?

Do you keep RAW, TIFF and THMs in the same directories, probably not.

Cheers
jc

Hi!
Here's what I have:
Athlon XP2.2GHz processor on ASUS RAID mobo, 512MB PC2700 DDR RAM. I have 2x WD 120GB 8MB buffer, 7200 RPM HDs set up in RAID, stripe array so that I have essentially 1 240GB HD...sortof. I have 2 partions, a 10GB (system stuff) and 215GB (rest of the lot) ones. Obviously you will not see the whole 240GB due to FAT and formatting. It's a screaming system till you start making the RAW pics into TIFFs in Canon software. This RAID stripe array setup was a piece of cake to setup! That was due to the software it has.

I have Elements 2.0 that came with the camera and some other software that came out of thin air.

I save my files to HD and back them up on CDR for now. I'm waiting for the price to come down on the all in one DVD burners. That would be the DVD +/- R/RW and DVD RAM/ROM. I really like the idea of 9.4 GB to a DVD. It would make my life a lot easier as I can fit about 20 TIFFS from RAW format on CD. I'm making almost 37MB files from RAW pics I take. I'm not bothered about the space on the HD, it has only recently dented it. I have like 195GB left on the 215GB partition. I think I'm gonna repartition it as that's too much free and open space. That's a rainyday project.

Oh yes, sorting pics...I have a folder for the RAW pics only, then the manipulated pics go in separate folder under improved pics. That is more to destiguish originals from tweeked ones then anything else as well as the ones I can e-mail to family vs RAW pics that are no good w/o software! Hope this helps!

[ilya]

I'm into hobby kind of stuff, more driven by date rather then theme. So all my stuff is organized by date. Oh, and I have a progression of device dependent directories, starting with my scanned film stuff (\Scanner), going on to my Sony Vid Cam with 1.5 megapixel pix, then the Sony 4 megapixel, and now the EOS 10D.

I have 250gigs between two drives, and a DVD burner.

I save everything for now until I run out of space onto the big drive in the following directory structure: EOS10D\11-2003\11-15-2003\Raw

The processed Tiffs and Jpegs go into the "day" subdirectory, and original raw obviously goes into "raw".

Each "month" directory will also have a sort of best of that month subdirectories entitled 4WEB and 4PRINT. That's where the occasional good shot goes formatted either for web and for printing. Its easy to find then the best of the best in each month, and upload to the gallery or print/reprint.

Each subdirectory for the day may or may not have an extra blurb in the directory name to highlight a particular activity or place (golf or ski trip or nick's portraits or whatever).

I will burn a DVD once every couple of weeks or so, and store it. I know, I know I should do that every time I download, but I don't.

Your ACDSee (assuming its ver 6) has a catalogueing system which probably works pretty well, and it looks like it could burn a dvd/cd with all the bells and whistles, although I've not tried either feature; and I'm not sure how good ACDC is at reading raw. Photoshop CS has a tagging category feature which could be worth something, though I've not played around with that either.

My system is pretty simple, but it works for me. When I run out of space, I figure I'll permanently archive stuff on DVD. That's one investment everyone has to make, a DVD burner.

[sp00g3]

If you had mission Critical data (Like a database that houses 100,000 records+ I would go higher than RAID 1. But for your usage of it.. Mirroring is prolly the best and cheapest bet. It's slower, but its just data storage. I wouldn't worry bout shelling out big bucks on RAID 5 controller plus the extra drives. (If you did go this way whats one more drive to make a hot spare?) Now, if this was for video than I would recommend the RAID 5. But you would want to have more disks anyways because of the space consumption.

Hope this helps

[James01]

Anything over raid 1 on a home system is just about out of the question. Besides raid (any level) is good only for drive failure, if you corrupt or delete any file by accident then its gone either way. Sorry to be the barer of bad news but the stripe (raid 0) is the worst. If you lose either drive due to failure then you lose the data on both drives. If you are going to waste money on raid at home (via SATA) then go with raid 1 at least. You get half the space but your files are backed up on 2 drives. Even with raid (from 1 to 50) on all our servers the data MUST be backed up because disks are to volatile, at home there is no question that if you have valuable photos you should back them up to DVD/CD.

Side note:
What Harry does rocks… nothing like syncing to another computer nightly… just don’t sync the bad data over the good data :)

[Ikinaa]

...

The problem is that you can't go to BestBuy or Circuit City and tell the sales droid you want a machine with Raid. The technology is not generally used on desk top machines, it is more common on servers.

...

Check this http://www.adaptec.com/worldwide/product/proddetail.html?language=English+US&prodkey=AAR-1210SA&cat=%2fTechnology%2fRAID%2fSerial+ATA+RAID
for $75 you get a raid0/1 controller...

[James01]

Ikinaa wrote:
...

The problem is that you can't go to BestBuy or Circuit City and tell the sales droid you want a machine with Raid. The technology is not generally used on desk top machines, it is more common on servers.

...

Check this http://www.adaptec.com/worldwide/product/proddetail.html?language=English+US&prodkey=AAR-1210SA&cat=%2fTechnology%2fRAID%2fSerial+ATA+RAID
for $75 you get a raid0/1 controller...




Exactly... well except for the SATA stuff now... A lot of the newer PC's come with the ability.

[GenEOS]

This is definitely a thread that could take lots of time to explain in detail, and everyone will have a different technique depending on their equipment and knowledge.

For the first time this year, I have actually followed through with a workflow for archiving.

I shoot all RAW files, so I have many, many CDs.

I don't have a DVD yet, but that would be nice.

I am limited on HD space, so I keep 2 copies of each archive CD and move the back-up set to a diffent location.

Here is a short run down of my system:

1.) Shoot an event in RAW
2.) Download to computer in event specific directory, per my naming rules that I chose to follow.
3.) Preview all RAW files and delete all bad files, careful, because once gone, there gone to photo heaven.
4.) Select my images I will be submitting for the assignment.
5.) I write the selected files to a sub-directory, the way I submitted them
6.) I thne copy this directory to my other computer's hard drive.
7.) After collecting about a CD's worth of directories, I burn them to 2 CDs. A master and Back-up.
8.) I then check each CD for errors.
9.) I enter the CD information into a spread sheet I use to track all my shoots.
10.)I then delete the files from both hard drives. (this is the part that always makes me feel funny when I do it.)

It may not be the "right" way to do it, but I have followed it this year and it has worked out well

The spreadsheet I use really helps in searching for images....

[James01]

GenEOS wrote:

10.)I then delete the files from both hard drives. (this is the part that always makes me feel funny when I do it.)



HAHA this is funny... I am terrified of this part to. I just
got the x-drive and still afraid to erase my memory card.

[JCalvert]

Thanks to everyone for your replies. This has been terrific.

1) It appears I had better buy a DVD burner, and forget about RAID.

2) I am convinced RAW shooting is the only way to go.

3) Thanks to maderito for the IMATCH suggestion. That needs a good look.

Great info!

[arthurb]

Ikinaa wrote:
for $75 you get a raid0/1 controller...


If a computer professional ever tells you to use RAID 0 and says that it is for the protection of files, Run, don't Walk away.

Raid 0 is used to combine smaller drives into seeming like a larger drive. For example you need 10 gigs of space but only have 2 8 gig drives, Raid 0 can make those look like a 16 gig drive.

It is a dangerous and foolish practice, since a failure of one drive can corrupt all of your data.

[Phil Hall]

This method works for me

1] shoot RAW
2] convert from RAW to JPG using Breezebrowser and save on HD
3] Review JPG's in ACDSEE, delete as required
3] Save RAW and JPG files to DVD or CDR
4] convert files for printing or editing in TIF using BB
5] edit in Photoshop and print.
6] backup working files on 2nd HD
7] Archive disks at another site.

[CyberDyneSystems]

jcalvert wrote:
Thanks to everyone for your replies. This has been terrific.

1) It appears I had better buy a DVD burner, and forget about RAID.

2) I am convinced RAW shooting is the only way to go.

3) Thanks to maderito for the IMATCH suggestion. That needs a good look.

Great info!

Eventual archival storage onto CD-DVD media is a great solution.. the problem is maintining your regement of back ups,. and then maintaining some sense of order as you archive.

I am much to weak in my own personal organizational skills to be confident that I will allways be up to date with CD-R/DVD back ups.. (besides,. my DVD-R drive is on the fritz! )

So I took the issue out of my hands by using two methods of redundant hard drives.

First I traded in my old very fast RAID 0 array for a newer nearly as fast RAID1 array (two 160gig drives,. same data)

This solves the "what if my hard drive "dies" (ei inrevoverable drive failure)

The second system is a third hard drive,. not on the RAID controller,. but just a seperate 80 gig drive.

I use a program called MirrorFolder 2.0 which is a sort of software RAID backup program. It can be used a a straight software RAID,. but it is much more flexible. You tell it what specific folders to maintain a copy of,.. so you don't need to waste space on junk. I "mirror" "My Documents" and the folders that I keep all my images in.

This gives me three copiies at all times on three hard drives with no effort on my part.

As far as accidental deletion?

Again,. MirrorFolder has options that solve this too...

You CAN set it to "real time" mirroring. But I allready have that with hardware RAID1,. so instead I set the Mirror folder to "mirror" the folders I selected every 2 days.

It does it in the back ground,.. and it won't do it while you are working on the files.

Lastly,. you can tell Mirror folder to make Zip archives of you mirrored folders at specifice intervals.. and archive them. I don't use this option,. but it could be a 4th layer of safety net to protect your self from accidental deletion.


either way,. I would recomend both optical storage as a hard back up... but I would HIGHLY recomend you do this in addition to a Hardware or software RAID1 set up with at least two hard drives.

[Ikinaa]

arthurb wrote:
Ikinaa wrote:
for $75 you get a raid0/1 controller...


If a computer professional ever tells you to use RAID 0 and says that it is for the protection of files, Run, don't Walk away.

Raid 0 is used to combine smaller drives into seeming like a larger drive. For example you need 10 gigs of space but only have 2 8 gig drives, Raid 0 can make those look like a 16 gig drive.

It is a dangerous and foolish practice, since a failure of one drive can corrupt all of your data.



The controller is for raid 0 and 1, but i agree, using raid 0 isn't a good solution for protecting files :D , but as the controller can do raid 1, it's a good solution anyway...

[kellylipp]

This is not an advertisement.

I am a principal in a company that specializes in Backup/Archive and Disaster Recovery. The only way to truly protect your data is to have it in at least two places and one of those places should be geographically separate from the other. You are trying to protect against two types of disaster: the little disaster and the big disaster.

The little disaster is the loss of a drive or disk (your hard drive or the CD you put your stuff on).

The big disaster is the whole damn place burns down, or shakes down, or is washed down, or whatever the most likely natural disaster is.

So, minimally, you should copy your stuff to another disk, CD, DVD or tape and then take that copy some distance away from your primary storage.

I'm lucky: I have access to our corporate backup server and a scheduled backup gets my stuff whenever I am in the office or can see the office via the VPN. I also put my images on CD so I will have them locally. I intend to build a new image processing workstation with a large hard drive (and I'll back that up to the company server via the VPN) and also keep local backup copies on DVD.

RAID will help you minimize the little disaster but does not help you with the big disaster.

Bottom Line: make a copy and take it to your Mom's house.

[robertwgross]

kellylipp wrote:
...
Bottom Line: make a copy and take it to your Mom's house.

Or send it over on the net. Your Mom has a T1 drop to her sewing room, doesn't she?

---Bob Gross---

[Ikinaa]

Here's how I do it...
I got a G3 and an Ixus 300, make about 6000 photos a year, makes me about 4 Gigs of data per year (== 3000 photos left)
I transfer the photos from the CF to my PC (manually on two different harddiscs on the same PC). Burn 2 CD-RW of the actual stock until it reaches 700 Megs, reformat the CF only after that.
When I have 700 Megs, I burn 3 CD-Rs of 3 different brands, and store them on 3 geographically different places (at home, at my mom's, at my office where i copy them to my office-harddisc so to access them rapidly)

So in the end, i have them in 6 different physical places at 3 different geographical locations.

Now... isn't that paranoid? :D
But unless a 10km rock from space hits western Europe, I always got my pictures...