DNG and RAW - working files and archives

[JoesLdy]

Does this sound completely retarded? I have been all over the net today trying to come to a conclusion on what to do with my damn files, and I haven't been able to make a decision one way or the other with DNG format.

I've read the DAM chapter 1, will most likely be purchasing the book soon. From that standpoint DNG files are the way to go in order to be able to transfer your IPTC info. My biggest concern is not losing all of the editing I have done in Lightroom to my RAW files. I've started at ZERO with my files, will be weeding through, properly keywording and cataloging my files etc...

My thought is to import in to Lightroom RAW, do my initial run through weed out duds etc, edit to my liking, then burn the RAW file and XMP sidecar to DVD (I will also keep them on a 2nd hard drive- two different sites).

Now, here is where I am hung up. I don't want to use Lightroom to catalog my files. But right now I don't have the funds to plunk down on iview. I have ACDSee Pro but apparently they don't support IPTC info in DNG (I believe they can read it from sidecar files but that defeats the purpose!).

So I thought, I could back up RAW + XMP, then export as DNG out of Lightroom and keep those files on my local drive since Lightroom embeds all of the XMP data in to the DNG file when you export. Then I'll have everything ready for when I do purchase iview (Ill catalog and keyword in lightroom for now).

Or should I just stick with the RAW files, keep the XMP sidecars, and wait til I get the DAM solution in line and import from there (assuming there is a way to do this).

I hope I make sense, my brain is jelly at this point

[JohnJ80]

Does this sound completely retarded? I have been all over the net today trying to come to a conclusion on what to do with my damn files, and I haven't been able to make a decision one way or the other with DNG format.

I've read the DAM chapter 1, will most likely be purchasing the book soon. From that standpoint DNG files are the way to go in order to be able to transfer your IPTC info. My biggest concern is not losing all of the editing I have done in Lightroom to my RAW files. I've started at ZERO with my files, will be weeding through, properly keywording and cataloging my files etc...

No big deal. Just select the photos you want to convert to DNG and then select the "Convert to DNG.." menu item and you are done. Simple.

DNG is the way to go. Embedding the metadata into the file is infinitely better than a raw file with a sidecar xml file.



My thought is to import in to Lightroom RAW, do my initial run through weed out duds etc, edit to my liking, then burn the RAW file and XMP sidecar to DVD (I will also keep them on a 2nd hard drive- two different sites).

What you want is ImageIngestor. It does ALL of this for you so you don't have to. See:

http://www.imageingester.com/ImageIngester/index.php

You add keywords and most of the metadata when you import. This will then create the DNGs for you and put them where you want them to be and then call the program you use and set it up to import them. It also adds the metadata to xml sidecar files and puts them with the RAW in the backup directory you designate. You use II to do this as you import directly from the card.



Now, here is where I am hung up. I don't want to use Lightroom to catalog my files. But right now I don't have the funds to plunk down on iview. I have ACDSee Pro but apparently they don't support IPTC info in DNG (I believe they can read it from sidecar files but that defeats the purpose!).

Well, I have iView and LR is better. iView is ok, but it is now owned by Microsoft and from what I can tell, it is already hard to tell what they are up to with it. I'd definitely recommend LR over iView now.

LR does a great job in reading DNG.



So I thought, I could back up RAW + XMP, then export as DNG out of Lightroom and keep those files on my local drive since Lightroom embeds all of the XMP data in to the DNG file when you export. Then I'll have everything ready for when I do purchase iview (Ill catalog and keyword in lightroom for now).

Or should I just stick with the RAW files, keep the XMP sidecars, and wait til I get the DAM solution in line and import from there (assuming there is a way to do this).

I hope I make sense, my brain is jelly at this point

You are making this too difficult and you're thinking too much about this. Go get imageingestor and follow the backup strategy. II was created right straight from the DAM book. Then use LR with your (now) only DNG files. You then have the option of importing those DNGs into any program that reads DNG in the future because the metadata is embedded right into the image file and stands alone.

J.

[René Damkot]

I prefer iView over LR. Runs better on my (old) Mac... Then again, there are about 45000 files in the catalog...

I use Image ingester, open in iView and add the more image-specific keywords.
Then I ditch the XMP files ImageIngester created, and let iView write the metadata into the CR2. That way it's there also if I use DPP to convert the raw files.

I make a selection in iView, and import that selection into LR. After editing, I convert all files to DNG. Both CR2 and DNG get archived.

If I use DPP, I let DPP write the edits back into the file, so they are preserved as well (as long as I don't change IPTC or so using iView afterward; that wipes out the edits :mad:).

[tim]

I'm undecided, so I keep a copy in each format. On my working drive they stay RAW, the offline storage in my office is RAW, but my offsite hard drives are DNG.

DNG is the way to go. Embedding the metadata into the file is infinitely better than a raw file with a sidecar xml file.

I don't get why people have a problem with XMP files. They don't bother me in the slightest. Occasionally i've edited the XMP file by hand, which is pretty useful.

[René Damkot]

I don't get why people have a problem with XMP files.

- Not all software I use supports them (iView, DPP)
- I move files about in the finder
- I rename files sometimes

[JohnJ80]

- Not all software I use supports them (iView, DPP)
- I move files about in the finder
- I rename files sometimes

- directory trees get corrupted
- library links can be lost
- it is just a pain to keep them together

[JoesLdy]

OK I've never used the imageingester program, Ill have to go check that out. Can't I just do the same thing in LR though or is it just quicker with II?

I'm honestly scared to convert everything I have to DNG right now, because after all of the research Ive done I still find a lot of negative feedback on the format. And it wouldn't shock me if they just ditch it in a couple years for something else. This is why I thought of just doing all the keywording etc... on the RAW file with XMP, back that up, put that in my off site storage facility, then use exported DNG files on my local drives to do my copies from.

The problem I personally have with the XMP sidecars is that for one thing, I don't see them consistently coming out of Lightroom. I've never turned them off, yet I don't have them for all of my edits (which REALLY pissed me off because now I have thousands of RAW images I have to re-edit that I thought I wouldn't have to).

Also, I renamed all of my files from the capture names, when i did so I did it in batches using ACDSee (should have done it in LR in hindsight). I renamed the sidecar files as well, now Lightroom doesn't recognize them- even when I attempted to import them. Maybe I am missing something in that aspect... just picked up a book for Lightroom so I can use it more efficiently.

Keep in mind, I don't shoot CR2 files, I am shooting CRWs out of a 10d and 300d (yea I know, I am upgrading to a 5D and 40D soon). I don't think you have as much capability with metadata etc... i.e. writing anything to the original files (damn that would be nice!).

I know I am overthinking this- thanks for the feedback! :)

[tim]

- Not all software I use supports them (iView, DPP)
- I move files about in the finder
- I rename files sometimes

That's reasonable, software support. If you only use Bridge you're fine, if you use other tools you could have a problem.

I rename files constantly... from within bridge. It renames the XMPs too.

- directory trees get corrupted
- library links can be lost
- it is just a pain to keep them together

That's kinda paranoid. I've never lost a directory tree, I don't have a library in bridge but they can be rebuilt, and since I don't move files around after the initial download keeping them together is a non-issue.

Do whatever you like, of course, but XMP files aren't a big deal.

[JohnJ80]

OK I've never used the imageingester program, Ill have to go check that out. Can't I just do the same thing in LR though or is it just quicker with II?

II does it all for you - adds the keywords, copyrights etc..., backups the the raw files and puts an xml sidecar with them, converts them to DNG and imports to whatever database you want. This way you never miss a step.



I'm honestly scared to convert everything I have to DNG right now, because after all of the research Ive done I still find a lot of negative feedback on the format. And it wouldn't shock me if they just ditch it in a couple years for something else.


Wow. A number of camera's use it as their native format. You think they'll ditch it - the inventors of the PDF file?

I've not seen much that was negative and I sure haven't experienced it. I have many thousands of images in DNG and also in 3 raw formats (now there is a pain). The RAW data is included in the DNG, the DNG is mostly about a way to store what would be in the xml file in the actual image file (were, IMO, it belongs).


This is why I thought of just doing all the keywording etc... on the RAW file with XMP, back that up, put that in my off site storage facility, then use exported DNG files on my local drives to do my copies from.

Now, XML files is where I've had a problem. I've had a couple of instances of corrupted libraries where the xml files got mixed up from the actual files (different directories or overwritten). Talk about a mess! Fortunately it was backed up.

I have had much less problems since I went to DNG.

I also want my files to have all the copyright info, author contact info etc... in them. XML files don't do that unless you export the raw file and force it to contain that.



The problem I personally have with the XMP sidecars is that for one thing, I don't see them consistently coming out of Lightroom. I've never turned them off, yet I don't have them for all of my edits (which REALLY pissed me off because now I have thousands of RAW images I have to re-edit that I thought I wouldn't have to).

Also, I renamed all of my files from the capture names, when i did so I did it in batches using ACDSee (should have done it in LR in hindsight). I renamed the sidecar files as well, now Lightroom doesn't recognize them- even when I attempted to import them. Maybe I am missing something in that aspect... just picked up a book for Lightroom so I can use it more efficiently.


xml sidecar files simply suck. Not much nice to say about them, IMO. It is a cobbled up means around the fact that the file format is proprietary.



Keep in mind, I don't shoot CR2 files, I am shooting CRWs out of a 10d and 300d (yea I know, I am upgrading to a 5D and 40D soon). I don't think you have as much capability with metadata etc... i.e. writing anything to the original files (damn that would be nice!).

I know I am overthinking this- thanks for the feedback! :)

I don't think you have any less capability for writing metadata to the files. There is also the problem that not every programs treats an xml file the same way or interprets it the same way - that is a huge problem.

I convert everything to DNG and have been very happy with that.

Finally, II does not require you to make a DNG file, it just has a place that asks you if you want to run a processing program on your files - you can do that or you can do nothing.

The DAM Book is very well thought out. I'm tired of trying to dope out my own system and then finding a problem later. I'm using the one in the book since it is far more straightforward and it works.

J.

[JohnJ80]

That's kinda paranoid. I've never lost a directory tree, I don't have a library in bridge but they can be rebuilt, and since I don't move files around after the initial download keeping them together is a non-issue.

Do whatever you like, of course, but XMP files aren't a big deal.

No, its not. It has happened to me twice in the last couple of years. Hard drives fail and laptops get banged around. The newer drives coming out are better with accelerometers in them but they are not perfect at all. I've been doing this a long time and I have a whole bone yard full of failed drives.

My main working library is around 16K images or so (give or take). I have much more archived. If this gets whacked it is a large enough problem that it is pretty much a full time career for a long time to get it straightened out. Time is money and besides that it just plain sucks to have to do it.

Yes, i do backup daily. If I am paranoid anywhere, it is with infrequently used restore software. I've also had problems there as well.

If you ever get a file unlinked from an sidecar file you have no idea what the image file is for or how to figure it out.

While the potential for a problem is low, the potential for catastrophe is high. I'm not going to take any risks and I'm going to do whatever I have to do to protect the years of work that I have in my library.


J.

[tim]

Maybe that's a good reason not to use libraries, but it's not a reason not to use XMP files. With backups the directly unlinking issue isn't an issue at all. Since the XMP file has the same name as the RAW file it's pretty easy to work out which goes with which.

My degree is in computer system engineering and software development and i've been using computers for going on 20 years, before hard drives were common, and i've never had a single hard drive failure (touch wood).

DNG files might be a better option, with the metadata and tweaks inside the file, but since the format's still reasonably new i'm not trusting it 100% yet. Having RAW and DNG is insurance for me.

[JohnJ80]

Maybe that's a good reason not to use libraries, but it's not a reason not to use XMP files. With backups the directly unlinking issue isn't an issue at all. Since the XMP file has the same name as the RAW file it's pretty easy to work out which goes with which.

My degree is in computer system engineering and software development and i've been using computers for going on 20 years, before hard drives were common, and i've never had a single hard drive failure (touch wood).

DNG files might be a better option, with the metadata and tweaks inside the file, but since the format's still reasonably new i'm not trusting it 100% yet. Having RAW and DNG is insurance for me.

Disagree - my degree is also in electrical engineering and computer architecture with 30 years experience (got you there). I've developed OS software, I've designed computer systems, I've managed computer system design projects, and I am quite experienced in reliability and failure analysis. I can remember the day when I thought I had died and gone to heaven when my PDP-11 got a 5MB RLO5 drive that was the size of my washing machine. But, a digression.

XML files are no good unless they are in the same directory. It doesn't take too much to scramble a file systems structure and have a linear list of files after you try manage to get the data off the failed drive. Trying to match up 16,000 image files with 16,000 xml files is not fun (been there, done that). As well, it is not unusual (in the world of file system failures) to have names get pointed at the wrong files. Then, the recreation is impossible for all intents and purposes (had that happen too).

Failed drives are far more common that what you describe - in my business we just had 2 server drives fail (funny, when I figure it out they were right at the spec'ed 50,000 MTBF after running 24/7 for a whole long time - should have been paying attention). Also, if you use laptops, it is not unusual at all to have a drive failure relatively often. I just replaced a drive in one of my guy's laptops that had failed after 1 year. I personally have had at least 3 disk crashes in the the last 5 years. I find it odd, in fact, that you have never had one.

One of them, when I had my backups screwed up, required that I send the drive out to one of those companies that take the drive apart, take the platters out and get your data off. Guess what - the data came off as a linear list of files in one big directory and it was a mess plus some files were corrupted. On a tangent, if drives didn't fail, then it is surprising that such a number of companies exist to recover data from failed drives.

In my professional life, I work with more than one drive manufacturer on an OEM component level. I assure you that drives fail and are not nearly as reliable as we would like them to be.

The problem comes in that the reliability of the drives is quite good in the 1-3 year range (depending on mfg), and then starts to drop after that with some significant decrease per unit of time. So, you have a small probability of a drive failure in, say, one year, but after 5 years your probability of drive failure becomes quite unacceptable (although that is purportedly being addressed by mfgs with 'enterprise' grade drives). Since, I presume, most photographers want to keep their libraries intact longer than that, then one has to protect against device failure AND make sure that recreation strategies on several levels exist.

On a photographic level, all of this is MUCH easier if the file contains all its information and stands alone. Is xml workable? Yes, of course but it is not as good.

J.

[JoesLdy]

See, I would love to keep the DNG with the embedded orginal RAW file, but that makes it twice as big. Since I was going to keep the original RAWs anyway obviously it's not a space issue- I just want to keep my working files as small as possible (my computer is a tad on the old side, and I plan to upgrade my glass and bodies before it, so I have to coddle it a bit heh).

Help me clarify as well, Lightroom generates what kind of sidecar file? XMP or XML? I think I've been trying to take in too much info at once... oy. I know that XMP is a format for IPTC info (correct me if I am wrong) I thought the sidecars from lightroom were XMP info written in to a small text file (no Ive never looked at the extension). Are they XML files?

I agree whole heartedly that this info needs to be kept in the "digital negative" no matter what acronym we give it (RAW or DNG). That is why I am contemplating whether to go the DNG route at this point. I also know enough about computers to be hit home with the point of linking up 16k worth of files as given in an example if I have a failure. On that note, I have had about 3 drive failures in my computing time, and that was long before I taxed them the way I do today.

My comment on the ditching of the DNG format, I don't necessarily think that Adobe will ditch the idea. I do think that either something better will come out soon enough that I will have wished I hadn't converted, or every other software out there wont jump on the bandwagon like they predict.

I've seen some negatives such as the DNG does not actually hold all of the original data on the RAW files, that although it is a lossless file type there are still bits that get lost in the process like it or no. I've read stories about it being unstable. I can't think of everything, but it's just like every other option we face- there are always two sides of the fence and the grass is always greener on the other side ;)

I mean hell, I hear people actually say RAW is a waste and to shoot JPEG which I find shocking coming out of a pro photographer... but... people still say it.

So I just want to make an informed decision.... and keep myself from ever having to do this again with my files. I have literally sat here non stop for two and a half days with this, I don't EVER want to do that again. And I haven't even begun to re-edit!


I'm going to go check out II, sounds pretty cool! I'll probably still keep my original RAWs for a while maybe a year or two, just to see how it all pans out. I'm with Tim I don't trust DNG completely just yet. I'm thinking I will still edit in LR after all of the conversions and back up, and I still don't know what I will use to catalog (I really want to use ACDSee I love it- alas it doesn't read IPTC info from DNG).

[JohnJ80]

See, I would love to keep the DNG with the embedded orginal RAW file, but that makes it twice as big. Since I was going to keep the original RAWs anyway obviously it's not a space issue- I just want to keep my working files as small as possible (my computer is a tad on the old side, and I plan to upgrade my glass and bodies before it, so I have to coddle it a bit heh).

Concur. You should consider keeping a backup on an external (usb or firewire) drive anyhow. I keep my RAWs on a separate drive from my library/DNGs and both are backed up. The good news is that external drive will work on any new computer you get so you might want to consider that now anyhow.



Help me clarify as well, Lightroom generates what kind of sidecar file? XMP or XML? I think I've been trying to take in too much info at once... oy. I know that XMP is a format for IPTC info (correct me if I am wrong) I thought the sidecars from lightroom were XMP info written in to a small text file (no Ive never looked at the extension). Are they XML files?

I'm not an expert, but I did read through the DNG spec and stuff on Adobe's website. While I could be wrong (maybe someone else could jump in here), I believe that XMP is a format of the data and it is written out into either an XML file or the original image file. In other words, XMP is not a file but a format.



I agree whole heartedly that this info needs to be kept in the "digital negative" no matter what acronym we give it (RAW or DNG). That is why I am contemplating whether to go the DNG route at this point. I also know enough about computers to be hit home with the point of linking up 16k worth of files as given in an example if I have a failure. On that note, I have had about 3 drive failures in my computing time, and that was long before I taxed them the way I do today.

My comment on the ditching of the DNG format, I don't necessarily think that Adobe will ditch the idea. I do think that either something better will come out soon enough that I will have wished I hadn't converted, or every other software out there wont jump on the bandwagon like they predict.

Either way, there will be converters simply because it is relatively widespread, there are other software makers using it and there are a number of camera makers for which DNG is their native RAW format. Secondly, these are the same folks that brought you the PDF file and that has been handled well and all files can be read by the most current reader. I would expect much the same strategy here.



I've seen some negatives such as the DNG does not actually hold all of the original data on the RAW files, that although it is a lossless file type there are still bits that get lost in the process like it or no. I've read stories about it being unstable. I can't think of everything, but it's just like every other option we face- there are always two sides of the fence and the grass is always greener on the other side ;)

There are two kinds of data - the RAW data from the camera and the linear data. You an convert from RAW to linear but not back although it is a one to one mapping (lossless). The linear mode can lose some mfg specific data and could be an issue in certain cases (the way I understand it). So, just don't convert it to linear.



I mean hell, I hear people actually say RAW is a waste and to shoot JPEG which I find shocking coming out of a pro photographer... but... people still say it.

LOL. I think this is probably more of the same and in that vein. I shoot everything in RAW now (even sports). Disk space is super cheap and the benefits are huge.



So I just want to make an informed decision.... and keep myself from ever having to do this again with my files. I have literally sat here non stop for two and a half days with this, I don't EVER want to do that again. And I haven't even begun to re-edit!

I feel your pain. Been there, done that. I don't think you can ever get away with saying "never", but this looks to me like a more stable methodology going forward. At least, you'd have to do it 'less.'

[/quote]


I'm going to go check out II, sounds pretty cool! I'll probably still keep my original RAWs for a while maybe a year or two, just to see how it all pans out. I'm with Tim I don't trust DNG completely just yet. I'm thinking I will still edit in LR after all of the conversions and back up, and I still don't know what I will use to catalog (I really want to use ACDSee I love it- alas it doesn't read IPTC info from DNG).[/QUOTE]

I keep all my original RAWs - and one of the reasons I use II - it helps to do that painlessly AND includes xml files with the metadata that is added when the files are ingested. That helps to find the original should I ever have to do that (haven't so far).

LR does a decent job of cataloging. So I think most people are good with that.

J.

[tim]

Disagree - my degree is also in electrical engineering and computer architecture with 30 years experience (got you there). I've developed OS software, I've designed computer systems, I've managed computer system design projects, and I am quite experienced in reliability and failure analysis. I can remember the day when I thought I had died and gone to heaven when my PDP-11 got a 5MB RLO5 drive that was the size of my washing machine. But, a digression.

XML files are no good unless they are in the same directory. It doesn't take too much to scramble a file systems structure and have a linear list of files after you try manage to get the data off the failed drive. Trying to match up 16,000 image files with 16,000 xml files is not fun (been there, done that). As well, it is not unusual (in the world of file system failures) to have names get pointed at the wrong files. Then, the recreation is impossible for all intents and purposes (had that happen too).

Failed drives are far more common that what you describe - in my business we just had 2 server drives fail (funny, when I figure it out they were right at the spec'ed 50,000 MTBF after running 24/7 for a whole long time - should have been paying attention). Also, if you use laptops, it is not unusual at all to have a drive failure relatively often. I just replaced a drive in one of my guy's laptops that had failed after 1 year. I personally have had at least 3 disk crashes in the the last 5 years. I find it odd, in fact, that you have never had one.

One of them, when I had my backups screwed up, required that I send the drive out to one of those companies that take the drive apart, take the platters out and get your data off. Guess what - the data came off as a linear list of files in one big directory and it was a mess plus some files were corrupted. On a tangent, if drives didn't fail, then it is surprising that such a number of companies exist to recover data from failed drives.

In my professional life, I work with more than one drive manufacturer on an OEM component level. I assure you that drives fail and are not nearly as reliable as we would like them to be.

The problem comes in that the reliability of the drives is quite good in the 1-3 year range (depending on mfg), and then starts to drop after that with some significant decrease per unit of time. So, you have a small probability of a drive failure in, say, one year, but after 5 years your probability of drive failure becomes quite unacceptable (although that is purportedly being addressed by mfgs with 'enterprise' grade drives). Since, I presume, most photographers want to keep their libraries intact longer than that, then one has to protect against device failure AND make sure that recreation strategies on several levels exist.

On a photographic level, all of this is MUCH easier if the file contains all its information and stands alone. Is xml workable? Yes, of course but it is not as good.

J.

This is why we all have offsite backups :) I might also start to replace some disks soon, my Seagate 120GB system drive is probably 4 years old and works perfectly, but I don't want to wait for a failure to find out something's wrong.

[JohnJ80]

This is why we all have offsite backups :) I might also start to replace some disks soon, my Seagate 120GB system drive is probably 4 years old and works perfectly, but I don't want to wait for a failure to find out something's wrong.

Yep. I believe that seagate has a 50K hour mtbf. If that has been on for 24/7, it is due to fail some time in about the next year to 18 months (on average).

The current crop of drives is supposed to have longer MTBFs by a considerable margin - like more than double. However, they are also more sensitive to altitude and temp so unless the conditions are perfect, you won't see that whole benefit.

J.

[tim]

I'll probably keep using it until it gives errors and dies, my regular backups will keep the data safe, and a replacement drive can be sourced by the next day... as long as you can get online to order one!

[JohnJ80]

I'll probably keep using it until it gives errors and dies, my regular backups will keep the data safe, and a replacement drive can be sourced by the next day... as long as you can get online to order one!

I've been getting pretty interested in this thing. I like the idea:

http://www.drobo.com/products_drobo.aspx

Accepts any SATA drive, and it automatically does a quasi RAID thing backup data. A drive goes bad, you replace it and don't have to worry about reloading the data.

I'll probably pick one up next month after I do a little more research.

J.

[JoesLdy]

OK I think I'm decided then, I'll keep my original RAW files off site and keep the DNGs here.

I think the loss of data I was referring to was the mfg data, I'm not real worried about that.

Since were all talking about hard drives, I've kind of been in a cave as far as technology goes for a while now so I am soooo behind on what is available. Isn't there some box out there that you can purchase hard drives for that you pop in to it to read? I'm using USB external drives right now for my storage in house, but I want to keep my back ups in a safe deposit box at the bank, and the ones I'm using right now are too bulky for that. I'm thinking there is a way to get slim hdd's that go in to a device... am I right here? If so anyone know of one they can point me to?

Thanks everyone for your help! :)

[JohnJ80]

OK I think I'm decided then, I'll keep my original RAW files off site and keep the DNGs here.

I think the loss of data I was referring to was the mfg data, I'm not real worried about that.

Since were all talking about hard drives, I've kind of been in a cave as far as technology goes for a while now so I am soooo behind on what is available. Isn't there some box out there that you can purchase hard drives for that you pop in to it to read? I'm using USB external drives right now for my storage in house, but I want to keep my back ups in a safe deposit box at the bank, and the ones I'm using right now are too bulky for that. I'm thinking there is a way to get slim hdd's that go in to a device... am I right here? If so anyone know of one they can point me to?

Thanks everyone for your help! :)

THere is also a spot in the DNG file for proprietary mfg data. So that may not be lost. You can also put the whole RAW file into the DNG file too.

What you are contemplating is pretty much exactly what I'm doing. Really though, check out ImageIngestor. It will make that so easy as part of your work flow.

The drive box you are talking about is the Drobo. See:

http://www.drobo.com/

I'm going to be getting one of these. Otherthan it is USB only, it looks like it would be great.

J

[kblair210]

/offtopic

"Other than USB only"

Being USB only is a very large deal, especially for the price. USB isn't very fast at all when it comes to hard drive access. I'd do some scouting on NewEgg for a better alternative.

/ontopic

From reading the thread it seems that storing the RAWs and then using DNGs is the ideal solution. Good information posted here, thanks.

[tim]

I've been getting pretty interested in this thing. I like the idea:

http://www.drobo.com/products_drobo.aspx

Accepts any SATA drive, and it automatically does a quasi RAID thing backup data. A drive goes bad, you replace it and don't have to worry about reloading the data.

That's a good idea, handicapped by a slow USB interface. Don't buy it.

OK I think I'm decided then, I'll keep my original RAW files off site and keep the DNGs here.

Why not keep the original RAW files with you, and archive the DNGs? Why did you decide on this split? I chose the opposite, just because i'd prefer to work from the originals no the DNGs. Either will work though, and I don't think it much matters which way you do it.

One thing to consider is when you update the DNG/XMP file you'll want that update in your archives. Make it easy for yourself.

[kblair210]

If you're going to work from the original RAWs anyway, what's the point of making the DNGs? Guessing to have a non-proprietary format archived?

[JohnJ80]

If you're going to work from the original RAWs anyway, what's the point of making the DNGs? Guessing to have a non-proprietary format archived?

Go back and re-read this thread. It was all explained and debated.

J.

[JoesLdy]

Why not keep the original RAW files with you, and archive the DNGs? Why did you decide on this split? I chose the opposite, just because i'd prefer to work from the originals no the DNGs. Either will work though, and I don't think it much matters which way you do it.

One thing to consider is when you update the DNG/XMP file you'll want that update in your archives. Make it easy for yourself.

I am going to go through and do the basic edits to these in RAW format in Lightroom, including my keywording and metadata entries, back up the RAW and sidecars for storage- then convert to DNG. The reason I thought it better to keep the DNG here to work with is that currently when I open files up in programs other than lightroom (unless I do it THROUGH lightroom) I lose all of my edit info. I figure the DNG will be up to basic edited par for me then I can go from there, and the info will be there for me. Make sense?

I tweak every image I capture, not sure if it's the old cameras I use or just me... but they all come out pretty flat. So I am trying to get my basic files up to par in the first place before I back them up (curves adjusted, wb corrected etc...) then have something more compact and faster to work from here in office to create web galleries, print orders etc... etc.. from.

On the HDD thing, I don't want a USB interface. They are slow as molasses- the two I run right now drive me insane. I could have sworn there was a kind of internal HDD carriage if that makes sense. It looked like you popped an internal drive in to it, they would carry about 4 drives... but I seriously doubt this was a USB device. I wish I could remember what they were, this was years ago- I thought they would have come up with something else by now new and improved.

[JohnJ80]

Keep your RAW files as the master backup up. Just edit in LR or whatever. Send it to whatever editing program that has what you need past LR through LR that way you have one copy of an edited file instead of two partially edited files all over - that is the worst nightmare of all. Also, the idea in keeping the RAW separate and as backup is that nothing is applied to it so you can always go back to the starting point. Even if you never go to DNG then you should still keep one set of RAW untouched as a backup and work another copy.

If you edit things without going through LR (or whatever database you are using) you are going to get things all hosed up. Either LR won't know what you did or your LR edits will get lost or something bad. All of the cataloging programs have mechanisms to keep track of what they did and to avoid this. In other words, this is not the place to get creative.

Incidentally, ImageIngestor will apply a profile to an image on import if you have a standard set of presets that you need to apply as you import it (if you want it to). Read up on it in the II manual.

So, if you use II then it reads your card, applies keyword and other information that you give it. It backs up the RAW files and stores an xml xmp sidecar file with them, creates your DNG files with the embedded metadata and, if you tell it to, will apply a Camera Raw Develop Setting preset to them. Then it opens LR, tells it what to import. LR them patiently sits there waiting for you to tell it go with the Import. Done - your workflow is managed and pretty much automated up to setting up LR.

On disk - The USB (I also hate USB) DROBO is the only one that I know of that you can plug in various sized sata drives. Otherwise, if you want automatic backup you are constrained to a RAID setup or get some auto backup software (or get a mac with its autobackup s/w Time Machine).

J.