Bulk converting 100,000 RAW images to jpeg - recommend software / technique

I have around 100,000 raw images in CR2 and NEF format, spread across a deeply nested set of folders. I want to create a medium resolution jpeg image of each raw image, in a separate but matching folder heirarchy, so I can back them up online. There are some psd and other file formats too, total around 1TB.

This is a last line backup of 2000px Q6 jpeg files. I don't tend to back up the raw images online, but they're on at least three hard drive in three locations.

I've done something similar using the PS CS6 image processor in the past, but it tends to crash, and there's no way to restart without manually working out where it got to. I don't have Lightroom. I'm happy to use open source / free software, so long as the image quality is good enough to give to customers if all my other backups fail - which is unlikely. Total size is 1TB so it's not very practical to use an online service.

Anyone have any recommendations?

XNViewMP:

http://www.xnview.com/​en/xnviewmp/

You can point it to a hierarchy of files and folders and it will convert and maintain the hierarchy.

It is free/donationware for private use, and low-cost if you intend to use it for professional use.

XNConvert is a separate tool that is the batch conversion core:

http://www.xnview.com/​en/xnconvert/

See if that is better for your task than the full-fledged XNViewMP.

kirk

Also - for the raw files, you may be able to extract full-res JPEGs and resize/compress those as well. Depending upon the camera settings and how you expose in-camera, this might sufficient. dcraw will do this (dcraw -e *.CR2 on a directory of CR2's for example) - you may be able to script this to work recursively through the hierarchy.

Have any of these RAW files been post processed? If so then they will need to be exported using the same software of the post processing will be lost.

I don't have any other answers, but was also going to suggest jpeg extraction vs/ conversion. Extraction takes less than a second,. very fast.

Breezebrowser is what I used to use. (always shot RAW, and if I needed jpegs would extract,. entire folders, hundreds of images in maybe two minutes or less.)

DCRAW can do it,. I have never found the tool in DPP, if it exists.

Faststone supposedly can do it with multiple brands (Breezebrowser is only Canon) but again, I have never used it.

P.S. in another thread, I mentioned that I used to use a batch file to make copies of everything but the RAW files in a fully in tact directory structure identical to the original.
This copied only the file formats specified,. jpeg, tiff, psd. ie: only the files I have processed, and not all those unculled RAWs

Here is the text for the batch file.


xcopy D:\stuff\*.gif D:\test\ /d/s/v

It works great. If you use /e instead of /s it will copy the entire directory structure, even folders without .gif (in this example) but the folders without gif will be created empty.

I'm about to run this bad boy now on near 3TB of directories stuffed with RAW. It will result in a back up of maybe 500GB,. not sure yet.

Here's the big one with three file types;

:: tif
xcopy f:\digita~1\*.tif o:\tifbak\ /d/s/v

::jpeg
xcopy f:\digita~1\*.jpg o:\tifbak\ /d/s/v

::psd
xcopy f:\digita~1\*.psd o:\tifbak\ /d/s/v

Dan Marchant wrote in post #18354181
Have any of these RAW files been post processed? If so then they will need to be exported using the same software of the post processing will be lost.

Unfortunately, I think this is the answer. I'll have to export using Bridge or batch using the PS image processor. Thanks for pointing that out

Thanks for the ideas Kirk, Jake. They would've been worth looking at if not for what Dan said. I need fully processed, high quality.

I know that you said that you do not use LR, but in this case it would be the perfect tool. You should be able to do it within the trial period too. LR will quickly import all the images, since they will be staying in place on the disk. LR will also pick up all of the processing that you have previously done in ACR. It will also collect TIFF and PSD files too. Go to the All Images view, Ctrl+A will select all the images for you. All you then need to do is export them as your 2000px long edge JPEG in sRGB and use Quality 50, which corresponds to Q6 on the 0 to 12 scale.

The only difficulty that I can see is that of duplicating the file structure. The best I can think of is to put them in a subfolder of the original folder, to keep them separated, just in case you already have some JPEG files in the tree. Pulling the subfolder, which will have the same name at every level of the tree, to move it should then be a much simpler matter. At least this way you have a JPEG file for every image, complete with all the editing that you used in ACR.

Alan

the nef/raw have previously been edited by ACR?

with sidecar files?

Sadly you don't have LR, which can certainly do it. Not 100,000 at once but you can break them down into smaller batches and start several exports running at once. Can you not just batch export in PS but do several smaller batches?

BigAl007 wrote in post #18354365
I know that you said that you do not use LR, but in this case it would be the perfect tool. You should be able to do it within the trial period too. LR will quickly import all the images, since they will be staying in place on the disk. LR will also pick up all of the processing that you have previously done in ACR. It will also collect TIFF and PSD files too. Go to the All Images view, Ctrl+A will select all the images for you. All you then need to do is export them as your 2000px long edge JPEG in sRGB and use Quality 50, which corresponds to Q6 on the 0 to 12 scale.

The only difficulty that I can see is that of duplicating the file structure. The best I can think of is to put them in a subfolder of the original folder, to keep them separated, just in case you already have some JPEG files in the tree. Pulling the subfolder, which will have the same name at every level of the tree, to move it should then be a much simpler matter. At least this way you have a JPEG file for every image, complete with all the editing that you used in ACR.

Alan

I really want them in a separate tree. I suspect LightRoom could do this but I don't know how. I really don't like LR.

I'm currently running through year by year with image processor, it's working ok.

http://www.techfleece.​com …les-to-jpeg-in-photoshop/

Yeah, if you need fully processed output you are tied to the application(s) that generated the edits.

kirk

DagoImaging wrote in post #18354600
http://www.techfleece.​com …les-to-jpeg-in-photoshop/

So... what I said I had already tried?

It's actually working ok. I'm not trying to batch 100,000 images at a time, I'm doing it by year, which is working better. So I think Image Processor is the winner.