Lightroom export time

Then sRGB got me to thinking. I chose a single typical image and exported it at all 4 color gamut choices. Look at the resulting file size differences.

I have 61,000 jpg images in this PC across 2 drives. Going forward if I can save 4MB per image it adds up to a significant savings in drive space.

IMAGE LINK: https://flic.kr/p/2kmY​dWz

chuckmiller wrote in post #19173557
My C and D drives are 1 year old ADATA 1TB NVMe M.2 mounted on the mainboard, my next internal drive is a 1 year old 4TB Samsung SSD (SATA) 860 Pro I think, and another internal is an older (SATA) 1TB SSD Samsung 840 pro I think. Today I Lightroom exported the same 68 original test files to all 4 drives with nearly identical timed results. That tells me that speed demon or not I don't have a significant bottleneck on any drive controller. I popped a screen shot showing the resource utilization for each task. All CPU 8 cores and 16 logical processors working 100%. RAM was 50% /used/available. GPU was barely touched, naturally. After all that I copied a big load of files from C to D and popped another screenshot. The CPU usage was vastly different. LR is taking full advantage of every CPU core, Windows does not, at least not my Win 10 Pro version .

So, using my original transfer as a baseline, my LR on this desktop PC will export approx 40-45 images per minute to any internal drive I have. 99% of the time I put 5D4 large RAW images into LR and exporting no crop full size sRGB JPG at 100% quality.

IMAGE LINK: https://flic.kr/p/2kmX​rYd LR Export 1 by Chuck Miller, on Flickr

IMAGE LINK: https://flic.kr/p/2kmY​1vd c to d file copy by Chuck Miller, on Flickr

Yes, the export is purely CPU bound whereas most "typical" software is still single or lightly-multithreaded and doesn't take advantage of all cores. Visual Studio does take full advantage during full compiles, as does Photoshop, Phocus, and Capture One. Pretty much the faster the cores, with the most optimal cache schema, and the least amount of noise within the QPI pathways and CPU-to-RAM, the faster the export. Cache-thrash will kill the performance of the cores, not matter how fast or how many there are.

With SSD or NvME, the JPG and database reads in LR are miniscule, as well as the actual committing of the exports. Is purely bound in the processing engine. Have you tried to see how it handles exporting untouched vs 1 edit layer vs files with tons of edits? That probably has a lot to do with the time as LR would have to process each of those edits during export, unless it kept a cumulative "flattened" record in the DB.

chuckmiller wrote in post #19173559
Then sRGB got me to thinking. I chose a single typical image and exported it at all 4 color gamut choices. Look at the resulting file size differences.

I have 61,000 jpg images in this PC across 2 drives. Going forward if I can save 4MB per image it adds up to a significant savings in drive space.

IMAGE LINK: https://flic.kr/p/2kmY​dWz LR color gamut exported file size comparison by Chuck Miller, on Flickr

That makes sense to me, as one of the techniques of compression takes advantage of repetitive patterns, so larger color gamuts would end up with more repeats. Especially on small format stuff, since most of it is 12 to 14-bit max anyways coming from the sensor, so you'll have a lot of zeros in the data. With medium format and other RAWS where 16-bit data is coming from the sensor, it can be different as the higher color bits would be filled with unique data instead of null bytes, so the JPG algorithm would take less advantage of that type of compression/optimizati​on.

chuckmiller wrote in post #19173559
Then sRGB got me to thinking. I chose a single typical image and exported it at all 4 color gamut choices. Look at the resulting file size differences.

I have 61,000 jpg images in this PC across 2 drives. Going forward if I can save 4MB per image it adds up to a significant savings in drive space.

IMAGE LINK: https://flic.kr/p/2kmY​dWz LR color gamut exported file size comparison by Chuck Miller, on Flickr

Pretty much the same with 16-bit RAW files. Of course, despite the different output gamuts, they're all still 8-bit output (JPG limitation):

docholliday_sc001 wrote in post #19173663
Have you tried to see how it handles exporting untouched vs 1 edit layer vs files with tons of edits? That probably has a lot to do with the time as LR would have to process each of those edits during export, unless it kept a cumulative "flattened" record in the DB.

Remember a RAW file is not a raster, it's a heavily compressed table of data from the sensor and LR, or any raw processor has to uncompress it and run an algorithm on that data to turn it into a raster file you can view (called demosaicing). LR does this to create the previews and then updates the raster from the raw as you make edits, which simply change variables in the algorithm. The settings for each variable are saved in the database with a reference to each RAW file (or along side it as an XML file).

When you do an export LR then looks up the data base for the variables and applies them to its algorithm which it runs on the raw file.
Essentially the time is spent looking up the database then decompressing and processing the RAW file into a raster format which is then written to disc as a TIF or compressed if saving as a JPEG.
The number of variables you change, and the amount you change them, when editing the raw has very little, if any, impact on the processing time, as even at default settings LR still has to run it's algorithm to demosiac the raw file and produce a raster.

docholliday_sc001 wrote in post #19173481
...The box is 2x Xeon E5-2667v4 @ 3.5GHz, 512GB DDR-4 2400 ECC, 4x NvME 1TB 960Pro (1-boot, 1-catalog, 2 for swap/temp in RAID0 on a Highpoint card), 8x Seagate Enterprise SAS12 900GB 15K in RAID 50+1 with 8GB cache (RAW files). .... The system has 128GB of RAM dedicated to disk caching using Primocache, so once read in (image load), the disks aren't touched for read again. The same cache also buffers out the writes to the SATA.

That is a heavy weight contender. Is it your daily driver?

chuckmiller wrote in post #19173836
That is a heavy weight contender. Is it your daily driver?

Yes, that's my daily desktop (well, she sits on the floor and weighs a bit). She's a bit older, as the procs are 4 years out, but I have her with 6x 24" monitors and it's what I do most of my work with. A lot of Visual Studio, SQL Server, Capture One, Photoshop, InDesign, and Studio One stuff. I also use quite a few sessions of VMWare Workstation for testing. I've built her for < $2000 using mostly used parts. My backup system is pretty much the same setup, using older E5-2667 v2s and 256GB RAM.

This will be an anomaly as I use Capture One 21. But, for comparison's sake......Although this appears to be disk I/O dependent more than anything.

Software: Capture One 21
Camera: 1DX3
Files: 78 RAW >> 100% JPG
Computer: Processor Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz 3.70 GHz
Installed RAM 64.0 GB
System type 64-bit operating system, x64-based processor
Total Output size: 1.54G
Time: 58.68 seconds

HKGuns wrote in post #19174061
This will be an anomaly as I use Capture One 21. But, for comparison's sake......Although this appears to be disk I/O dependent more than anything.

Software: Capture One 21
Camera: 1DX3
Files: 78 RAW >> 100% JPG
Computer: Processor Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz 3.70 GHz
Installed RAM 64.0 GB
System type 64-bit operating system, x64-based processor
Total Output size: 1.54G
Time: 58.68 seconds

It's not really that much of an anomaly...I use C1 (20 & 21) and Phocus as well as LR (LR for small format, C1 for my IQ backs, and Phocus for H5/H6). LR is horribly inefficient and the slowest. I can do the same 68 files from a 1DX in C1 in under 15s, but with the MF files, it's slower. It's also *very* dependent on how many layers are in your edits. An unedited file exports in C1 so fast that the Activities window doesn't pop up, but with some that have max layers (15), it can take 2-3 seconds to export.

The thing that I love about new C1 is the speed edit stuff...no need for a third-party crappy MIDI controller to get fast editing with one hand on keyboard, one on mouse!

docholliday_sc001 wrote in post #19174076
It's not really that much of an anomaly...I use C1 (20 & 21) and Phocus as well as LR (LR for small format, C1 for my IQ backs, and Phocus for H5/H6). LR is horribly inefficient and the slowest. I can do the same 68 files from a 1DX in C1 in under 15s, but with the MF files, it's slower. It's also *very* dependent on how many layers are in your edits. An unedited file exports in C1 so fast that the Activities window doesn't pop up, but with some that have max layers (15), it can take 2-3 seconds to export.

The thing that I love about new C1 is the speed edit stuff...no need for a third-party crappy MIDI controller to get fast editing with one hand on keyboard, one on mouse!

Yeah, it is hard for me to verify I actually hit the button when I output a single file I've been working on it goes by so quickly. I was actually surprised it took as long as it did to export those files. But, it is what it is........I've never used LR so I am ignorant to any comparisons.

6 displays? What video card? Do you run a boot manager? Is Win 10 your primary OS?

docholliday_sc001 wrote in post #19173481
...

The box is 2x Xeon E5-2667v4 @ 3.5GHz, 512GB DDR-4 2400 ECC, 4x NvME 1TB 960Pro (1-boot, 1-catalog, 2 for swap/temp in RAID0 on a Highpoint card), 8x Seagate Enterprise SAS12 900GB 15K in RAID 50+1 with 8GB cache (RAW files). Exported to old 7200 RPM Barracuda 1TB spinning rust in SATA3...

58 seconds is pretty speedy. Good hardware performance, or do you have to give the credit Capture 1?

HKGuns wrote in post #19174061
This will be an anomaly as I use Capture One 21. But, for comparison's sake......Although this appears to be disk I/O dependent more than anything.

Software: Capture One 21
Camera: 1DX3
Files: 78 RAW >> 100% JPG
Computer: Processor Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz 3.70 GHz
Installed RAM 64.0 GB
System type 64-bit operating system, x64-based processor
Total Output size: 1.54G
Time: 58.68 seconds

chuckmiller wrote in post #19174436
58 seconds is pretty speedy. Good hardware performance, or do you have to give the credit Capture 1?

Both actually, given what the doctor said above. My machine is pretty robust but apparently CO performs well as compared to LR.

chuckmiller wrote in post #19174428
6 displays? What video card? Do you run a boot manager? Is Win 10 your primary OS?

Dual Quadro P4000's. No boot managers with Win10 Pro as OS. I can't stand Linux, as I preferred Solaris and was a DEC VAX and AT&T System V guy back in the day. I do run VMWare on Win10 with a few other OS installed for testing software.

HKGuns wrote in post #19174486
Both actually, given what the doctor said above. My machine is pretty robust but apparently CO performs well as compared to LR.

The i9 is pretty much a stripped down Xeon. The extra L3 cache and additional cores help to prevent cache thrash and reduce the bus chatter & latency from cores-to-cache-to-RAM. And with each core running much higher, it helps speed up the export. In a nutshell, pretty much exporting is a single-threaded operation, so each core in a system gets assigned one image and runs to completion before getting another. There's the work stealing and offloading of math functions to OpenCL/CUDA, but there is much less multi-threading to an export operation than other functions.

Single threaded benefits more from high core speed, whereas multi-threaded functions don't always do so. Cache latency and size can affect multi-threaded a lot as too much data or poor data block sizing in code can cause a cache eviction when a core switch occurs. L1 is the lowest latency, L2 the next and so on with RAM reads begin the slowest. Each time data gets evicted, it goes to the next higher ring and to read it back in takes longer. If the data is complete evicted from the proc, it'll have to be read back in from RAM, which is an eternity to a processor (and the core "stalls" waiting for data to come in).

On top of all that is the efficiency and quality of the code and how it handles blocks of data. LR is sloppy. Give it big files and watch it choke. C1's code design is much more efficient (when they don't have a stupid bug that takes them a month to fix). But, LR has better DAM organization whereas C1 does have a neat album/group/project method, but isn't optimized for the hobbyist cataloging their life's work on it. Phocus seems to be the fastest with the "tightest" code, but doesn't really organize at all.