Fun With Highlight Testing/Recovery!

Hey all!

A while back, someone posted in a thread here about the fact that a Raw file held at least a stop more highlight data than a jpeg.

Now, a lot of us accept that "in theory", but the poster presented a practical test:

It's always considered "safe" to expose whites to +2EV to get them nice and bright, but a level that a jpeg could handle. The test, then, was as to whether you could set whites to expose to +3EV, which would totally blow a jpeg away!

Now, how often would I do this in the "real world"? Hmm...maybe I'd want to back it off "a bit" -- in fact, the idea was tossed around in that thread that maybe +2 2/3EV would be "safe" for a Raw shooter.

But either way, it's fun and intriguing to explore the "real" limits of "ETTR", in other words, how far can you push those highlights?

Now, one thing here is that it is also helpful to use your histogram and highlight "blinkies". But those warnings can be misleading, in that they represent a jpeg rendered using your in-camera Picture Style. So, those of us who have been interested in exploring ETTR/HAMSTTR have typically set our Picture Style to Neutral and turned the Contrast and (for colors) Saturation settings down to -4, so that we don't start getting those warnings at, say, +2 1/3 EV. The warnings are more accurate.

I remember running my 1D3 through this exercise, and it was quite successful -- when I exposed white as +3EV, the camera didn't sound of warning bells!

Well, earlier today I decided to take on a more thorough test project with my old 5DC. Through some preliminary tests, I suspected the 5D metering would not hold out so well once you got to +2EV...

Well, I was right, the camera unfortunately has some problems there, and definitely at +3EV.

But I shrugged my shoulders and plodded on, using a Photoshop New Document as a white metering target, and worked my way from a "medium" exposure up by stops to +3EV, and got some pretty cool results, I just had to ignore the histogram and the blinkies!

I'll post a couple images, from my +3EV shot, one just using the Lightroom defaults, and then one after processing to tame the wild whites.

But, for those who are either interested or bored, here is the whole project and write-up, including notes for the various shots:

http://www.pbase.com …2011_highlight_​limit_test

So here is the +3EV shot, obviously "over-exposed", and this is what your jpeg would tend to look like and be limited to as far as highlight recovery:

Tony...I've read this, and the link, about 4 times and I can't understand your "test". Your shooting a pure white subject with no texture or anything else in the image. Would not have taken a series of images of something white that has a commonly recognizable form have served you better?

I do know that if I'm shooting egrets and push that histogram more than one stop off the right side, trying to dial it back in LR with highlight recovery produces something that reminds me of a unattractive, muddy gray color.

Hi Tony,
Entirely by coincidence I did a bit of playing with this yesterday. I took a few shots of a blank wall in order to check something about LR's rendering of a medium (12.5%) tone and decided while I was at it to expose a few frames where I ETTRed the wall using a nonconventional method of metering I have adopted recently - using the Live View real time histogram - to see how much clipping alarm in LV is a false alarm. I used the Uni-WB custom WB in order to make the histogram more representative of the RAW capture and so that only the green channel would clip. The clipping alarm in LV is that the +2 on the EC scale flashes and I took 6 shots:
1. When the flashing first started and the histogram was still somewhat distant from the right margin (+2.33).
2. When the green histogram was a hair's breath before the margin was 1 stop more (+3).
3. Another 1/3 stop (+3.33).
4. +3.67.
5. +4.
6. +4.33.
First I opened them in DPP because it has the RAW tab which is marked in stops. #1 lined up slightly above the +2 line, #2 was halfway between +2 and +4. So far so good, but here's the weird part - the histogram for all the other four was identical, around 3.5, as was the numerical readout (237/250/244), although there was a a whole stop difference between #4 and #6. Nor was DPP's clipping alarm activated although the camera review was blinking like crazy from #3 onward.

Next I imported them into LR3 with all the Basic sliders zeroed. [Because they were shot with Uni-WB the WB sliders were at 2050 and -150. I mention this because this is a useful setting when you want to remove the effect of WB from the LR histogram when your image seems to be clipping, in order to know if it is really clipped.] Profile was Camera Neutral. Shot #1(+2.33) had the green channel at 78%, well below clipping, and #2 had the green at 92%, still with a safe bit of headroom. #3 set off the clipping alarm but only -0.1 Exposure or Recovery 3 was needed to turn it off. #4 needed -0.5 Exposure or Recovery 12. #5 and #6 were thoroughly blown and nothing could remedy that.
Conclusion: Considering the very slight clipping at +3.33, I think it's reasonable to say the 5D2 hits saturation at 3.25 stops above medium grey.

One final point: the results above were obtained with the Neutral profile. Changing that to Adobe Standard inflates all the levels. In #1 green became 96% and #2 was blown and required -0.4 Exposure to recover. #3 and #4 needed -0.52. Camera Standard raised levels much less, taking #1 to 80% and #2 to 97%.

chauncey wrote in post #13364910
I do know that if I'm shooting egrets and push that histogram more than one stop off the right side, trying to dial it back in LR with highlight recovery produces something that reminds me of a unattractive, muddy gray color.

That is the because of the way Adobe does recovery. Data from the unclipped channel(s) is interpolated into the clipped channel(s), but because this is interpolated, not original, data, there is a danger of creating false colors. Adobe solves the problem by equalizing the channels to render the recovered highlight as grey. Other converters inspect the surrounding (unclipped) pixels and render the highlight the same color. Sometimes one method works better, sometimes the other.

BTW, and entirely OT from the OP: Although I have dozens of times advised people to test their cameras doing the wall test, I have never done it before with my 5D2 because I always shoot RAW and I don't care much how the camera processes jpgs or hoe the lcd review looks. Anyway, I discovered that the camera makes jpgs a full stop overexposed. That surprised me because the 40D and the 350D underexposed jpgs by about a third of a stop.

I might suggest Elie, that is because of how you have dialed in the camera settings.

chauncey wrote in post #13365281
I might suggest Elie, that is because of how you have dialed in the camera settings.

chauncey, you're brilliant!

It was because of the Uni-WB. I made another custom WB for the light on the wall and the exposure was spot on. Now I have to go away and sit in the corner very quietly and think about this.

My first thought is that Canon is pulling (reducing) the green channel during white balancing, but this is contrary to what is written in the metadata.

Aw pshaw...now I'm gonna need to repair my rotator cuff from excessive patting myself on the back.

tonylong wrote in post #13362225
Now I know this is boring as all get-out, but there is at least one important thing to note:

If you look closely, you will see Radio Frequency Interference that the camera picked up. This RGI was noticeable in my shots that had a lower exposure, quite so at 0EV and +1EV, but at +2EV you'd have to look really close to see any evidence at all, but as you can see, at +3EV the highlight clipping/blowing swallowed any trace of it. But the Raw file did capture it -- in fact when I pull the Exposure slider all the way to -4EV it's very noticeable, more than it was at "medium"/0EV. In other words, the detail in the white area was captured by the Raw!

Tony, is the RFI you're referring to the purplish lines on the monitor? I believe that is due to the refresh rate of your monitor, not RFI. I tested this, a couple of months ago. If you set your shutter speed to match the vertical refresh of your monitor, you will really see those lines

Hey, I'm just getting back to this for the first time today, so I'll take a bit to respond one-by-one.

And thanks, gang, for taking the time to read the post and check out the whole project!

Any of you who do find some value in it, leave a comment in PBase and say "HI" (you can do it as a guest, just tell me where you're coming from!

chauncey wrote in post #13364910
Tony...I've read this, and the link, about 4 times and I can't understand your "test". Your shooting a pure white subject with no texture or anything else in the image. Would not have taken a series of images of something white that has a commonly recognizable form have served you better?

Chauncey, that was of course a concern. But one thing that was at the back of my mind is that monitors have actual dots, pixels according to the monitor's resolution. I didn't check this out, but it's reasonable to hypothecize that if you brought the exposure down enough and then did a 100% view from the 23MP image, that those pixel "details" might be quite visible. But, I didn't shoot from a tripod, I was just sitting at my desk chair...playin' around...

But, did you note the patterns, whether interference or (as suggested below) monitor refresh "artifacts"? Those were clearly details in the white screen (especially at 0EV and +1 EV), right? And the +3EV "blew them out" until I applied the exposure/recovery/high​light adjustments, correct? So that to me is the "proof of concept" I was after! And, if you noted at the very end of the project page, when the whites were truly clipped, then those details aren't there at all!

OK, Elie mentioned one factor, which is the Adobe "method" for doing highlight recovery.

If you look at the last three pics in the test page, you will see that if there is any data in a "white" image, and you pull back on the exposure, it will make the white, well, a shade of gray! But if the whites are truly clipped, then you willd see that Lightroom/Camera Raw won't turn that white to gray. Did you note that in that last example I had pulled Exposure all 4 stops back, Recovery all the way to 100 and Highlights all the way back to -100? And the result in the clipped areas was still pure white.

Now, as to the white critters, well, tell me -- when you say you "push that histogram more than one stop off the right side" are you referring to the fact that you are setting your exposure so the whites are one stop "over the edge"? That is pretty much what I've covered, you are setting "about" a +3EV for the whites, correct? If so, that means that the white is not actually "clipped", meaning that there is data there, and that pulling back the exposure will result in white becoming gray, together with the stuff Elie was discussing.

In that case, if say a white egret/goose/whatever was in the sun (or nice light ambient, I'd want it to turn out a nice bright white, meaning I'd expose it to something up to +3 EV although, as I've mentioned, I'd be more likely to back off a bit from that, maybe +2 2/3 EV, but as I said, that would be if I wanted it to be white. If it was in the shade, I'd likely take other things into consideration, maybe expose another object, or maybe expose the critter to be a bit brighter than "medium".

tzalman wrote in post #13365103
Hi Tony,
Entirely by coincidence I did a bit of playing with this yesterday. I took a few shots of a blank wall in order to check something about LR's rendering of a medium (12.5%) tone and decided while I was at it to expose a few frames where I ETTRed the wall using a nonconventional method of metering I have adopted recently - using the Live View real time histogram - to see how much clipping alarm in LV is a false alarm. I used the Uni-WB custom WB in order to make the histogram more representative of the RAW capture and so that only the green channel would clip. The clipping alarm in LV is that the +2 on the EC scale flashes and I took 6 shots:
1. When the flashing first started and the histogram was still somewhat distant from the right margin (+2.33).
2. When the green histogram was a hair's breath before the margin was 1 stop more (+3).
3. Another 1/3 stop (+3.33).
4. +3.67.
5. +4.
6. +4.33.
First I opened them in DPP because it has the RAW tab which is marked in stops. #1 lined up slightly above the +2 line, #2 was halfway between +2 and +4. So far so good, but here's the weird part - the histogram for all the other four was identical, around 3.5, as was the numerical readout (237/250/244), although there was a a whole stop difference between #4 and #6. Nor was DPP's clipping alarm activated although the camera review was blinking like crazy from #3 onward.

Next I imported them into LR3 with all the Basic sliders zeroed. [Because they were shot with Uni-WB the WB sliders were at 2050 and -150. I mention this because this is a useful setting when you want to remove the effect of WB from the LR histogram when your image seems to be clipping, in order to know if it is really clipped.] Profile was Camera Neutral. Shot #1(+2.33) had the green channel at 78%, well below clipping, and #2 had the green at 92%, still with a safe bit of headroom. #3 set off the clipping alarm but only -0.1 Exposure or Recovery 3 was needed to turn it off. #4 needed -0.5 Exposure or Recovery 12. #5 and #6 were thoroughly blown and nothing could remedy that.
Conclusion: Considering the very slight clipping at +3.33, I think it's reasonable to say the 5D2 hits saturation at 3.25 stops above medium grey.

One final point: the results above were obtained with the Neutral profile. Changing that to Adobe Standard inflates all the levels. In #1 green became 96% and #2 was blown and required -0.4 Exposure to recover. #3 and #4 needed -0.52. Camera Standard raised levels much less, taking #1 to 80% and #2 to 97%.

Cool, Elie! You covered the same ground, and then filled in some details! You saw that I just did full stops, right? So, since I was happy with the +3EV results, I just ran off the +4EV shot to show the results of "true clipping".

So, you finding that there is recoverable data at "maybe" +3.25EV confirms what I'm after -- 'course I wouldn't know how to hit that with a camera, meaning that +3EV may be as close as we can get without clipping something!

Did you see my reply to Chauncey, where I pointed out that you only turn white to gray if there is "some" data in the white shot, and how, at least in Lightroom, a truly clipped white won't be pulled back into gray?

tzalman wrote in post #13365244
BTW, and entirely OT from the OP: Although I have dozens of times advised people to test their cameras doing the wall test, I have never done it before with my 5D2 because I always shoot RAW and I don't care much how the camera processes jpgs or hoe the lcd review looks. Anyway, I discovered that the camera makes jpgs a full stop overexposed. That surprised me because the 40D and the 350D underexposed jpgs by about a third of a stop.

chauncey wrote in post #13365281
I might suggest Elie, that is because of how you have dialed in the camera settings.

tzalman wrote in post #13365416
chauncey, you're brilliant!

It was because of the Uni-WB. I made another custom WB for the light on the wall and the exposure was spot on. Now I have to go away and sit in the corner very quietly and think about this.

My first thought is that Canon is pulling (reducing) the green channel during white balancing, but this is contrary to what is written in the metadata.

Heh! Are you suggesting that even us old dogs can learn new "tricks"?

Str0ntium wrote in post #13366421
Tony, is the RFI you're referring to the purplish lines on the monitor? I believe that is due to the refresh rate of your monitor, not RFI. I tested this, a couple of months ago. If you set your shutter speed to match the vertical refresh of your monitor, you will really see those lines

OK, this is the patterns I was referring to (close crop here):

Yeah. That's definitely your monitor's refresh rate. The slower your shutter speed gets, the more pronounced it will be, too. I generally don't take pictures of stuff on my screen, so it's not too much of an issue haha.

You ever notice that if you see a CRT monitor on TV, you can see the lines rolling on the monitor? Same thing.

Ah, well, maybe, although at the fast shutter speed (1/250) it's pretty pronounced!: