Dynamic Range-Can't they or Won't they?

Finally we get to your key misunderstanding of the difference between the real world of sensors (which record photons and NOISE) and file systems that are artificial. In the computer you assign values - a pixel that is "white" is given maximum (16,000ish, e.g.) and one that is "black" is assigned a value of 0. So you think that down in the black there is no noise. That is true in the computer synthesis of an image but it is completely wrong when thinking about how sensors work. The bold text in your quote above is completely false - Wilt pointed that out before (#445).
The importance of understanding the concept of noise in the real world data captured by a sensor is still something you have not grasped, and that is confusing you no end. Wilt's diagrams in #320 and my rainfall bucket analogy (#330) seem to have been ignored by you - or you do not understand them (or as you have put it "you show ignorance on noise"). This also leads you to call the discussion (in which I and others have tried to help you understand and learn) "asinine".
In #440 you said

I still do not understand what you mean by a "full exposure" and what the sentences mean. In a 16 bit computer file, any pixel can have an assigned luminance value and it can be between 0 and 16384. No problem. Real world not so much.
When we expose a pixel to incoming light, some of the photons gets converted to electrons and stored until read-out. Let's say we have 200 photons landing on each pixel on average and half are converted to electrons. We have 100 electrons on average in such a pixel. Statistics of photon counting (unavoidable in the real world) means that 2/3 of those pixels will have 90-110 electrons and the rest will have <90 or >110 electrons. So there is one source of noise. When the electrons are "read-out" more noise is added from the circuitry and interference from the "electronic environment" and then they are digitized. This read noise is in addition to the statistical noise. So those pixels that all got, on average, 200 photons will be read out as 80-120 ± read noise. So some of the 80 pixels will be reported as 60 and some as 100, if the average read noise is say 20. And some of the 120 pixels will be read out as 100 and some as 140. So now our incoming photons value will be reported as somewhere between 60 and 140, ready to be encoded into the digital word.
In the computer you assign all these pixels (in a simulation) as having a value of 100, but those same pixels coming from a real sensor will report values anywhere from below 60 to above 140 - simply because of the noise in the capture/read/digitize process. So "black" is nowhere near "no noise" - in fact it is the opposite: it is where the number of photons is smaller than the noise, so we can no longer distinguish between 10 and 20 photons because the noise is too high. Thus we cannot see image detail from those photons because of the "snow" from the noise. In the terms we (but, so far, not you) have been using above, the "photon signal" gets lost in the "sensor system noise". Now, it should be apparent that two sensors with different "read noise" will have different abilities to resolve differences between e.g., 40 and 60 photons and to distinguish such a signal from the noise.

Perhaps a slightly different example will help. Back to the rain bucket
The upated system now directly reads the amount of rain and there is a display that goes up to 16,384.
We shine a light on the pixel and find out how bright it needs to be to read 16,384ish (top of detector DR)
Now we keep lowering the brightness and the number goes down. Someone at the back of the room asks "What happens with no light? Let's put the lens cap on " We now look at the display and, horrors, it doesn't read "0". It is flickering around what looks like a value of 32 (goin from perhaps 20 - 44 at any instant, but averaging 32. Now we take the lens cap off and slowly turn the light up from its off position. As the light gets brighter and brighter, from 0.1 to say 5, we can't tell just by looking at the display, whether the light is on or off. Perhaps by 10 we'll be able to see a difference between turning the light on and off. Now, by 32, it will be a definite that everyone can tell when the light is on and when it is off. We have now found the bottom of the DR (from 16,000 down to 32). The Canon display flickers around ~32 while the Sony display flickers aroud ~8 (relatively speaking in this analogy). So we can accurately record light levels 2 stops lower, for the same amount of noise.

That was my last attempt to explain the importance and concept of noise - how it occurs in the real world but not in simulations.

(Added: Not totally coincidenatally some real data: Read noise at ISO 100 for 5D2 is 30.6 electrons and saturation is 61072 electrons from http://www.sensorgen.i​nfo/CanonEOS-5D-Mark-II.html )

Tom Reichner wrote in post #17747897
.
Ok, I'm going to go back to the beginning of this thread and try to address the issue the OP brought up by providing an example of an image in which I think that the dynamic range was not sufficient to capture the scene the way I wanted to.

Ok, then........here is an example that is very "real world". . And it was not underexposed; in fact, I had to over-expose the highlights in order to try to minimize the noise.

Someone saw an (iPhoto) edited version of this on my website, and wanted to order a 48" by 32" print. . It would be for a smallish room in which the print would be viewed from a distance of 4 to 10 feet.

Here is an unedited version of the photo:

Hosted photo: posted by Tom Reichner in
./showthread.php?p=177​47897&i=i2607689
forum: Canon Digital Cameras

Unfortunately, I couldn't offer such a print, as the image quality is absolutely horrendous. . I showed the image to the potential buyer on my computer - zoomed in to what it would be like if blown up to 48" - and they were sooooooo disappointed. . The flame in the tree is blown out; there was actually a lot - and I mean A LOT - of detail in the burning portion of the tree. But if I exposed for that then the dark areas and mid-tones would have REALLY bad noise grain. . So I pushed the exposure a bit so that I would just get really bad noise instead of REALLY bad noise. . I knew this would be a problem when I was out there shooting it, but didn't know what I could possibly do to get really awesome IQ at every level. . All the while I kept thinking about the "superior dynamic range" that Sony is known for, and was wondering; if I had one of those "7" cameras (you know, the Sony mirrorless ones that are all the rage) would the IQ be better with regards to the ability to capture all of the detail in the flame and still get really smooth noise-free dark areas?

I don't really know how well a Sony would have done here. . But as it is, the image I shot with my Canon is really un-usable when it comes to decent sized prints. . BTW, I did shoot a few hundred frames at various exposure values and ISO settings. . Unfortunately blending (HDR) isn't really possible because the flying sparks are different in every frame, as is the smoke above the tree.

Basically what I learned is that if one is shooting super-bright burning things in the middle of a pitch-black night, and some of those things are dancing and flying in the wind, and you want/need to capture the scene with one exposure and get awesome IQ, basically SOOC ..... my Canon 1D4 simply isn't up to the task. . It didn't do the job I gave it to do. . It couldn't do the job I gave it to do. . Would the Sony have done the job? . Would I have been able to sell the print if I took this shot with that Sony mirrorless camera? . I don't know. . All I know is that Canon didn't, and I lost a sale.

Fortunately, most of what I shoot/make a living at shooting does not involve a need to capture a high dynamic range.

At such high ISOs, even the Sony results wouldn't have been good. You can only push high ISO shadows before they fall apart. From all indications, Canon is actually a bit better in its latest offerings at high ISO, or at least some well known reviewers point this out. Sony's strength is in shooting lower ISO and pulling up shadows.

That image would have been difficult to juggle shutter speed, DR, and ISO, for anyone, regardless of brand, IMO. It's too bad it didn't work out for you, that would have been a great large print content-wise!

Here is a lower ISO where I took a single shot, and tried to bring up the shadows. I am not very good about using curves, exposure settings, brightness, etc for this type of work however. The artificial light sources don't help any either.

Raw to JPG, then altered the raw to bring shadows up, then curves in PS on the JPG. Not great, but some of the detail is still there with no banding. This is what, perhaps 3 stops?

Whoa....looks like you're still not understanding file structures again. In relation to DR, a Sony sensor has a DR that fills the entire tonal range of a 14bpc system. If you expose the sensor to it's saturation point, the value of light past that gets clipped. Here I thought we were making progress. So you don't agree that if the Canon sensor gets 12 stops of DR, the last two tonal stops can be "noisy" in 16bpc? I do understand sensor DR, and I've tried to explain how bit depth works. If you don't want to acknowledge that sensor DR and bit depth are tied together, then this is going back to noise

Numerous times I've tried explaining that "acceptable" noise (or "measured" as you've sometimes said) is that "value" that's black....and the point of "saturation" is the "value" that's white. Also how "tone-mapping" works. In a digital system, the camera has to find where the black value is and where the white value is (where the noise threshold is and where the point of saturation is), and then "tone-map" it. If you refuse to acknowledge that, or want to try explaining to me how the Sony sensor can record over 14 stops, then we've made no progress.

That reply you quote clearly shows how the last "stop" of the Nikon is clean black in a 14bpc tonal system! The last bit (2 tonal values, 0 or 1) DEFINES BLACK!!!!!!!

Some good info there, hopefully he understands it.

I tried describing this once using a concert as my analogy. Let's say you wanted to record the concert, down to the very quietest note. Now let's say that you have a crowd of people constantly whispering and humming, throughout the entire presentation. When you take your master recording, you will be dismayed to find out that all the whispering and humming by the people in the audience interferes with the quietest points of the concert, to the point that if you use sound filters to get rid of that, you also lose the actual concert notes.

Your professional recording equipment was able to capture much more "audio DR", however the people restricted your usable "audio DR". The electronics in the camera, and the way light falls onto the sensor depending on the optics and design of the sensor wells, depth of sensor wells, etc, are the people whispering and humming. Your sensor is statistically capable of giving you so much more but unfortunately there are other factors that hinder that.

I am often really bad at analogies, so this may not work.

davesrose wrote in post #17747975
Whoa....looks like you're still not understanding file structures again. In relation to DR, a Sony sensor has a DR that fills the entire tonal range of a 14bpc system. If you expose the sensor to it's saturation point, the value of light past that gets clipped. Here I thought we were making progress. So you don't agree that if the Canon sensor gets 12 stops of DR, the last two tonal stops can be "noisy" in 16bpc? I do understand sensor DR, and I've tried to explain how bit depth works. If you don't want to acknowledge that sensor DR and bit depth are tied together, then this is going back to noise

Numerous times I've tried explaining that "acceptable" noise (or "measured" as you've sometimes said) is that "value" that's black....and the point of "saturation" is the "value" that's white. Also how "tone-mapping" works. In a digital system, the camera has to find where the black value is and where the white value is (where the noise threshold is and where the point of saturation is), and then "tone-map" it. If you refuse to acknowledge that, or want to try explaining to me how the Sony sensor can record over 14 stops, then we've made no progress.

That reply you quote clearly shows how the last "stop" of the Nikon is clean black in a 14bpc tonal system! The last bit (0-1) DEFINES BLACK!!!!!!!

You talkin' to me?

I totally understand file structures when creating images from scratch without a sensor : set 0 to black and (2^n)-1 as white where n is the bit depth.The computer lets you do that. However, you still have completely, and by now I think it's fair to say "wilfully", ignored the concept of noise in the real world and trying to RECORD "black" - when we talk about sensor DR that is the real world and file structure is COMPLETELY irrelevant. Did you see the data from Sensorgen that has been presented (or linked) for you many times? Those values are the ones BEFORE the signal is digitized (i.e. the raw data). The lowest value recorded is not zero. Once the processor decides what to ASSIGN as black, THEN it is assigned the last bit in the file it creates and writes. (there are special pixels that don't see any light that are used expressly to determine dark "noise" and help the processor decide to assign the values in the digital word).

TeamSpeed wrote in post #17747978
Some good info there, hopefully he understands it.

Not yet, apparently - still stuck in the file structure in the computer but perhaps my addendum will help...

Yes, I'm talking to you!!! You clearly don't understand bit depth. No recorded file from a DSLR goes above 14 stops of tonal range. If a Nikon camera exposes 14 stops of light, it's 14 stop sensor will fill the full 14 stops of tonal range. Like this:

2,4,8,16,32,64,128,256​,512,1024,2048,4096,81​92,16384

A Canon sensor, if exposing 14 stops of light, will have it's tonal range compromised in the last 2 bits (the "black" value will start at the 3rd stop because of the higher noise floor). It's tonal range will look like this (where 0 can be noisy)

0,0,8,16,32,64,128,256​,512,1024,2048,4096,81​92,16384

The Nikon doesn't exhibit noise because it's sensor has been able to fill the last tonal stop. If you're exposing a scene that's over 14 stops of light, THE NIKON WILL HAVE BLOWN HIGHLIGHTS (the value of white has exceeded the sensor's and recorded image
s range). If you underexpose, you're also reducing recorded tonal range!

THIS IS END OF STORY!!!

Completely off topic at this point, but hey, in some sense, I am being consistent to how the thread is panning out.

Many image libraries use HSL or RGB to assign pixel colors and intensities, we don't get into powers of 2.

For example, here is one of many imaging libraries for the PHP programmers out there.

http://php.net/manual/​en/ref.image.php

Use this to set a single pixel: http://php.net …unction.imagese​tpixel.php, and the color identifier you send it is an RGB set. This works for the GNU licensed imaging libraries out there, as well as ImageMagick, and a ton of others.

I think we have somebody that does alot of imaging manipulation, but they are using tools to do it, and not writing the code but act like they understand the code under the covers, thus why the strange terminology. I see this alot in my field when I hire programmers, and I can tell almost immediately based on Q&A that they don't have a degree in CS, but rather moved into programming when their other choices didn't pan out or they lost interest.

TeamSpeed wrote in post #17748072
I think we have somebody that does alot of imaging manipulation, but they are using tools to do it, and not writing the code but act like they understand the code under the covers, thus why the strange terminology. I see this alot in my field when I hire programmers, and I can tell almost immediately based on Q&A that they don't have a degree in CS, but rather moved into programming when their other choices didn't pan out or they lost interest.

I sure hope that isn't a back-handed way of trying to discredit anyone who is posting to this thread......it seems like it would be wrong to do that.

So possible attempt at suggesting bit depth isn't relative to sensor DR? This thread is never going to get into any range above the "noise floor" I would hope we'd be at least able to get into the black (white is never going to happen)!

You can basically sum up this entire thread real easily. Don't shoot in poor lighting. A skilled photographer will know how to properly expose or light his subject.

And that a Nikon will still have blown highlights after 14stops. Back to noise....

Just a thought Dave, the Sony records a 12bit raw file but has 13.9 stops of DR.

ADCs have been 14bit for awhile. DXO isn't pulling up on me, but I think Nikon D810 has a rated sensor DR of 14.6? Anyway, no single capture device I'm aware of can "record" more then 14stops at one exposure (Sony RAWs don't show noise in post because the set black value is above the noise floor).

TeamSpeed wrote in post #17747978
Some good info there, hopefully he understands it.

I tried describing this once using a concert as my analogy. Let's say you wanted to record the concert, down to the very quietest note. Now let's say that you have a crowd of people constantly whispering and humming, throughout the entire presentation. When you take your master recording, you will be dismayed to find out that all the whispering and humming by the people in the audience interferes with the quietest points of the concert, to the point that if you use sound filters to get rid of that, you also lose the actual concert notes.

Your professional recording equipment was able to capture much more "audio DR", however the people restricted your usable "audio DR". The electronics in the camera, and the way light falls onto the sensor depending on the optics and design of the sensor wells, depth of sensor wells, etc, are the people whispering and humming. Your sensor is statistically capable of giving you so much more but unfortunately there are other factors that hinder that.

I am often really bad at analogies, so this may not work.

(Don't let Dave hear this 'cause the audio analogy will be similarly dismissed. He will say that the last bit in the 16-bit audio file on a CD is pure silence! Any hiss from the microphone used to capture the analog sound before it was digitized is irrelevant. Oh, wait, I could then play him the CD file from my 1936 Robert Johnson studio recordings and perhaps he might finally understand the noise we are talking about. But then again, maybe not.)