Is there a way to do a dark frame subtraction?

I was wondering if there was any way to do a dark frame subtraction at faster shutter speeds? I am referring to long exposure noise reduction.If not, I think we should bug Canon until they give a firmware update or implement it in future bodies. What do you think?

I'd say the impact of such an extra task is too small to be worth it.

I think that you are worrying about something that does not warrant worrying.

apersson850 wrote in post #10692470
I'd say the impact of such an extra task is too small to be worth it.

... and, I agree 100%. It would just be wasted processing with no observable benefit. If you want to see what a dark frame looks like, set the shutter to the desired time and the desired ISO (just keep in mind that high ISO NR is a completely different issue) keep the lens cap on and go into a dark closet and then fire away. Look at the dark frames in Photoshop and process them exactly the same as the properly exposed light frames. You will see that the dark frames are just that ... dark, with very little noise. You can go one step further and use Photoshop to stack the two frames and then take the difference between them. For a normal low ISO exposure, you will see no improvement in the image after doing this.

The type of noise encountered in low ISO long exposures is fixed pattern noise which is mainly a function of ambient temperature and length of exposure, but the duration needs to be long enough to allow the noise to become noticeably greater than the random noise floor.

Cameras put out in the past three years of so have long exposure noise reduction -- they take a second exposure with the same settings but with the sensor blocked out then do a dark frame subtraction.

apersson850 wrote in post #10692470
I'd say the impact of such an extra task is too small to be worth it.

Normally I'd agree.

But if he has a 7D that's experiencing vertical banding in the shadows or a 5Dmk2 that's experiencing maze pattern noise then it could easily be worth it.

That's not the way to combat the different readout levels from different columns on the sensor in the 7D (the main cause of the banding). The variation will depend also on light level, so a dark frame will not provide the proper compensation information for a normal picture.

Canon applies mathematical compensation in the camera, and in DPP, to combat this problem. The problem's complexity is probably increased by the fact that different 7D camers have different levels of inconsistency between sensor columns, so one particular level of fix doesn't cover more than one particular camera. They have to be more intelligent than that.

Bill Boehme wrote in post #10692704
I think that you are worrying about something that does not warrant worrying.

... and, I agree 100%. It would just be wasted processing with no observable benefit. If you want to see what a dark frame looks like, set the shutter to the desired time and the desired ISO (just keep in mind that high ISO NR is a completely different issue) keep the lens cap on and go into a dark closet and then fire away. Look at the dark frames in Photoshop and process them exactly the same as the properly exposed light frames. You will see that the dark frames are just that ... dark, with very little noise. You can go one step further and use Photoshop to stack the two frames and then take the difference between them. For a normal low ISO exposure, you will see no improvement in the image after doing this.

The type of noise encountered in low ISO long exposures is fixed pattern noise which is mainly a function of ambient temperature and length of exposure, but the duration needs to be long enough to allow the noise to become noticeably greater than the random noise floor.

As long as it would offer to no benefit, i'll leave it alone. I juus didnt want to be missing something we could benefit from. Thanks! i can rest easy now.

apersson850 wrote in post #10693095
That's not the way to combat the different readout levels from different columns on the sensor in the 7D (the main cause of the banding). The variation will depend also on light level, so a dark frame will not provide the proper compensation information for a normal picture.

Hmm...but from what I've read elsewhere on the issue (e.g., in dpreview.com threads such as the one here), a dark frame subtraction actually worked for removing most of the banding, although the removal wasn't complete -- the resulting image retained a lower frequency periodic component, but said component was also of significantly lower amplitude.

While not perfect, it's a significant improvement...

OK, then it is of some value there after all. To me, it seems smarter to do the math afterwards instead of taking yet another exposure, be it dark, since the dark exposure also consumes buffer memory, but math takes less. Canon does that already today, but those shooting RAW and then convert in other software than Canon's don't get any benefit from it.

apersson850 wrote in post #10699598
OK, then it is of some value there after all. To me, it seems smarter to do the math afterwards instead of taking yet another exposure, be it dark, since the dark exposure also consumes buffer memory, but math takes less. Canon does that already today, but those shooting RAW and then convert in other software than Canon's don't get any benefit from it.

Canon's software solution is incomplete at best. Even with the latest version of DPP, I can "fix" things so that I easily see the banding in the shadows, and it takes a lot less than you might think.

I have not yet shot with a 7D for which that isn't the case.

I think it's pretty clear that either their software people don't really understand the problem, or they don't have sufficient knowledge (which may include the necessary line adjustment values -- that could easily be a per-camera thing that they simply cannot know or deduce) or expertise to fix the problem.

Someone remind me why I should want a 7D again??

Would a black frame subtraction improve the situation, then?

The 7D seems to have a thinner line of masked pixels outside the sensor than previous cameras. Before, 30 or so was more or less the rule. I think the 7D has 8. This line is used to set the black point, and could also be used to compare the level of different columns. It's like a black frame, just not over the whole area.

Softwarewise, it works pretty well to read about 30 pixels high across the columns, average these 30 and use the values computed to compensate for different column levels, which cause a great deal of this noise, when the RAW conversion is done. Different RAW converters do have procedures to compensate for different levels in different columns, so they are prepared for that. The main problem with the 7D is that the difference is so large from one camera to another, that the software vendors can't have a fixed setup for a certain model, but must adapt for each individual camera. They aren't prepared for that.

apersson850 wrote in post #10699983
Softwarewise, it works pretty well to read about 30 pixels high across the columns, average these 30 and use the values computed to compensate for different column levels, which cause a great deal of this noise, when the RAW conversion is done. Different RAW converters do have procedures to compensate for different levels in different columns, so they are prepared for that. The main problem with the 7D is that the difference is so large from one camera to another, that the software vendors can't have a fixed setup for a certain model, but must adapt for each individual camera. They aren't prepared for that.

The real problem is that even the camera's internal dark frame subtraction doesn't take care of the banding (when you do long exposures). This means the banding in question is added after that internal subtraction process. It's really weird.

So the only effective way to deal with this would be an external dark frame subtraction during postprocessing. And very few raw converters, and none of the mainstream ones that I'm aware of, are capable of that.

tonylong wrote in post #10692911
Cameras put out in the past three years of so have long exposure noise reduction -- they take a second exposure with the same settings but with the sensor blocked out then do a dark frame subtraction.

yep. It's helpful for long exposures (15 seconds or more is where I like to use it). But canon does a piss poor implementation, you actually have to wait for it to take the second shot after you took the first. So imagine a 15 minute star trail shot. Shoot for 15 minutes, then the camera shoots for ANOTHER 15 minutes while you wait.

that's the way it's always been done, but somehow pentax sorted out how to shoot their dark frame while the camera is actively shooting another shot. At the end it won't let you turn the camera off if it's still taking a dark frame (it puts a message on the screen). If your battery dies you lose that one dark from which is no big loss.

Waiting for a second long exposure SUCKS though.

apersson850 wrote in post #10699983
Would a black frame subtraction improve the situation, then?

The 7D seems to have a thinner line of masked pixels outside the sensor than previous cameras. Before, 30 or so was more or less the rule. I think the 7D has 8. This line is used to set the black point, and could also be used to compare the level of different columns. It's like a black frame, just not over the whole area.

Softwarewise, it works pretty well to read about 30 pixels high across the columns, average these 30 and use the values computed to compensate for different column levels, which cause a great deal of this noise, when the RAW conversion is done. Different RAW converters do have procedures to compensate for different levels in different columns, so they are prepared for that. The main problem with the 7D is that the difference is so large from one camera to another, that the software vendors can't have a fixed setup for a certain model, but must adapt for each individual camera. They aren't prepared for that.

interesting, I never knew what they hell the extra pixels were for, thanks!

As a generalization, astro photographers are just about the only ones needing dark frame subtraction for really long exposures. When they do dark frame subtraction, it is not done in the camera, but as a software operation after collecting many exposures that are combined into a "stack". Also, it is not absolutely necessary to create the dark frame immediately after the "light" exposure since we are dealing with "noise" that is repeatable (i.e., "fixed pattern" noise). The most important thing is to make the "darks" with the same exposure time as the lights. It also helps to have the camera at the same ambient temperature as when making the original images.

jacobsen1 wrote in post #10700555
.... But canon does a piss poor implementation, you actually have to wait for it to take the second shot after you took the first. So imagine a 15 minute star trail shot. Shoot for 15 minutes, then the camera shoots for ANOTHER 15 minutes while you wait.

I would not be so hard on Canon about this since I do not know of another NR implementation that can be substituted for this with equal results if you really want to do it in the camera. You could always do it the way that the astro photographers do it, but you can't get around the long dark exposure time. If there were a better way, I am sure that folks would be doing it. If your long exposures are always precisely the same length of time and at approximately the same temperature, then you could create a "dark frame" library.