Photo Algorithm Makes Reflections Disappear

"MIT Photo Algorithm Makes Window Reflections Disappear"

... Besides finding double reflections, the researchers' technique also involves looking for statistically anomalous pixels—those that don't seem to be a part of the overall image. The researchers fed an algorithm 50,000 images to teach it how to differentiate pixels that are meant to be there from pixels that aren't. When the "trained" algorithm divides an image into an 8x8 pixel square, it can reliably distinguish between reflection and overall image.



source: http://www.popularmech​anics.com …rithm-window-reflections/

Well I wasn't terribly impressed with the after shot in the linked article. There was still a lot of the reflection left over. Actually the "removed" bit actually seemed better to me, almost as if it were designed to isolate and enhance the reflection rather than the non reflected image.

Alan

I agree that the results are not great. I do find the “trained algorithm” idea interesting, though. And this technology will probably have uses beyond removing reflections.

The article at MIT News contains more information and includes the following quotation:

“The ideas here can progress into routine photography, if the algorithm is further robustified and becomes part of toolboxes used in digital photography.”

I hope I live long enough to enjoy those robustified algorithms.

I agree with Alan. Someone should be a little concerned about their research funding on this one

nqjudo wrote in post #17557002
I agree with Alan. Someone should be a little concerned about their research funding on this one

The theory is interesting, even if the current implementation is wonky, or perhaps even misdirected.

First of all this depends on the phase difference between the inner and outer reflections of the glass, but the image needs sufficiently high resolution for that difference to be detectible in the first place in a static image; I can't imagine the difference being more than a few pixels on a 20mp+ sensor and close up to the glass surface.

However, while deducing the origin of the reflection post-fact may be difficult, if they integrate this algorithm into a camera's software and have it dynamically track reflections as the camera is shifting in space, by the time the exposure is taken the camera might already have the reflection data figured out, possibly even as part of a double-exposure. This of course will only work on cameras that constantly read off the sensor, meaning point & shoots and some mirrorless systems, although I'm guessing these are the kinds of cameras most people tend to shoot through glass with in the first place.

Alternatively, the purpose of this algorithm may be more useful in other applications, like machine vision or something technical where reflections may be a problem. One thing I can think of is surveillance, since video cameras constantly read the sensor, they would be optimal candidates for reflection removal.

Kolor-Pikker wrote in post #17557180
Alternatively, the purpose of this algorithm may be more useful in other applications, like machine vision or something technical where reflections may be a problem. One thing I can think of is surveillance, since video cameras constantly read the sensor, they would be optimal candidates for reflection removal.

Even reading the initially linked article it seemed as if this was far more about the problems faced by machine vision systems, which often don't actually need to produce a good looking image but a reliable data stream to make decisions from. Even on that single example image, I am sure you would get a much more reliable edge detection on the outside image data once this algorithm had done it stuff, especially if you ran them on the RGB colour separations. Add in that the pulled off reflection image could also have the edge detect run on it, and compared to the results of the other version. Given that it seems to be able to detect the phase difference of the reflections on the two surfaces, it could even become possible to allow for the refractive effects of the glass that the system is looking through, possibly by pinging an IR laser pulse through it. You would then even be able to measure the thickness of the glass.

Alan

Kolor-Pikker wrote in post #17557180
The theory is interesting, even if the current implementation is wonky, or perhaps even misdirected.

First of all this depends on the phase difference between the inner and outer reflections of the glass, but the image needs sufficiently high resolution for that difference to be detectible in the first place in a static image; I can't imagine the difference being more than a few pixels on a 20mp+ sensor and close up to the glass surface.

However, while deducing the origin of the reflection post-fact may be difficult, if they integrate this algorithm into a camera's software and have it dynamically track reflections as the camera is shifting in space, by the time the exposure is taken the camera might already have the reflection data figured out, possibly even as part of a double-exposure. This of course will only work on cameras that constantly read off the sensor, meaning point & shoots and some mirrorless systems, although I'm guessing these are the kinds of cameras most people tend to shoot through glass with in the first place.

Alternatively, the purpose of this algorithm may be more useful in other applications, like machine vision or something technical where reflections may be a problem. One thing I can think of is surveillance, since video cameras constantly read the sensor, they would be optimal candidates for reflection removal.

LOL. Well yes, of course. One can conclude many interesting directions, possibilities and uses from a given research or result. I agree with your comments. Winky emoticons however are useful for denoting some kind of playfulness within the context of the discussion. I thought it might be somewhat more lighthearted than using several paragraphs to state the obvious.