MTF charts, anybody?

[PacAce]

Just curious. How many of you lens purchasers consult the MTF charts and how much of your purchase decision is based on it?

[CyberDyneSystems]

I have only ever really looked up the MTF charts AFTER I allready have the lens...

At which point I usually say to myself.. "yup,. I thought it was nice too" :) :) :)

Seriously,. I take word of mouth,. (of type?) and images I have seen as the largest influence... that and a trial when I have the chance.

[sugar_babygirli]

I voted the last choice. :P Please explain haha.

[Sean-Mcr]

Went for number 3 and i've yet to test my new lenses using the chart

[PacAce]

I voted the last choice. :P Please explain haha.
Here's a good starting point: :)

http://www.luminous-landscape.com/tutorials/understanding-series/understanding-mtf.shtml

[PacAce]

Went for number 3 and i've yet to test my new lenses using the chart
Ummm, it's not a chart for testing lenses. It's a result chart of a tested lens. :D
Actually, the MTF chart shows you in a graph form what the contrast and resolving power performance of the lens is at the widest aperture and another aperture like f/8 across the entire viewing angle of the lens.

[slin100]

I heard that the MTF charts posted on Canon's USA site are not accurate. Supposedly, the ones from the Japan website are more reliable. Nevertheless, I really don't know how to read Canon's MTF charts. They have so many plots that I don't know what each line represents. I know there are lines for horizontal/veritical MTF and diagonal MTF but their charts have 8 lines. :confused:

Even if I could read them, I don't think I would base my decisions on them. Another factor is that 3rd-party manufacturers don't supply these charts, so that would make it a bit hard to perform a comparison.

[PacAce]

Actually, the reason I brought this MTF subject up was because I was browsing through my "EF LEns Works III" book this evening and came across a couple of pages showing the MTF charts of all the currently available EF lenses which I found interesting, especially when comparing different lenses. I wanted to hear from others who are knowledgeable about MTF charts, what they thought of them. Are they truly representative of lens performance across a broad range of samples or do they only represent a lens that was hand picked just for the purpose of testing them and getting good results?

[CyberDyneSystems]

It's actually not to hard to read the "jist" of the charts..

Essentially the straighter the line and the higher it is on the chart.. the better.. All of the best lenses have charts which read all four lines almost straight across the top.. like the

-------------300mm f/2.8 IS ----------------------- Or the 400mm f/2.8L--------

http://www.usa.canon.com/html/eflenses/images/lenses/ef_300_28/ef_300_28mtf.gif http://consumer.usa.canon.com/app/images/lens/ef_400_28mtf.gif

A not so good lens looks like this;
http://consumer.usa.canon.com/app/images/lens/ef_28-200_35mtf2.gif


Of course there is more specifics you can gain be learning what each of the lines means,. but as I say,. you tell at a glance that the two primes simply rock!

[PacAce]

I heard that the MTF charts posted on Canon's USA site are not accurate. Supposedly, the ones from the Japan website are more reliable. Nevertheless, I really don't know how to read Canon's MTF charts. They have so many plots that I don't know what each line represents. I know there are lines for horizontal/veritical MTF and diagonal MTF but their charts have 8 lines. :confused:

Even if I could read them, I don't think I would base my decisions on them. Another factor is that 3rd-party manufacturers don't supply these charts, so that would make it a bit hard to perform a comparison.
Steven, I had a hard time with the MTF charts myself. No matter how many times I read through the brief explanations I still had a hard time understanding what the graphs were telling me. Then it became crystal clear when I read a more detail explanation in the "EF Lens Work III" book published by Canon. I guess it was more like I had different pieces to a puzzle but they didn't start to come together until I read this book.

[PacAce]

It's actually not to hard to read the "jist" of the charts..

Essentially the straighter the line and the higher it is on the chart.. the better.. All of the best lenses have charts which read all four lines almost straight across the top.. like the

-------------300mm f/2.8 IS ----------------------- Or the 400mm f/2.8L--------

http://www.usa.canon.com/html/eflenses/images/lenses/ef_300_28/ef_300_28mtf.gif http://consumer.usa.canon.com/app/images/lens/ef_400_28mtf.gif

A not so good lens looks like this;
http://consumer.usa.canon.com/app/images/lens/ef_28-200_35mtf2.gif
It doesn't necessarily have to be a straight line as long as the line are at or above 0.8 to be considered very good. If they're at least above 0.6, then they're satisfactory. Although a straight line does denote consistent performance across the entire frame.

For 1.6x cameras (20D, 10D, etc.) the lines should stay at or above 0.8 up until the 13 point on the horizontal axis. For 1.3x cameras, it should maintain that until the 17 point mark on the horizontal axis. If they don't, that means that you'll see a degradation in lens performance towards the outer edges or corners of the frame. For anything other than a FF camera, where the graph goes to the right of the point mentioned above isn't really that important since they fall outside the bounds of the sensor.

And this I picked up tonight. The thicker lines (the 10l/mm in the Canon MTf charts) represents lens contrast and the thinner lines (the 30l/mm) represents the resolving power of the lens. And last, but not least, the closer the solid lines and the dashed lines are to each other the more natural looking the blurred background becomes. :)

[cc10d]

I like using the MTF charts for comparing lenses. My dissapointment is that Cannon does not seem to be continuing to make charts available for the more recent lenses. EFS types especially, so now I rely more on 3rd party tests. Some of which include MTF charts or scores. But they do not necesarilly correlate direclty with the Canon charts due to differences in testing. However one can use them when comparing reports from the same test lab. General tendencies will correlate accross different test labs, but the numerical values are not as transferable in an absolute sense.

[slin100]

Ok, I read Canon's description of MTF in their glossary. There are 8 lines in their MTF charts because they show results at two apertures (wide-open and f/8 ). Each aperture displays MTF using four sets of lines. Two sets are spaced apart 10 lines per millimeter parallel to a 45 degree diagonal, and the other two sets are spaced 30 lines per millimeter perpendicular to the diagonal.

It doesn't necessarily have to be a straight line as long as the line are at or above 0.8 to be considered very good. If they're at least above 0.6, then they're satisfactory. Although a straight line does denote consistent performance across the entire frame.

But which lines should be above 0.6? If you look at the MTF chart for the 85/1.2L, the thin black lines are well under 0.6.
http://consumer.usa.canon.com/app/images/lens/ef_85_12mtf.gif
Compared to the MTF chart for the 300/2.8 and 400/2.8, the 85/1.2 looks positively dreadful. But we all know how legendary the 85/1.2 is. So, doesn't this show that MTF isn't all that it's cut out to be? Canon even points out that MTF charts don't reveal the following lens characteristics:
While MTF charts don’t include many factors that can be important when selecting a lens (size, cost, handling, closest focusing distances, AF speed, linear distortion, evenness of illumination, and of course features like Image Stabilization which may produce superior real-world results), they can indicate to the knowledgeable reviewer some of the optical characteristics they can expect from a particular lens.

And this I picked up tonight. The thicker lines (the 10l/mm in the Canon MTf charts) represents lens contrast and the thinner lines (the 30l/mm) represents the resolving power of the lens. And last, but not least, the closer the solid lines and the dashed lines are to each other the more natural looking the blurred background becomes.
Once again, compared to the 300/2.8 the solid and dashed lines for the 85/1.2 don't look that close, but the 85's buttery-smooth bokeh is unmatched.

[Tom W]

Well, there's a bit more to study with the charts, but here's some basic stuff.

All the blue lines are taken at f/8.
All the black lines are taken at maximum aperture.

Consider what maximum aperture is when comparing black lines, as an f/1.2 lens is not generally going to resolve wide open as well as an f/4 lens, if both are quality lenses. Always consider lens speed with looking at the wide-open MTF curves.

The fat lines are measured at 10 lines/mm and indicate the lenses' contrast capabilities.
The skinny lines are measured at 30 lines/mm and indicate the lenses' resolving (sharpness) capabilities.

The horizontal axis of the chart indicates the distance from the center line of the lens. The farther right, the farther from center. The scale in mm is along the bottom of the chart.

On the verticle scale is a scale of 0 to 1, with 1 being the perfect fidelity, and 0 being no image at all. The closer to 1 the readings are, the better the image fidelity. Quality at f/8 is usually much higher than at maximum aperture.

The solid lines are measurements made in the Sagittal plane (radial from the center).
The dotted lines are measurements made in the meridional plane (circular, perpendicular to the radial lines being measured).

Generally, the closer the dotted and solid lines remain together, the smoother the bokeh.

[CyberDyneSystems]

fa! 85mm f/1.2L Pooo.. look at that chart,.

Now here's a chart! (500mm f/4L IS) :mrgreen:
http://www.usa.canon.com/html/eflenses/images/lenses/ef_500_4/ef_500_4mtf.gif

[Tom W]

Ok, I read Canon's description of MTF in their glossary. There are 8 lines in their MTF charts because they show results at two apertures (wide-open and f/8 ). Each aperture displays MTF using four sets of lines. Two sets are spaced apart 10 lines per millimeter parallel to a 45 degree diagonal, and the other two sets are spaced 30 lines per millimeter perpendicular to the diagonal.

They measure all those things, but not exactly in the manner you note. Each pair of liens (dotted and solid) represents a diagonal (actually, sagittal) and perpendicular. Both directions are measured at 10 and 30 l/mm, at both f/8 and wide open.


But which lines should be above 0.6? If you look at the MTF chart for the 85/1.2L, the thin black lines are well under 0.6.
http://consumer.usa.canon.com/app/images/lens/ef_85_12mtf.gif
Compared to the MTF chart for the 300/2.8 and 400/2.8, the 85/1.2 looks positively dreadful. But we all know how legendary the 85/1.2 is. So, doesn't this show that MTF isn't all that it's cut out to be?

A couple of things - first, the black lines are measuring the lens wide open. You're measuring an f/1.2 lens against an f/2.8 lens. There's no comparison, wide open. Also note that MTF lines are best used to compare lenses of similar focal length. You can't compare a 14 mm lens MTF to that of a 300 mm lens without some resounding differences showing up. It becomes blindingly obvious that the telephoto design is superior in terms of sharpness and contrast compared to a wide angle lens, which requires a great deal of correction to maintain something close to a rectilinear image.

Once again, compared to the 300/2.8 the solid and dashed lines for the 85/1.2 don't look that close, but the 85's buttery-smooth bokeh is unmatched.

Again, don't forget that the 85 gives you f/1.2. You're bound to get much more blur out of a wider aperture than from a smaller one. Also note that bokeh smoothness is related not only to equal meridiontal and sagittal performance, but also to lens softness. The 85/1.2 gets soft away from center wide open, and that does contribute to bokeh.

In other words, the lines in the MTF chart are only part of the story.

[CyberDyneSystems]

The thin lines on the chart are measuring at 30 lines/mm and indicate the lenses' resolving power for very fine details.. only the super large "dinner plate" sized optics of the super telephotos can push the resolving power as high as the thick lines...

...for magnifications under 200mm those thin lines simply don't need to be so high on the chart,. as the lens is not trying to pull in detail from like the super telephotos do from hundreds of feet away,.. And the wider lenses can't be built wiht the resolving power so high really.. maybe with a lot more glass they could, but I doubt it.

Lastly,. the difference you see in the 300mm f/2.8's chart and the 85mm's chart is very indicative of image quality. You can have all the finest L glass you want at 100mm and under and think you've seen it all,. but the first time you take some pics with one of these $5K telephotos you'll see stuff you've never seen before. The clarity, contrast and detail are just beyond comparison.

[Redbird_xo]

[QUOTE=CyberDyneSystems]...but the first time you take some pics with one of these $5K telephotos you'll see stuff you've never seen before...[QUOTE]

:D ...such as...

1. Spend a night in the dog house watching the empty ceiling
2. Watching Saturday Night Live at home alone while buddies are out drinking at a bar because the weekend pass has been revoked by the wife
3. Other "side" of the wife as if the wife had just gone through an extreme makeover (of course, not a successful one!)

Just some of the possibilities that I can think of...I could be missing others...:D

[pierrot]

MTF charts are nice piece of information on a purely technical basis : brutal, dry and boring.
They will describe the performance of a lens used in a lab and evaluated with computers and other specific test gear.

They won't describe the look and feel of the lens, and its ability to transmit real, vibrant colours for example.

As far as I am not involved in photograpying Al Qaida's car plates from a spy satellite 20,000 miles high, I will eventually have a look on those charts but will much more rely on actual shots made in the real life ;)

[ron chappel]

I used to look at MTF charts alot but have now given up. They simply don't tell the truth!
Canon has the 75-300 model lenses outperforming the 100-300usm lens at the long end....but testing and experience show them to be the other way round.

Similarly,the EFs 18-55 lens has pretty good MTF numbers...but falls flat on it's face in day to day use.

On the other hand the EF300/2.8 is lens has the best looking numbers i've ever seen....and so far i've seen no reason to doubt them! What a lens!!!


As for the canon MTF charts (and their accuracy) -I seem to remember that canon use theoretical charts that don't take into account the inevitable effects of diffraction.
While this is not ideal, it is still OK to use them for comparing one canon lens against another ...or would be if they actually meant much in the first place:rolleyes: :rolleyes:

These days i go by others comments or online tests

[PhotosGuy]

I seem to remember that canon use theoretical charts that don't take into account the inevitable effects of diffraction. I wish they'd adopt my theoretical MSLP too! (Manufacturer's Suggested List Price) Makes as much sense to me as the "Self Policing" policies our gov has for meat packers & chemical plants!
I like to see pics, but more important, I want to hear what other people I "know" are experiencing with the lens. ;-)

[Tom W]

I used to look at MTF charts alot but have now given up. They simply don't tell the truth!
Canon has the 75-300 model lenses outperforming the 100-300usm lens at the long end....but testing and experience show them to be the other way round.

Similarly,the EFs 18-55 lens has pretty good MTF numbers...but falls flat on it's face in day to day use.

On the other hand the EF300/2.8 is lens has the best looking numbers i've ever seen....and so far i've seen no reason to doubt them! What a lens!!!


As for the canon MTF charts (and their accuracy) -I seem to remember that canon use theoretical charts that don't take into account the inevitable effects of diffraction.
While this is not ideal, it is still OK to use them for comparing one canon lens against another ...or would be if they actually meant much in the first place:rolleyes: :rolleyes:

These days i go by others comments or online tests

There's a couple of problems with the MTF - they're very generic. Canon doesn't test individual lenses for MTF. The charts, whether theoretical or whatever you might call them, are more of a "typical" performance indicator. Idividual copies of each lens will vary in the +/- direction.

Also note that there are subjective vs. objective measurements. People who pick up their first prime lens are likely to be awed by its performance if all they had before was one of the 28-80 kit lenses that came with their original Rebel. Yet another afficianado might pick up the same prime and find its output horrible compared to the Leica lenses or Canon 35/1.4 he's been shooting for 10 years.

The theory is that the MTF chart would avoid the subjective and concentrate on objective, comparable, measureable results. Unfortunately, if the individual lens isn't tested (as Photodo used to do), you're not really looking at anything but a ballpark indication of performance.

[slin100]

...for magnifications under 200mm those thin lines simply don't need to be so high on the chart,. as the lens is not trying to pull in detail from like the super telephotos do from hundreds of feet away,.. And the wider lenses can't be built wiht the resolving power so high really.. maybe with a lot more glass they could, but I doubt it.

I don't follow this line of reasoning. Resolution tests like MTF are designed to factor out focal length. In the old days, you had a chart and you were supposed to fill the entire frame, no more or less. Doing this factors out the focal length.

Besides, why wouldn't I want my non-supertelephoto lenses to have high resolving power? If anything, it should be an easier job for a shorter focal length because it doesn't have to pull in detail from so far away. What's so magical about 200mm?

Lastly,. the difference you see in the 300mm f/2.8's chart and the 85mm's chart is very indicative of image quality. You can have all the finest L glass you want at 100mm and under and think you've seen it all,. but the first time you take some pics with one of these $5K telephotos you'll see stuff you've never seen before. The clarity, contrast and detail are just beyond comparison.
I don't disagree but a resolution test chart cares not a whit about focal length. If the viewer can't see what lens is mounted when looking through the viewfinder, then it's a level playing field.

I think Tom W made a good point about the black lines. They represent wide-open apertures. At f/1.2 it's probably really hard to design a lens to have high resolving power.

[CyberDyneSystems]

If anything, it should be an easier job for a shorter focal length because it doesn't have to pull in detail from so far away.

This is really the same point I was trying to make,. perhaps poorly.

Since the SuperTelephotos have to pull in the detail from so far away,. since this is there very job, the design is such that it simply out resolves other lenses. They have to.

Nothing really Magic about 200mm by the way other than if you look at the charts for the primes,. it is at about 200mm where the MTF charts go right to the top of the charts.. with few exceptions. Supertelephotos of any manufacture allways are at the top of the charts,. and using them gives you every reason to understand why they test so well.

The other correlation I draw from these charts and experiences is glass diameter = higher resolution. This should make perfect sense. larger glass means more glass area for higher res..

Compare the 400mm f/5.6 Vs. the 400mm f/2.8.. do the same with the 200mm f/2.8 Vs. the 1.8. Whats different? Way way more glass is needed to get that larger aperture, the by product is more resolution.

Again,. it's only the fast super telephotos that have such fat glass. If you had one of those naval brass telescope sized telephotos they sell at Ritz for $200.00 at like 800mm but at f/16 or f/22... the MTF would be piss-poor. It's little silver dollar sized lens elements certainly can not have the resolving power of the frisbee sized elements in the 600mm f/4.

Lastly,. I'm not at all saying I know why this is the case based on any scientific papers I have read or whatever... just that this has been my observation of both real world use and reading the MTF charts. These are the corellations that I have seen and drew my own conclusions.

[Tom W]

I don't follow this line of reasoning. Resolution tests like MTF are designed to factor out focal length. In the old days, you had a chart and you were supposed to fill the entire frame, no more or less. Doing this factors out the focal length.

Besides, why wouldn't I want my non-supertelephoto lenses to have high resolving power? If anything, it should be an easier job for a shorter focal length because it doesn't have to pull in detail from so far away. What's so magical about 200mm?

Lens design at longer focal lengths is much easier in some respects. As the lens gets wider, the amount of correction becomes increasingly necessary. Light is coming in at steeper angles and must be redirected and projected on the sensor/film. And it must be done in such a way that distortion is minimized (distortion is one aspect that the MTF does not measure). Otherwise, the wider the lens, the more fisheye it would become.

But its not just correction for fisheye - the lens must also correct for relative distance from the lens to the subject. Let's take a big rectangular building, 36 meters wide by 24 meters tall. Take a 600 mm lens and stand back far enough to frame the entire building. Now measure the distance from the lens to the middle and each corner of the building. The distance will be almost the same. So, the longer lens doesn't need much planar correction.

Now take a 20 mm wide-angle lens and position yourself to frame the same building. You'll note that the distance from the lens to the center of the building differs greatly from the distance from the lens to each corner. The wider lens has to correct for this or else your plane of focus will be curved - the middle of the building where you focused will be sharp while the corners will fall away from that plane and will be out of focus.

This phenomina is why wide-angle lenses perform so drastically different from telephoto lenses. Enter a zoom into the equation and its a wonder the things even work at all!

I don't disagree but a resolution test chart cares not a whit about focal length. If the viewer can't see what lens is mounted when looking through the viewfinder, then it's a level playing field.

The field's level, but the players have different roles (if that makes any sense). A wide angle lens is simply more difficult to make well than is a telephoto. The curves do show improvement as the lens gets longer, at least until the 1200 mm which has its own design issues.

I think Tom W made a good point about the black lines. They represent wide-open apertures. At f/1.2 it's probably really hard to design a lens to have high resolving power.

Yes, its important to realize what the lens' max aperture is when looking at the black lines. I think that's why Canon chose to also include f/8.

[rdenney]

As for the canon MTF charts (and their accuracy) -I seem to remember that canon use theoretical charts that don't take into account the inevitable effects of diffraction.

I can't imagine diffraction would have much effect even at f/8, and certainly not wide open. At a given aperture setting, however, it will have less effect with long lenses than with short lenses.

But I have also heard that Canon uses design charts rather than test charts. They take their lens design and let the computer calculate MTF, rather than taking samples of actual lenses and measuring their performance. Thus, the charts published by Canon don't consider variations in glass, in construction, and in mechanical tolerances.

Rick "who considers Canon's charts a statement of what their designers intended" Denney

[Longwatcher]

I rather like the MTF charts, they seem to work fairly well for me at least. They are not the only thing I use, but they help me make up my mind between say the 85/1.2 and 85/1.8.

The first thing I pay attention to is the blue lines if you compare the 85/1.2 to the 85/1.8, the solid blue lines are slightly better on the 85/1.2. more importantly the hold the line better to the edges then the 85/1.8. Not a problem for the 1.6x and 1.3x folks, but huge difference for FF folks.

Then I look at the Black lines, taking into account the 85/1.2 is a f1.2 and thus at least 50% if not 100% harder to design. It has pretty good heavy black line, but the thin line does look pretty bad, except that both of them are a much gentlier fall off then the 85/1.8; which should make for a better image overall.

One other interesting thing is that the 85/1.2 falls off fairly consistently, where as the 85/1.8 actually increases in resolution as you go towards the outside of the lens, until a sudden drop off at the edges. I don't know about you, but I prefer to have my subject be the focus of attention and the background be just that. background and blurry.

Thus comparing these two lenses using MTF charts helps me decide that since I have a FF camera I should go for the 85/1.2. Had I only a 1.3 or 1.6 then the 85/1.8 would have been more acceptable, but I still would have preferred the 85/1.2.

Lastly in comparing both of those to the 135/2. The 135/2 dusts those two lenses based on the MTF chart. This is where common sense comes in, the 135/2 is too darn long for my studio. Although I still want it. So 85/1.2 first, 135/2 second and then maybe 85/1.8 because the one real world grip with 85/1.2 is auto focus speed is real slow and I can see this as a problem in some situations, just not in my studio doing portraits.

So use the MTF charts wisely, but don't use them as your only decision source.

Just my opinion,

PS: did I mention that I am a former image analyst who works with image scientists on a frequent basis so that I can actually understand some of this stuff and what it means, but even I had to look up what color was what, I could just figure out what it all means when I read about it.

[ed2day]

I like looking at MTF charts, I guess it's my technical (geek) background. Just to reiterate a couple things that were mentioned: The charts are theoretical, probably derived from a computer model. They don't even attempt to portray manufacturing variability. As such, I look at them as sort of a ceiling showing how good a lens can be. It can be worse, but not likely to be better. Also you can't compare lens of different focal lengths. And they certainly don't tell everything about a lens. They show resolution/contrast but only at a couple different resolutions and a couple different apertures. And even if those holes were filled in it would still just be a partial picture of sharpness--just one aspect of a good lens(for instance it doesnt address a lens ability to resolve small tonal gradations). So it's useful info, but far from the whole story.

[slin100]

Earlier I said that the MTF charts on Canon's U.S. site were reputed to be less accurate or downright wrong. I finally found Canon's Japanese website. The MTF chart for the 85/1.2L is the same but the ones for the 17-40/4L are completely different and much better, I might add, at 40mm!

The Japanese website is here: http://cweb.canon.jp/ef

I've also found some additional considerations.




MTF is usually determined for a lens focused at infinity. The MTF at closer distances may be different. This may be especially important to keep in mind for macro lenses.
MTF at apertures smaller than f/8 may also be quite different when effects such as diffraction come into play. This may be important for landscape or macro photographers where stopped-down performance is more critical than performance wide-open or even at f/8.
A poor MTF response near the edge of the frame can be misleading because field curvature is often the cause and not poor resolving power. MTF is based on a flat target where field curvature can play a big factor. In the real world, field curvature may not be as important in certain applications.
MTF says nothing about color response, such as warmth or coolness, or the presence of chromatic aberrations, nor does it indicate resistance to flare.

[ed2day]

I've always wondered how far the target is when generating the charts and does it change with focal length? If they focus at infinity it's obviously far away. Which brings up another thing the charts don't show--how well does it focus?

[Sean-Mcr]

Ummm, it's not a chart for testing lenses. It's a result chart of a tested lens. :D
Actually, the MTF chart shows you in a graph form what the contrast and resolving power performance of the lens is at the widest aperture and another aperture like f/8 across the entire viewing angle of the lens.


I'm sorry i meant i never looked at the mtf charts before i bought them and i've still not:o

[DavidEB]

well, I'm not an expert in optics and I don't fully understand MTF charts. Canon states their charts are theoretical (that is, computer generated based on the lens design) rather than real-world tested. So common sense says they don't account for factors that aren't programmed into the computer (like chromatic aberration, flare resistance, etc...). Color portrayal and chromatic abberation are features of how the MTF charts might change at different light wavelengths, and the Canon charts don't show that.

That said, actual comparison tests also have limitations. To be truly useful a real-world test should use several sample lenses and average the results, to overcome variance in quality. Also, testing conditions would have to be controlled well enough so that test-induced variance is less than the smallest lens defect you wish to detect. This is especially critical in several areas. For testing edge performance -- if the subject is not exactly perpendicular to the line of sight, a real-world test might be worthless. Vibration also impacts lens tests -- I've seen many posted comparison photos done in low light indoors conditions (since outdoors reduces control of other aspects of the test such as warm air currents). But I've almost never seen a statement about mirror lockup, tripod, cable release, etc... Finally there's an issue called assay sensitivity -- some posted lens tests seem to dwell on differences between lenses that are unlikely to be apparent in any real-world situation (such as edge detail in 100% crops). The content of the test should be meaningful.

Even relying on practical experience in real-world use has its problems. Uers of mid-range lenses often rave about them, particularly if they've just stepped up from the kit lens. I am very pleased with my Sigma 70-200mm f2.8 even though "L" purists don't seem to like it. You have to consider the source. Would you trust my recommendation? You also have to consider if your needs match the uses of the reviewer.

All said, it seems to me a good idea to consider each type of information in the context of what it's worth and what it's able to tell you, rather than relying on a single factor. When all data concur the decision is easy.