dxo mark reliability

http://www.dxomark.com …nd2)/Tamron/(ca​mera2)/836

i dont understand why the tamron is better than the canon. can i trust the score?

ICarumba wrote in post #16270112
http://www.dxomark.com …nd2)/Tamron/(ca​mera2)/836

i dont understand why the tamron is better than the canon. can i trust the score?

'

In my opinion: yes. Some might disagree with me here, but DXOmark evaluates only optical perfomance. And it might be one of the few companies which do that, a standard unbiased test. Which is exactly what we need in this industry.

Canon looses quite a bit because of it's poor light transmission (0.5 T-stops worse than the Tamron). For more info, check here:
http://www.dxomark.com …tocols/Light-transmission

However, you should remember there are other factors which DXOmark does not measure. And one should consider other factors as well when making your purchasing decision on a 70-200 zoom lens. The Canon is sharper at 185-200mm (slightly), has slightly more effective IS, and is generally considered to have faster focusing speed (especially in dim light). But Tamron has also it's strenghts. For one, it's cheaper. It also draws much less attention (black color), and has in my honest opinion slightly creamier bokeh (9 rounded aperture blades compared to Canon's 8?). To sum up: Tamron SP 70-200mm F/2.8 Di VC USD is an excelent lens and one of the most underrated zoom lenses on these forums. It's sharpness is also very close to the Canon. On the actual images, it is likely you are not going to notice any difference with these two lenses.

ICarumba wrote in post #16270112
http://www.dxomark.com …nd2)/Tamron/(ca​mera2)/836

i dont understand why the tamron is better than the canon. can i trust the score?

It makes you scratch your head, doesn't it?

Look at the details:

Sharpness: same
Transmission: the Canon offers slightly less (1/2 stop)
Distortion: same
Vignetting: the Canon has slightly less
Chromatic aberration: the Canon delivers less than half as much

Add it all up and the Tammy ranks higher? Sounds like "new math" to me when the Canon "wins" in two categories and "ties" in two, is only "worse" in terms of transmission... 1/2 stop (while 1/3 stop is considered "unnoticeable"). If their test is true and accurate, any difference in "transmission" also would be completely and easily correctible by using a slightly slower shutter speed or slightly higher ISO. Heck, if using any form of auto exposure it would be automatically corrected. Besides, for some reason other testers (who are using DxOMark, BTW) don't seem to see this difference.

I thought maybe they also factor in price, in which case the Canon is substantially more expensive (list price is about 150% the price of the Tammy's) and that certainly would tend to skew the final score. However, according to their description, they don't consider price.

You should also consider that they most likely test exactly one copy of each lens. Though today's lenses usually have fairly consistent quality, there is still some variability. Maybe they happened to get and be comparing a particularly good Tammy and a somewhat weak copy of the Canon... or vice versa. Who knows? And was the test lens bought independently, or provided by the manufacturer or distributor for testing, in which case they might provide a particularly "good copy". By rights, testing like this really should be done with three or more randomly chosen copies of any particular lens, with the results averaged. But few testers actually do that.

Besides, test results only tell you so much. Yes, I look for and consider them when thinking of buying a lens. But real world use may show up strengths or weaknesses no lab testing ever will. And their interpretation of the "facts" might or might not be valid for me. For example, for my purposes and looking at the above, I'd be more concerned about the Tamron exhibiting more than double the CA of the Canon than the difference in "transmission". But, further checking tells me both lenses show relatively low CA.

No lens is "perfect"... But a certain lens may be a perfect fit for your particular needs... or not. If at all possible, try before you buy.

EDITS: This review of the Tamron is quite scathing, in a British-reserve sort of way... Real world usage that reports two different copies of the lens showed signs of possibly decentered elements: strong haloing at 200mm, f2.8; and perhaps most significantly from my point of view, slow to respond stabilization. It earns a "not recommended" from them. But there is no comparison with Canon (I think the reviewer is a Nikon shooter) and I'd still take with a grain of salt.... Were the two lense from very early on in production? Have problems been addressed and ironed out now? Was the reviewer just having a bad day? Or, was the reviewer an employee of a competitor? (I.e., I believe about 25% of what I see and read on the Internet... but the really hard part is deciding which 25% to trust!)

But other reviewers seem to rank the Tammy and the Canon very close in most respects. I have always particularly liked Popular Photography Magazines mix of lab and real world testing: Tammy 70-200mm VC vs Canon 70-200mm Mark II. By the way, they (and many others) use DxO Mark lens testing method, too. Yet they manage to get different sharpness results (the Canon is sharper in their testing... across the board, but especially at 200mm).

Another really good place for "real world" comparisons is Lensrentals.com. They are among the first to get hold of most important models, handle and deal with multiple copies of each lenses, and work with them over a longer period of time so are often able to provide helpful and useful commentary that's simply not available from other reviewers who only see one copy of a particular lens, and sometimes just as a brief loaner that was hand picked by the manufacturer. Lensrentals take on: Canon 70-200 Mark II, vs the Tammy 70-200mm VC. I find it interesting they charge more to rent the Tamron than the Canon!

There can be significant differences from reviewers, too. While DxO ranks the two lenses same in sharpness, but Pop Photo ranks the Canon superior (though not as good as the Nikon VR II). Lensrentals also gives the nod to Canon (and even rank it as superior to the Nikon VR II). What makes it most interesting is that all three use the same tools to test sharpness!

Reviewers and testers make mistakes, too... For example Lensrentals indicate the Tamron 70-200 is crop compatible, while the Canon 70-200 is crop and FF compatible. Of course, the Tamron is fully compatible with both formats, as well. So this is just a typo on the Lensrental site. And the Pop Photo tested/true focal lengths for the Tamron and the Canon are identical... yet Lensrentals comparison of the two lenses clearly show that the Tamron comes up a little short. I have heard and read it tests to something like 185mm actual at the 200mm setting, compared to the Canon which tests to around 195mm actual. Either one is well within the allowable variance (10%, I believe), but there's enough difference that it might matter to some shooters. Pop Photo also shows both lenses have near identical close focusing distance (right at 46"), yet they show that the Canon delivers significantly higher magnification (1:4.45 vs 1:6.75 at 200mm setting)

It can also be revealing to look at Lensrentals Repair Data. But you have to carefully read all the observations and consider the info in context. For example, they have had a lot of Canon 70-200 Mark IIs in heavy use for a couple years now, so have some repair data on them. The Tamron 70-200 VC doesn't even show up on the list, because it's only been in general production and use for less than a year, and I'd wager Lensrentals has far fewer copies of it.

Lensrentals also cleans and thoroughly tests gear after each rental, has insights into common problems and makes heavy use of various repair facilities, as needed. It's particularly interesting comparing repair times and costs... Canon ain't the cheapest nor the most expensive, but is the fastest. Nikon is among the slowest (averaging nearly 5X as long as Canon) and the most expensive (and something not shown here is that last year Nikon notified all independent repairers that they would no longer sell spare parts to them, so with Nikon gear their factory repair service is your only choice... Meanwhile anyone can call Canon and order practically any part from them).

Isn't there a certain percentage of error in most such measurements - dependant upon individual sample(s)?

SamFrench wrote in post #16270234
My question is : Why do you need to "trust" the score?
Just curious.

Most of us could express opinions on which lens is better or which lens should be purchased, but trusting DxOMark provides some objective research to add to the mix . . . but only if you trust the research and results that are produced in the DxOMark labs.

Trusting the scores of an independent non bias evaluator like DXO is a good thing to consult and do.
Trusting any single optics or camera manufacturer blindly is certain blind faith and dangerous.
The truth is sharper than a 2 edge sword and some lenses.

there are some other crazy test as well, like 85 1.8 is better than 85 1.2. 24-70 is I is lot worse than 24-70 is II.

DxOmark lens scores test performance in low light only. Since the Canon has a T stop of T3.6 and the Tamron T3.1, that gives a huge boost to the overall score of the Tamron that isn't quite overcome by the better CA and vignetting scores of the Canon.

DxOmark's description of the lens score:

"The DxOMark Score is measured for defined exposure conditions corresponding to low-light scene with 150 lux illumination and an exposure time of 1/60s. These conditions were chosen as we believe low-light performances are very important for today’s photography and it is also important for photographers to know how well lenses perform at the widest aperture."

This is the first DXO thread on POTN in which I've seen the qualifiers 'non' or 'un' used in front of the word 'biased'.

tkbslc wrote in post #16270257
DxOmark lens scores test performance in low light only. Since the Canon has a T stop of T3.6 and the Tamron T3.1, that gives a huge boost to the overall score of the Tamron that isn't quite overcome by the better CA and vignetting scores of the Canon.

DxOmark's description of the lens score:

"The DxOMark Score is measured for defined exposure conditions corresponding to low-light scene with 150 lux illumination and an exposure time of 1/60s. These conditions were chosen as we believe low-light performances are very important for today’s photography and it is also important for photographers to know how well lenses perform at the widest aperture."

that's comparing apples to apples at least, and it's only one criteria, and a damn important one. I think a common mistake people do when using DXO is comparing two completely different lenses. It's a great tool if you have your own priorities in a lens. It's not the only source to look when comparing lenses, but should be a main source.

lensrental also does a very good job at breaking down lens sharpness, which *usually* translate to good overall optics.

I have a no problem with DxO's tests. The issue I have is with assigning a total score based on a narrow criteria. They should just post the individual metrics and let people decide how those translate into a total package based on what they want to do. Maybe you don't care about vignetting, but T-stop is super important because you are trying to shoot sports indoors. Maybe you care more about reach than light transmission. Either way, the overall DxO score can't decide that for you.

SamFrench wrote in post #16270234
Isn't there a certain percentage of error in most such measurements - dependant upon individual sample(s)?

Yup.

And any such measurements that don't make some sort of attempt to evaluate the likely errors aren't worth the dice used to generate the data.