How real and reliable are DXOmark scores?

timbop wrote in post #17250852
Yes, I get the concept of bayer sensors and RGB. I was thinking in terms of how quickly the wells reach saturation, and if that follows the intensity curve the way it should. I've always felt that the red channel in canon sensors clips a lot faster than green and blue; ie in warmer light I seem to get more of a red/pink hue than I should and under reddish light the highlughts clip pretty fast.

That would be interesting data for DxO to collect, admittedly. The problem is that it's not as simple as saying that the sensor saturates in, say, red, at a given light intensity level. The saturation level is likely to be a curve that is dependent upon light frequency.

Even so, that could be quite valuable to someone who is interested in how well a sensor records color information.

I guess DXO is important enough to always ruffle feathers and seem to be the most active threads. Why are some people so against their scores? (don't answer, it's a rhetorical question)

Yes, you take the whole system into consideration. From lenses, AF speed, high ISO, low ISO, DR, build quality, warranty, etc.

DXO measures only one aspect of it, but it's a very important aspect. You HAVE to take this into consideration when looking for a camera system (and yes, whether you consider DR important to you is considering it)

timbop wrote in post #17250763
So they don't really test color ACCURACY, unless I misread their attempt to confuse and befuddle anyone by displaying some intentionally important-looking formula paired with a bit of terminology from Isaac Newton. In the end, it seems they just compute the number of bits required to store calculated values for range - which isn't really the same thing as reproducing the colors that went into the front of the optics. But who can tell?

Apparently what they say and how they describe the interpretation of their results is not comprehensible or useful to you. If you are befuddled by equations, that would explain it. Then you have no idea whether it's good or bad science and a conspiracy theory of "attempt to confuse" is all in your mind Go look in any scientific or engineering paper, for exampe about color theory, color matrices etc and you will find nothing but "confusion" (aka equations and complex concepts)! But it's from that R&D "confusion" that your camera comes

As I said, their results are not easily converted into easily digested snippets for lay people. If you read carefully, you find that they do not do any raw conversion or white balancing, because that's all to do with software and they only evaluate raw data. The do show that Canon's red channel is much more comtaminated by green and blue than that of the Nikon in that comparison - heck they even show spectral responses to demonstrate the problem. Unfortunately some folks are as befuddled by graphs as they are by equations. This is the old issue of DxO generating data and presenting it in a way that many people don't understand - so it is understandable that the utiity of the data gets dismissed by those who don't understand it. That's the theme of this thread, apparently. Some get the utility of the underlying data from the tests, others don't.

They make a reasonable attempt to describe what they mean by (and how they measure) color sensitivity and it's complicated. What you are asking for requires the manipulation of the raw data to white balance and that's not a property of the sensor. (White balancing amplifies noise in some channels more than others, so the red channel issue is partly a noise issue). Basically what they measure, as quoted in my post above, is how many different colours the sensor can reliably distinguish (like how many grayscale tones in B/W tonality). They represent it by the number of bits needed to encode that number. If you want to know the actual number of distinguishable colors the sensor can record, just take the number of bits they provide (X) and use it in 2^X.

If you are familiar with lighting you may have heard of the CRI - color rendition index -a measure of how many color squares on the standard test card are correctly rendered when illuminated by a lamp (street lights are notorious for misrepresenting colours). DxO's measurement is like that on steroids If you ask about "color accuracy" would it be better to get only 10,000 colors distinguished but perfectly, or 100,000 colors with some slightly off?

Edit: For highest accuracy at any location, shoot a Passport ColorChecker card to do the fine tuning after the raw conversion

AJSJones wrote in post #17251817
... Unfortunately some folks are as befuddled by graphs as they are by equations. This is the old issue of DxO generating data and presenting it in a way that many people don't understand - so it is understandable that the utiity of the data gets dismissed by those who don't understand it. That's the theme of this thread, apparently. Some get the utility of the underlying data from the tests, others don't.....

Odd thing is that they do present it in a way anyone can understand. But when the results are negative to someone's point of view, the complex data is dragged out to show that the data must be wrong and DxO is just trying to confuse everyone.

One point though, DxO measurements are very specific and does not represent the entire package well. It's a data point to use in evaluating a camera, not an overall score of the entire camera itself.

gjl711 wrote in post #17251851
Odd thing is that they do present it in a way anyone can understand. But when the results are negative to someone's point of view, the complex data is dragged out to show that the data must be wrong and DxO is just trying to confuse everyone.

I wish that were so but it's clear many simply don't relate to them. It's the same sort of person who says "I want to see real world images - no-one shoots test cards" - they can do their own subjective evaluation based on those images, but can't provide "objective" measures for others to use. Of course, there are also people who fit your description

gjl711 wrote in post #17251851
One point though, DxO measurements are very specific and does not represent the entire package well. It's a data point to use in evaluating a camera, not an overall score of the entire camera itself.

Many have said that in this thread and the disclaimer is in many, many sections of the DxO site.

kcbrown, what drew me to have 2 5d3's, in order of descending importance:
1 - autofocus performance. otherwise, I would have kept my 5d2
2 - high ISO image quality. otherwise, I would not have replaced my 7d with a second 5d3
3 - speed of operation, controls, and handling. the 5d3's are the right blend of customizability, with the lone exception of only being able to assign the DOF preview button for onseshot <-> AI servo
4 - familiarity with canon UI, and canon's glass - otherwise I might have gone d700 6 years ago

timbop wrote in post #17251982
kcbrown, what drew me to have 2 5d3's, in order of descending importance:
1 - autofocus performance. otherwise, I would have kept my 5d2
2 - high ISO image quality. otherwise, I would not have replaced my 7d with a second 5d3
3 - speed of operation, controls, and handling. the 5d3's are the right blend of customizability, with the lone exception of only being able to assign the DOF preview button for onseshot <-> AI servo
4 - familiarity with canon UI, and canon's glass - otherwise I might have gone d700 6 years ago

Not much of that list is addressed by DxO, so you can safely ignore the rest of their data. The only item in your list is high ISO IQ. That is, just coincidentally, where Canon leads Nikon in DR

David Arbogast wrote in post #17250944
DXO scores are a huge deal here at POTN, as evidenced by explosive threads such as this one, because Canon sensors score lower. If DXO scored Canon sensors about the same or better than Sony sensors, it would be less of a discussion/debate.

I agree. If only Canon could make a sensor that would rival the Sony/Nikon.

timbop wrote in post #17251982
kcbrown, what drew me to have 2 5d3's, in order of descending importance:
1 - autofocus performance. otherwise, I would have kept my 5d2
2 - high ISO image quality. otherwise, I would not have replaced my 7d with a second 5d3
3 - speed of operation, controls, and handling. the 5d3's are the right blend of customizability, with the lone exception of only being able to assign the DOF preview button for onseshot <-> AI servo
4 - familiarity with canon UI, and canon's glass - otherwise I might have gone d700 6 years ago

I love the ease of use with Canon. There's no doubting that. I've been shooting Nikon for three years now and I've got a nice workflow with my cameras. Is it different? Slightly. I'm happy with my dual D750's, D4s, & D810 for wedding work.

I take DXO with a grain of salt.

timbop wrote in post #17251982
kcbrown, what drew me to have 2 5d3's, in order of descending importance:
1 - autofocus performance. otherwise, I would have kept my 5d2
2 - high ISO image quality. otherwise, I would not have replaced my 7d with a second 5d3

Aha, so you are concerned with the sensor performance, and it even ranks near the top!

DxO performs measurements that reveal the high ISO performance of the sensor, though I've seen nothing about them that will reveal or account for artifacts generated by the sensor. But as with all such data, you have to know how to interpret it. The signal to noise ratio graph, combined with the dynamic range graph, will tell you how well the sensor performs at high ISOs.

I have to say that Canon does have an excellent user interface, quite a lot better than the Nikon. I've adapted to both easily enough, but it's easier to find things with the Canon. Their UI is more logically presented and organized, and that makes it a bit more "discoverable".

gjl711 wrote in post #17251851
One point though, DxO measurements are very specific and does not represent the entire package well. It's a data point to use in evaluating a camera, not an overall score of the entire camera itself.

The other thing to note is that even restricted to analysis of sensor performance (something they always point out; though it's often ignored), the single 'headline' DxO number for sensor performance actually combines different aspects of sensor performance using a (last time I looked, undocumented) weighting scheme which may not reflect a photographer's personal weighting (eg. performance at base ISO may be more important to a landscape photographer; high ISO performance to a night sports photographer) and ignores other aspects of the sensor such as resolution and (to a certain extent) sensor size, which may be of considerble importance for someone's use. 'Drilling down' into the measures it is derived from is always indicated if you weight one or more of DxO's 'use case' numbers as more important than others.

And, of course and as you point out, that headline DxO number and even the subsidiary numbers it is derived from say nothing about any other aspect of the camera or camera system.

...Mike

kcbrown wrote in post #17252141
Aha, so you are concerned with the sensor performance, and it even ranks near the top!

...

I NEVER said I don't care about image quality, and obviously that comes from the sensor. You're trying to twist my words; the performance I am most concerned about is high ISO noise not base ISO DR.

timbop wrote in post #17252809
I NEVER said I don't care about image quality, and obviously that comes from the sensor. You're trying to twist my words; the performance I am most concerned about is high ISO noise not base ISO DR.

Understood. The point here, however, is that the fact that you are concerned about image quality means that at least some of DxO's measurements are relevant to you. The scores might not be (though, if you notice, the 5D3's high ISO score is right up there), but the measurements certainly are. And that includes the DR as measured at the ISOs you're interested in.