Metadata/lenses

[deblur]

I was having a strange issue with identifying lenses, I think I've kind of worked around it, but just wondering exactly how the metadata parsing is working?

All the shots taken with one of my Sigma lenses were being identified as random other Canon lenses upon upload. The upload messages indicated "GUESS LENS: | Searching maker: Canon", so I found if I changed the lens manufacturer to Canon for the Sigma lens in the Lens editor, it then identified it correctly. Obviously this seems somewhat odd, but it does work now.

I'm presuming the identification is done based on matching the recorded manufacturer, focal length and max aperture found in the EXIF data with the details in the lens database. I'm wondering though if it wouldn't be easier to match based upon the lens string in the EXIF data to avoid problems? I mean how does identification work if you there are two lenses in the database with all the same details (e.g. Canon 70-200 f/4 and the IS version)?

On a related note, is any of the other metadata saved that isn't specified on the photo details page? I mean there's lots of useful EXIF data various RAW viewers will show, such as flash information/compensation... and then there is other data that can be set by programs like Lightroom/Bridge such as ratings which would be nice to import too. I hope I'm not being too demanding here! I just have a lot of photos to import and I want to try and get things working right so I don't have to go importing them all again due to missing/wonky metadata! In a way it would be nice to have the full EXIF data available somehow...

[wkitty42]

I'm presuming the identification is done based on matching the recorded manufacturer, focal length and max aperture found in the EXIF data with the details in the lens database. I'm wondering though if it wouldn't be easier to match based upon the lens string in the EXIF data to avoid problems? I mean how does identification work if you there are two lenses in the database with all the same details (e.g. Canon 70-200 f/4 and the IS version)?
all i can say is thank you! this is a niggley type thing that has bothered me for a while but i've not had time to say anything or a means of identifying the problem that i've been seeing...

[deblur]

ok, this is still a problem in some cases, it seems to be related to the focal length of the image. For instance any images taken with a focal length of 17mm are identified as being taken by the 17-35mm lens, even when they weren't.

Even stranger, any images taken at 13mm are identified as having being taken with the 135mm f/2 lens!

[DavidW]

You can debug the lens guessing using the Server Tools menu. Change to the Upload folder, select a file, and press VIEW COMBINED CAMERA DATA. EE will show you the data it has to guess the lens, and it will then take you through the reasoning it uses.

I don't think there's any way you can tell using the metadata between images taken with an EF 70-200mm f/4L and an EF 70-200mm f/4L IS - the matching system isn't sophisticated enough to tell between two lenses with the same focal length range and maximum aperture. I believe only the latest bodies record the 'name' of the lens (my 20D certainly doesn't).

However, it should be possible to debug many other scenarios. Ensure that parameters for your lenses are correctly set up in the lens database, then cut and paste the 'guessing' stages here if it's not working properly.

Use the PHP EXIF module if you can; the Internal EXIF code is somewhat buggy.



David

[deblur]

ok, thanks, I'm using PHP exif...

First of all, I can get the lens string under the variable $xmp_lens from the XMP/XAP data listing. This string is also shown in Bridge etc. under 'Camera Data->Lens' (or more precisely, if I go to 'File Info' from the file menu, then 'Advanced' tab, under the 'aux:Lens' string under the 'http://ns.adobe.com/exif/1.0/aux' list. My Sigma lens lists as '18.0-50.0 mm' here and strangely just '18.0-50.0' under $xmp_lens. My 70-200 lists as 'EF70-200mm f/4 USM' in Bridge and as 'EF70-200f/4LUSM' in the $xmp_lens string.

This lens name data is available from both my 30D and the 300D (which I believe is older than the 20D, so I'd be surprised if it's not there too).

Now, getting on to the focal length ranges and max apertures which are actually used in identification, there certainly seems something odd. For the 18-50 lens, in the Combined Data test I get:
AVAILABLE DATA
xmp_focallenght 50
xmp_lens 18.0-50.0
exif_focal 50
canon_longfocal
canon_shortfocal
exif_focalrange

Searching maker: Canon | Searching aperture: 8 | Searching focal: 50 | Searching focal range: 18-50

MANUFACTURER HIT POINTS FOR IDS 2 4 6 7 9 10 11 12 16 17 18 19 20 21 22
MIN APERTURE HIT POINTS FOR IDS 1 2 3 4 5 6 7 8 9 10 11 12 16 17 18 19 20 21 22
ACTUAL FOCAL MM HIT POINTS FOR IDS 2 4 10 20
IF PRIME HIT POINTS FOR IDS 2
RANGE SEARCH HIT POINTS FOR IDS
highest ranking lens id found is: 2
Canon 50 f/1.4

note the correct id for the 18-50 lens is 20. The strange thing is it seems to be identifying it as a prime (this image was taken at 50mm). For an image with the same lens at 33mm, it is correctly identified (the 'IF PRIME HIT POINTS FOR IDS' doesn't have a number beside it in this case).
Looks like the max aperture info is missing/not being used for identification as well. Clicking on the 'DO THE SELECTED TEST' for EXIF (PHP MODULE) gives these items (among others):
COMPUTED => ApertureFNumber =>f/8.0
FNumber =>8/1
ApertureValue =>6/1
MaxApertureValue =>296875/100000

The true aperture value was f/8 and max aperture should be 2.8.
Bride lists the same items under the 'Advanced' tab of File Info, but correctly interprets the 'Max Aperture Value' as f/2.8 under the 'Camera Data 1' tab. Furthermore, the lens range is available under the 'aux:LensInfo:" string under the 'http://ns.adobe.com/exif/1.0/aux' list in Bridge, shown as "18/1 50/1 0/0 0/0".
The thing that is most puzzling me here is that EE has pulled the "Searching focal range: 18-50" from somewhere, and then not used it!

Running the same tests on an image taken with the 70-200 lens, I find
AVAILABLE DATA
xmp_focallenght 188
xmp_lens EF70-200f/4LUSM
exif_focal 188
canon_longfocal
canon_shortfocal
exif_focalrange

Searching maker: Canon | Searching aperture: 8 | Searching focal: 188 | Searching focal range: 0-200

MANUFACTURER HIT POINTS FOR IDS 2 4 6 7 9 10 11 12 16 17 18 19 20 21 22
MIN APERTURE HIT POINTS FOR IDS 1 2 3 4 5 6 7 8 9 10 11 12 16 17 18 19 20 21 22
ACTUAL FOCAL MM HIT POINTS FOR IDS 6 9 16 17 21 22
IF PRIME HIT POINTS FOR IDS
RANGE SEARCH HIT POINTS FOR IDS 6 16 17 22
highest ranking lens id found is: 22
Canon EF 70-200 f/4.0L + 1.4x II

Note the mistake (the 1.4x II shouldn't be there). The 70-200 lens alone should be lens id 17. In this case, it wasn't matched as a prime (I guess there was no 188mm prime in the database!), but it did find several lenses covering the focal range (although for some reason this range is shown as 0-200 instead of 70-200).
The EXIF (PHP MODULE) data now shows
COMPUTED => ApertureFNumber =>f/8.0
FNumber =>8/1
ApertureValue =>6/1
MaxApertureValue =>4/1

The correct data is clearly shown in Bridge again - this time the 'aux:LensInfo:" string is "70/1 200/1 0/0 0/0".

here is the stranger example with the 13mm image:
AVAILABLE DATA
xmp_focallenght 13
xmp_lens EF-S10-22f/3.5-4.5USM
exif_focal 13
canon_longfocal
canon_shortfocal
exif_focalrange

Searching maker: Canon | Searching aperture: 4 | Searching focal: 13 | Searching focal range: 13

MANUFACTURER HIT POINTS FOR IDS 2 4 6 7 9 10 11 12 16 17 18 19 20 21 22
MIN APERTURE HIT POINTS FOR IDS 1 2 3 4 5 6 7 8 10 11 12 16 17 18 19 20 21
ACTUAL FOCAL MM HIT POINTS FOR IDS 7 18
IF PRIME HIT POINTS FOR IDS
RANGE SEARCH HIT POINTS FOR IDS 4 11
highest ranking lens id found is: 11
Canon EF 135mm f/2 L
The correct lens is id 18 - which hasn't even appeared in the range search!

I have checked all the details in the lens editor are correct - other than I changed the Manufacturer of the 18-50 to Canon instead of Sigma as I mentioned previously. As far as I am aware there is no info in any metadata indicating the 18-50 as a Sigma lens (there is no lens manufacturer field, only camera manufacturer, therefore entering this in the lens editor is something of a mistake!)

One last point - I don't know if the typo in 'xmp_focallenght' is an issue?

[Imaginos]

I appologize for dragging up a thread that's 6 months old, but this is a issue in my current efforts with EE. It seems strange that I should have to enter lens information manually in this day and age, especially when both my 30D and 5D encode the specific lens data in the EXIF, pretty specifically as "EF70-200mm f/2.8L IS USM" or "EF100mm f/2.8 Macro USM." It seems that it should be possible to extract that from the EXIF during upload. If so, then this a coding drill to look up the correct lens based on one and only one field. While I am fluent in a veriety of programming languages and SQL, I'm new to php and I have no clue about the architecture of EE.

The things I know are: php is capable of getting the canon-specific lens data out of the exif; a breif investigation reveals that the file of interest may be admin/PHOTODATA_exif.php. Futher, ee_lens_model is stored in in the ee_lens table in mysql and the lookup assignment appears to be done in basecode/SCRIPT_lens.php. The select statment can probably be reduced to just "SELECT DISTINCT ee_lens.ee_lens_id WHERE ee_lens_model = <CANON_EXIF_LENS>."

The hard part will be getting the lens info from exif into the database so that it's available at the right step. It seems like the soltuion should be modifying maybe 10 lines of code in a copy/paste manner, then the setting up the lens model names correctly. After that, lens id should be fully automated and exact for the cameras that support it.

I fully intend to poke at it a bit when I get the time, but if someone who knows more about php feels like beating me to it, it'd be a big, big help. :D

[MMCM]

I too, think It would be fine, if lens detection would work 100%. I disabled that "feature" back with EE 1.5, because it mostly guessed wrong.
Maybe it would help too, if we could assign specific lens to specific camera bodies, so EE wouldn't guess a EF-S 17-85 with a 5D ;-) (Just an example, I don't have neither EF-S lens nor a 5D)
Another (optional) feature could be to store all metadata in the EE database, for later analysis, if new features or algorythms are developed later. In my EE gallery all EXIF info is stripped, to there's no way to redetect lens later with an improved version of EE.

[deblur]

Imaginos - I fully agree with your suggestion, I had been intending to try exactly that myself too, but lack of time and familiarity with the code have also held me back!

MMCM - keeping the exif is potentially a good idea, but maybe it could make the database get rather large? I know some places have limits of like 2 or 5MB for the database. It could I suppose store it in the picture file, but then everything is visible to the public which not everyone might want.

Perhaps the solution to that (and another issue!) would be to move the picture that gets processed in the uploads folder to an 'originals' folder. This could then be re-read whenever you want to update metadata, and has the added advantage that you can re-generate the different sized photos if you change the settings (such as watermarking, sharpening) or if you missed out a size path on initial upload.
Currently there is a 'rethumbnail' tool, which generates new thumbnails from the processed pictures. I don't see why there couldn't be a batch 'reupload' command for the big sizes. Currently this has to be done manually via the photo editor for each image, and can't even make use of files still in the uploads folder!! Hence it currently seems easier to wipe the exhibition and start again if you make a mistake - very time consuming.

[Imaginos]

After some poking around I have a functional prototype of how to do this.

I cannot stress enough that this isn't yet a beta. It's not even an alpha. It's crayola. It probably has some horrible unintended consequences. Use at your own risk.

What follows will read the xmp lens data from a jpg saved with Adobe CS2 and CS3 (and probably CS and earlier, but I don't have them to experiment with). It will attempt to make an exact match between the model you have in your lens data and the xmp data in the picture. If it finds a match, it assigns that lens. If no match is found, it does nothing to modify what EE has done otherwise.

I have tested it with the data encoded by a Canon 30D and 5D, but no others. Your lens table must have it's "model" field set to read the same as the camera will encode it. It's a straight forward encoding with these examples:
EF100mm f/2.8 Macro USM
EF17-40 f/4L USM
EF24-105mm f/4L IS USM
EF70-200mm f/2.8L IS USM
EF-S17-85mm f/4-5.6 IS USM

These are ones that I have functioning on my site. Some experimentation may be required to get the correct model for your glass.

In the file admin/SCRIPT_add_to_database.php

Locate the part of the file that reads like such (it's around 25% of the way through the file):
if ($s_upload_hide_photos == "0") {
$ee_photo_for_www = "1";
}
// do new photo with default values and get its id
$do_a_photo = mysql_query(
"INSERT INTO ee_photo (ee_camera_id,ee_photo_panorama,ee_owner_id,ee_cou ntry_id,ee_location_id,ee_photo_exif_date,ee_photo _exif_time,ee_photo_exif_mode,ee_photo_exif_meteri ng,ee_photo_exif_iso,ee_lens_id,ee_light_id,ee_wor kflow_id,ee_photo_for_www,ee_photo_exif_ev,ee_phot o_orientation)
VALUES ('$camera','$panorama','$owner','$country','$locat ion','$date','$time','$mode','$metering','$iso','$ lens','$light','$workflow','$ee_photo_for_www','$e v','$orientation')
");
ee_error ($do_a_photo,"do_a_photo","SCRIPT_add_to_database.php");
paste this immediately before it:


//
// Imagino's Super-Mega Tokyo Doriftou XMP mod.
//
$aux_lens_loc=$upload_source_directory . $k["filename"];
ob_start();
$aux_lens=readfile($aux_lens_loc);
$aux_source = ob_get_contents();
ob_end_clean();

$aux_xmpdata_start = strpos($aux_source,'aux:Lens="')+10;
$aux_xmpdata_end = strpos($aux_source,'"',$aux_xmpdata_start+1);
$aux_xmplenght = $aux_xmpdata_end-$aux_xmpdata_start;
$aux_lens = substr($aux_source,$aux_xmpdata_start,$aux_xmpleng ht );

$aux_query = 'SELECT ee_lens_id FROM `ee_lens` WHERE ee_lens_model = "'.$aux_lens.'"';
$aux_result=mysql_query($aux_query);
$aux_lens=mysql_result($aux_result,0);
if($aux_lens>0) $lens=$aux_lens;

//End Mod
Save and conquor.

Known issues:
I know this will fail if, for any reason, EE renames the photo during the upload process. That's because I've not yet found where EE stores the new filename. It's probably got some other errors, so when you find them, please post them here.

[deblur]

ok I've tried the mod... but no luck so far :(
I'm sure I've entered the names right in the lens model field, though the lens name (multilingual field) in each case is different. I don't think the renaming is an issue, I tried it with a simple filename.
I'm using lightroom to export the photos.

I'm wondering if the method doesn't work with the internal EXIF? (I don't have the php on the server).

[Imaginos]

The above hackware doesn't read the EXIF, it reads xmp data straight from the file (should never have a conflict with internal or external EXIF). Lightroom may not put EXIF and/or XMP data in the JPGs much in the same way that Photoshop will remove it if you do a "save for web" vice "save as." I don't have Lightroom so I can't experiment with it to know for sure.

Also, the lens model information goes in the "model:" field underneath "Manufacturer:" and above the "Photo URL," not the big black multilingual field at the top.