Approve the Cookies
This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and our Privacy Policy.
OK
Index  •   • New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Guest
New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Register to forums    Log in

 
FORUMS Post Processing, Marketing & Presenting Photos RAW, Post Processing & Printing 
Thread started 02 Jul 2017 (Sunday) 20:31
Search threadPrev/next
sponsored links
(this ad will go away when you log in as a registered member)

Photo libraries are an unorganized disaster. I want to consolidate, export everything out of Apertur

 
qwerty11
Member
51 posts
Joined Dec 2009
     
Jul 02, 2017 20:31 |  #1

As the title says. I have a couple Aperture libraries, I also have several folders that I have exported pictures out of Aperture into. I also have folders with photos that are not in Aperture. The problem is, I'm not sure what has been exported or what is duplicated in the Aperture libraries. I am wanting to clean house and move everything over to Lightroom. I am trying to figure out the best way to do this. My intuition is to import/merge everything into on Aperture library, export this to Lightroom, and then use some kind of duplicate photo scrubber program to get rid of the excess.

I really could use some opinions on this. If I do not take care of this now, it will just be a larger problem latter.




  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)
Dan ­ Marchant
Do people actually believe in the Title Fairy?
Avatar
5,526 posts
Gallery: 19 photos
Likes: 1956
Joined Oct 2011
Location: Where I'm from is unimportant, it's where I'm going that counts.
     
Jul 03, 2017 00:29 |  #2

Surely it would be better to do the search/delete of duplicates before going to the trouble of moving stuff into Aperture and then out to Lightroom.


Dan Marchant
Website/blog: danmarchant.com (external link)
Instagram: @dan_marchant (external link)
Gear Canon 5DIII + Fuji X-T2 + lenses + a plastic widget I found in the camera box.

  
  LOG IN TO REPLY
john ­ crossley
Goldmember
2,830 posts
Likes: 1042
Joined Nov 2009
     
Jul 03, 2017 06:03 |  #3

qwerty11 wrote in post #18392836 (external link)
As the title says. I have a couple Aperture libraries, I also have several folders that I have exported pictures out of Aperture into. I also have folders with photos that are not in Aperture. The problem is, I'm not sure what has been exported or what is duplicated in the Aperture libraries. I am wanting to clean house and move everything over to Lightroom. I am trying to figure out the best way to do this. My intuition is to import/merge everything into on Aperture library, export this to Lightroom, and then use some kind of duplicate photo scrubber program to get rid of the excess.

I really could use some opinions on this. If I do not take care of this now, it will just be a larger problem latter.

You can set up Lightroom NOT to import suspected duplicates.


I've just caught my wedding vegetables in the flange!!!

  
  LOG IN TO REPLY
-Duck-
my head is usually in the way
Avatar
1,711 posts
Gallery: 19 photos
Best ofs: 1
Likes: 751
Joined Apr 2016
Location: Shelton, CT USA
     
Jul 03, 2017 20:38 |  #4

Unfortunately you do not have an easy task ahead of you, regardless of any automation. As I like to tell my students, "it's all monkey work."

I have had two clients that felt the blow to their catalog when Aperture came to a halt. In both cases I installed the plugin to migrate the Aperture images into Lightroom, however there are some significant structural differences with the two programs that created some issues. The first is that Aperture loves folders and subfolders and sub-subfolders. Who the hell thought that was a great ideas should be taken out and quartered. Secondly, all the generated images from the edits hanging around all over the system. Well, with both my customers they did not have any kind of workflow in place so their images were everywhere with no real sense of order. Wrangling them took forever. they had all kinds of ridiculous orphaned duplicates labeled 'filename(1).jpg, filename(2).jpg, filename(3).jpg and so on). Some were direct edits from Aperture, some were edits of the edits and some were resized versions of the edits.

There is a plugin that will search through your existing library and seek out suspected duplicates. The Lightroom Duplicate Finder (external link) is a plug-in that uses the EXIF data in your files, comparing model, serial number, lens, ISO, and other attributes, to flag images in your catalog that it suspects are duplicates. However, you still need to manually go through each image, evaluate what is what and decide on what action to take. Hence the term, monkey work.

The thing you need to understand is that it will search the library, not your computer. If there are duplicates on your system NOT in the library it won't flag it. The best suggestion would be once you import your images from Aperture, do a sweep of your system to import any stray images Lightroom finds (obviously staying out of system folders). This will import those stray images that you can then run a duplicate search on.

One thing I teach in my Lightroom workshop, besides organization (all exports in one place), is not to hold on to exported images once you've used them (i.e. posted to Facebook, a forum, Flickr or your portfolio page). There's just no need for it. If you somehow need that JPG again either take it from where you uploaded it to or generate a new output from Lightroom. This saves on disc space and the headache of having orphaned duplicates all over the place.


"If you didn't learn something new today, you wasted a day."
Unitas Photography (external link)Facebook (external link)

  
  LOG IN TO REPLY
Dan ­ Marchant
Do people actually believe in the Title Fairy?
Avatar
5,526 posts
Gallery: 19 photos
Likes: 1956
Joined Oct 2011
Location: Where I'm from is unimportant, it's where I'm going that counts.
     
Jul 03, 2017 21:50 |  #5

-Duck- wrote in post #18393642 (external link)
One thing I teach in my Lightroom workshop, besides organization (all exports in one place), is not to hold on to exported images once you've used them (i.e. posted to Facebook, a forum, Flickr or your portfolio page). There's just no need for it. If you somehow need that JPG again either take it from where you uploaded it to or generate a new output from Lightroom. This saves on disc space and the headache of having orphaned duplicates all over the place.

To this end I have a separate folder "Public". All exported JPEGs go into sub folders in this folder. I can then easily find them to delete them once I have finished with them.


Dan Marchant
Website/blog: danmarchant.com (external link)
Instagram: @dan_marchant (external link)
Gear Canon 5DIII + Fuji X-T2 + lenses + a plastic widget I found in the camera box.

  
  LOG IN TO REPLY
Nick5
Goldmember
Avatar
3,170 posts
Gallery: 3 photos
Likes: 249
Joined Mar 2007
Location: Philadelphia Suburbs
Post edited over 2 years ago by Nick5.
     
Jul 03, 2017 23:25 |  #6

Big project for me as well as I am in the same boat. As an avid Aperture user myself, I am now confident enough in Lightroom that I am in the process of copying all 4TB's of Aperture Library Masters to other external drives.
I will then Import each folder of images which are dated in the Aperture hierarchy and then import into Lightroom. Since I decided with the help of many here, I import by date into Lightroom. I then add a description to each dated folder in Lightroom 2017-05-30 Lisbon for example. Yes this will be a very big, time consuming project. However, the import by date will be consistent through out the years.
I did look at the Auto import Adobe created for Lightroom. However it seems to put them into Collections rather than the same DATE/ Description folder format I have created since Jan. 2017.
Once imported, I will then export all of the Masters from Lightroom that are on another External Drive onto additional back up drives.
Time Cosuming.....Yes.....b​ut very organized and consistent.


Canon 5D Mark III (x2), BG-E11 Grips, 7D (x2) BG-E7 Grips, Canon Lenses 16-35 f/4 L IS, 17-40 f/4 L, 24-70 f/4 L IS, 70-200 f/2.8 L IS II, 70-200 f/4 L IS, 70-200 f/4 L IS Version II, 100-400 f/4.5-5.6 L IS Version II, TS-E 24 f/3.5 L II, 100 f/2.8 L Macro IS, 10-22 f3.5-4.5, 17-55 f/2.8 L IS, 85 f/1.8, Canon 1.4 Extender III, 5 Canon 600 EX-RT, 2 Canon ST-E3 Transmitters, Canon Pixma PRO-10 Printer

  
  LOG IN TO REPLY
BigAl007
Cream of the Crop
Avatar
8,024 posts
Gallery: 543 photos
Best ofs: 1
Likes: 1632
Joined Dec 2010
Location: Repps cum Bastwick, Gt Yarmouth, Norfolk, UK.
     
Jul 04, 2017 11:08 |  #7

Nick5 wrote in post #18393728 (external link)
Big project for me as well as I am in the same boat. As an avid Aperture user myself, I am now confident enough in Lightroom that I am in the process of copying all 4TB's of Aperture Library Masters to other external drives.
I will then Import each folder of images which are dated in the Aperture hierarchy and then import into Lightroom. Since I decided with the help of many here, I import by date into Lightroom. I then add a description to each dated folder in Lightroom 2017-05-30 Lisbon for example. Yes this will be a very big, time consuming project. However, the import by date will be consistent through out the years.
I did look at the Auto import Adobe created for Lightroom. However it seems to put them into Collections rather than the same DATE/ Description folder format I have created since Jan. 2017.
Once imported, I will then export all of the Masters from Lightroom that are on another External Drive onto additional back up drives.
Time Cosuming.....Yes.....b​ut very organized and consistent.


You don't need to export "masters" from LR, it doesn't actually use that concept, since unlike the default option of Aperture, LR is a referenced system only. So while Aperture appends the original file data as part of the catalogue database file itself, which means if you lose the catalogue file you lose all the images, as well as the edits, LR only contains a pointer to the original files location on the disk. It does mean that you need to backup both the catalogue file (*.lrcat) and also all the original image files.

If you have a major loss of a drive when working with LR it does mean that you have to restore both the catalogue and the images separately, but as long as you put them in the same location LR will automatically match the recovered catalogue with the images. If does of course mean that you have to be careful not to move your image files outside of LR, since this breaks the pointer to the file. LR does provide tools to fix things if you do happen to move the files by accident.

At least this makes backing up your images files pretty simple, since all you really need to do is copy the folder tree with the images on it to a different physical drive, and remember to back up your catalogue file too, again to a different physical drive. By default LR wants to back up the catalogue to the same drive as the original.

Oh and if the export to LR is putting all of the images into one big images folder, and just creating collections by date, I would still go with that process. If the image has been put into a collection by date, it is then really easy to move all of the images in a certain collection to a new folder. Doing it this way should then preserve all of your other added information like keywords and other added EXIF data that would normally be stored in the catalogue or xmp sidecar file. Overall this should save you much work.

Alan


My Flickr (external link)
My new Aviation images blog site (external link)

  
  LOG IN TO REPLY
cacawcacaw
Goldmember
Avatar
2,862 posts
Likes: 19
Joined May 2010
Location: Ventura, California
     
Aug 08, 2017 22:23 |  #8

"Monkey work" is right, as we evidently haven't been able to get past a monkey-like level of thinking when it comes to organizing and consolidating photos.

We have all kinds of tools available, including image finders, file movers and categorizers, duplicate finders, sync enablers, and backup minders. But there is still no one tool, or even a simple "recipe" for taking old image collections, including duplicates, backups, and edits, and moving them into a single consolidated folder structure.

Lightroom has a proprietary catalog, preventing the use of any efficient duplicate finders. Apple Photo also has a proprietary way of handling photos, reducing the size of some and never letting you directly manage folders. Same with Google Photos. And none of these programs allow quick scanning of your photos without waiting for images to be imported or redrawn.

I had high hopes when I discovered Mylio but it's turning out to be almost as confusing as the others. (One thing it absolutely has going for it is an incredibly fast refresh rate for viewing blocks of photos. It's even better than Photo Mechanic.) I have a bunch of personal photos, including duplicates, copies, and backups. And, I have a large digital photo collection left by my Dad, also containing numerous duplicates and edited copies.

Why isn't there a simple way to start with a master photo library (in a regular folder structure, not a proprietary catalog) and then add to it from different sources (old computers, hard disks, online accounts, etc.) without creating layer upon layer of duplicates. Even better would be one that converts Lightroom and Aperture edits into self-standing files (by using a sidecar file) and handles them in the same way.

Several years ago, I posed this same question and figured I'd wait for the software to catch up. It hasn't. There doesn't even seem to be a documented process that details the steps for consolidating old photos. Even the experts have nothing better than the non-advice of "Roll up your shirt sleeves and get to it." There's got to be a more intelligent approach than just jumping into the monkey pit.

Yes, I know what I ought to do in the future. What i want is to know how to organize my existing photos in an efficient and safe manner.


Replacing my Canon 7D, Tokina 12-24mm, Canon 17-55mm, Sigma 30mm f/1.4, 85mm f/1.4, and 150-500mm with a Panasonic Lumix FZ1000. I still have the 17-55 and the 30 available for sale.

  
  LOG IN TO REPLY
Dan ­ Marchant
Do people actually believe in the Title Fairy?
Avatar
5,526 posts
Gallery: 19 photos
Likes: 1956
Joined Oct 2011
Location: Where I'm from is unimportant, it's where I'm going that counts.
     
Aug 09, 2017 03:11 |  #9

cacawcacaw wrote in post #18422749 (external link)
But there is still no one tool, or even a simple "recipe" for taking old image collections, including duplicates, backups, and edits, and moving them into a single consolidated folder structure.

True, and there probably won't be until we make some major advances in AI because pattern recognition is one of those things that current computers simply aren't ever going to be good at. Sure, if you have an NSA mainframe and a million dollar software package you can do it but that is a long way from being something that you and I can afford and it won't run on your desktop. Compare that to the human brain and there is no contest. We can recognise someone who is barely visible based on the smallest details of facial feature recognition or posture. We can even recognise people from the back.

Lightroom has a proprietary catalog, preventing the use of any efficient duplicate finders.....

Never used the other packages you mentioned but Lightroom's catalog certainly doesn't prevent the use of any duplicate finding software. The catalog is just a list of image file locations. It points to your image files and associates it's list of image adjustments with the correct image file. If you want to run a duplicate finder on those files LR couldn't care less.

Why isn't there a simple way to start with a master photo library (in a regular folder structure, not a proprietary catalog) and then add to it from different sources (old computers, hard disks, online accounts, etc.) without creating layer upon layer of duplicates.

There is - that is exactly what Lightroom does. It allows you to put your files almost anywhere you want in any file structure (regular or irregular) and it allows you to add images from other drives. It can even publish images to your online collection such as Flickr etc and keep track of those.

Of course there are limits. It can't keep track of all your images across different computers because it isn't a networked app (and making it so would be a far from simple task) and it can't detect duplicate images for the reasons I mentioned above.

Even better would be one that converts Lightroom and Aperture edits into self-standing files (by using a sidecar file) and handles them in the same way.

Can you explain what you mean by this? Lightroom does use Sidecar files (if you select that option) as I believe do other packages. But what do you mean by a "self-standing" file and how do you think they would help you?

Yes, I know what I ought to do in the future. What i want is to know how to organize my existing photos in an efficient and safe manner.

I suggest you get started sorting them yourself because, unless you can buy a Cray you are unlikely to have the raw processing power to do a brute force image compare and AI is some way off from being able to make the cognitive leaps necessary for what you want (and when it does it will probably decide it would rather exterminate us than do our monkey work).


Dan Marchant
Website/blog: danmarchant.com (external link)
Instagram: @dan_marchant (external link)
Gear Canon 5DIII + Fuji X-T2 + lenses + a plastic widget I found in the camera box.

  
  LOG IN TO REPLY
BigAl007
Cream of the Crop
Avatar
8,024 posts
Gallery: 543 photos
Best ofs: 1
Likes: 1632
Joined Dec 2010
Location: Repps cum Bastwick, Gt Yarmouth, Norfolk, UK.
     
Aug 14, 2017 15:49 |  #10

Sorry for the delay in posting, I've been in hospital for a week with back trouble. To expand a little on Dan's post the Lr catalogue is not a proprietary file at all. It is actually just a simple single user SQLite database, and as such anyone with SQLite DB development experience should have no issues at all playing with the catalogue outside of Lr. I have actually exported a small section of my Lr catalogue, so that it is not too large, the full file being 15 GB, and managed to open the file as SQLite in Microsoft Access 2007.

Also unlike Apple, who default to a managed system by default, and wrap all the image files into the database in Aperture, iPhoto, and the newer Photos apps, which personally I don't like one bit. Lr on the other hand is a purely referenced system, and only points to the image location on disk. If you turn it on in Lr it will also automatically write to a standard Adobe format .xmp file, which is a simple plain text file. It is perfectly feasible to apply ACR/Lr edits to an image by simply typing up a suitable set of commands in a text editor. You can even bring them into the Lr database by doing a read from xmp command on the file. This is how some extensions for both Lr6 standalone and the limited version of ACR in Ps Elements have implemented dehaze in the older version. I have even seen examples of people writing code for adding local brush and graduated filters to ACR in PSE. It's not that hard to do if you have examples to work from.

The program Fast RAW Viewer which is very good for selecting files before adding them to Lr or working on them in ACR allows you to apply basic edits as well as adding metadata to images, as it uses the same .xmp format as Adobe uses. Fast RAW Viewer is great as it allows the use of a true RGGB RAW histogram as well as the basic edits. If Lr finds a correctly named .xmp file in a folder when importing an image it will automatically read that, and use it's edits instead of the defaults.

There will never be a fully standard sidecar file format that will work with every RAW converter, since every RAW converter has a different set of edit commands. Each program will need to have it's own format. Adobe have made the xmp format that they use an open standard, so that although the image editing data will need to be proprietary, the associated metadata, keywords, gps location, copyright, and other EXIF and IPTC etc. information may be stored in an xmp file and shared between applications.

Since all of the image files themselves are simply stored within the normal file system, you should be able to use any duplicate file finder to match files, although actually pattern matching the image is going to be difficult. It should also be possible to run a check using database tools on the catalogue to see if both images are also in the catalogue. Saying that if the images are already in the Lr catalogue all you would really need to do is go to the view all images and order the images by creation time. Any duplicate images that are already in the Lr catalogue will show up next to each other in this view. You should also see any created PSD/TIFF/JPEG files, as well as original RAWs. If you have duplicates that are not in the catalogue then your other duplicate file finders will hopefully sort them out. All you then need to do is check to see which if any of the files is already in the catalogue, which shouldn't be overly difficult, even if ti requires a manual check. Since you now know where to look.

Alan


My Flickr (external link)
My new Aviation images blog site (external link)

  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)

3,574 views & 2 likes for this thread
Photo libraries are an unorganized disaster. I want to consolidate, export everything out of Apertur
FORUMS Post Processing, Marketing & Presenting Photos RAW, Post Processing & Printing 
AAA
x 1600
y 1600

Jump to forum...   •  Rules   •  Index   •  New posts   •  RTAT   •  'Best of'   •  Gallery   •  Gear   •  Reviews   •  Member list   •  Polls   •  Image rules   •  Search   •  Password reset

Not a member yet?
Register to forums
Registered members may log in to forums and access all the features: full search, image upload, follow forums, own gear list and ratings, likes, more forums, private messaging, thread follow, notifications, own gallery, all settings, view hosted photos, own reviews, see more and do more... and all is free. Don't be a stranger - register now and start posting!


COOKIES DISCLAIMER: This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and to our privacy policy.
Privacy policy and cookie usage info.


POWERED BY AMASS forum software 2.1forum software
version 2.1 /
code and design
by Pekka Saarinen ©
for photography-on-the.net

Latest registered member is rono258
852 guests, 222 members online
Simultaneous users record so far is 15144, that happened on Nov 22, 2018

Photography-on-the.net Digital Photography Forums is the website for photographers and all who love great photos, camera and post processing techniques, gear talk, discussion and sharing. Professionals, hobbyists, newbies and those who don't even own a camera -- all are welcome regardless of skill, favourite brand, gear, gender or age. Registering and usage is free.