Lightroom - How are you managing a large library?

I can't deal with the never ending Adobe bashing that seems to be the focus of almost everything in the Adobe forums, and there don't seem to be a lot of other places to talk about Lightroom. I believe that with Adobe's market precence, the open architecture of Lightroom, the low price, and the great UI, that Lightroom is going to rapidly become the defactor workflow hub, just as PS CS2 has become the defacto image editor.

So I'd like to get a good long discussion going on how to manage large libraries. I have in excess of 100K images that I need to manage, and Lightroom 1.0 starts to breakdown as you approach 20K images... in fact it seems to like staying closer to 10K. I have no doubt that this will improve in time, but I'd like to get going now and make some decisions that don't result in massive amounts of rework in the future.

My images breakdown into several major categories...

Professional
- Action Sports
- Portrait
- Commercial
Personal
- Family
- Wildlife
- Transportation
- Other

One could say that I could just create separate databases for each of these categories, but unfortunately they are not linear in size. The first two categories are by far the largest and easily exceed the size I'd like to keep my libraries at... even within a single year. So that doesn't work.

I'm trying to optimize my ability to find images when I want to retrieve them later. One of the problems I currently have is that I'm a pack rat... I keep everything. I never delete any image that makes its way into my office. I do delete a lot in the camera, but once I import it, I don't ever delete. I figure I may want to have an example of a blurry image to show someone one day, and so I'll keep them. This is stupid, and a huge waste of space, but its the reality with my library at present. But that said, I can end up with over 10K images from one event.

One of the ideas I had was the following. I could load everything for action sports into one library, then rank everything... say 1-5 stars. Initially I could delete all the 1s out of the gate. Then after a month, I could go back and delete the 2s and 3s. Maybe after a year delete the 4s. Of course as I delete these, they are still on my NAS box so that I really haven't gotten rid of them, only gotten rid of the database entries. Over time, this would slowly build a better and better library of sports images that are retained, while also keeping the newer full sets of images. I am assuming that exporting parts of a library will eventually be added back, and if so, then I could at some point split it by type of sport if it got too big... say pull all indoor sports out and leave outdoor... then eventually pull all of one sort out if it got too big. I honestly can't imaging having 20K 5 star images ever..., so I don't think that would ever be a problem.

Then there is a side of me that says I should just do this scheme now with all my images and go with one library.

So I'll now stop the rant and ask for input, opion, ideas? What are you guys doing to deal with this.

rating is good, so is color flagging, but keywording is key. that way you can actually go back and find things by searching instead of trying to remember. A heiracy of keywords is essential for any DAM, and its pretty easy to do with LR. I would suggest keywording as much as you can at import to save time.

I just keep one big library
Personal
Travel
Portraiture
Weddings
etc
and have a heirarchy underneigth each section for actual folders.

Collections is useful to, keep a portfolio together, favorite wildlife images, images a certain client wanted etc

coreypolis wrote in post #2946121
rating is good, so is color flagging, but keywording is key. that way you can actually go back and find things by searching instead of trying to remember. A heiracy of keywords is essential for any DAM, and its pretty easy to do with LR. I would suggest keywording as much as you can at import to save time.

I just keep one big library
Personal
Travel
Portraiture
Weddings
etc
and have a heirarchy underneigth each section for actual folders.

Collections is useful to, keep a portfolio together, favorite wildlife images, images a certain client wanted etc

Well I didn't bring up keywords, because I assumed that everyone would use keywords... as you said its what makes Lightroom so powerful (or any other DAM system). But keywords doesn't help with the subject of the thread. The question was how to manage a large library. I can't just stick them all in one library because its over 100K images and the thing would die a quick an painful death.

convergent wrote in post #2946150
Well I didn't bring up keywords, because I assumed that everyone would use keywords... as you said its what makes Lightroom so powerful (or any other DAM system). But keywords doesn't help with the subject of the thread. The question was how to manage a large library. I can't just stick them all in one library because its over 100K images and the thing would die a quick an painful death.

what makes you think that? Its been tested with over that and it works just fine. I wouldn't import all 100k + at once, but a little at a section at a time will work just fine. The actual size of the library will be huge though, so be prepared for that.

If you are looking to break it up, well then its pretty arbitrary, there is no right answer.

Another options is to import, then rate or what not and remove those images from the library but keep them on disk if you are a true pac rat

The latest Photovision DVD had a review of it, they recommend many smaller libraries. I wish I could (and there might be a way to) have smaller libraries but still have all of them linked and LR was smart enough to open and close as needed.

I have a large library sorted by folders, probably about 60gb worth of images and its quite speedy.

I'm pretty sure that in a future (soon hopefully) revision that we wil be able to merge, link or use multiple databases at the same time.

I'm also pretty sure that the resource (hog) issues with LR on a PC will be sorted out in a future (soon hopefully) release.

I really like LR and have about 10,000 images in my DB.

coreypolis wrote in post #2946158
what makes you think that? Its been tested with over that and it works just fine. I wouldn't import all 100k + at once, but a little at a section at a time will work just fine. The actual size of the library will be huge though, so be prepared for that.

If you are looking to break it up, well then its pretty arbitrary, there is no right answer.

Another options is to import, then rate or what not and remove those images from the library but keep them on disk if you are a true pac rat

What makes me think that is my own experience, as well as looking at the countless replies on other forums by folks that report performance falling off after you start to go over 20K images. If you haven't had that problem, I'm impressed. I think my system is pretty capable, so there isn't really any upgrade I can do to help. Keep in mind that 100% of my images are stored on a RAID5 NAS box, accessed over a GigaE network (Infrant ReadyNAS NV+). I also have the database on a semi-dedicated internal SATA drive (I also use it for Photoshop scratch disk, Photo Mechanic cache, etc.)

I do realize there is no one right way to do this... I was looking for what other people were doing, and for suggestions.

Convergent:

I saw you already commented in this thread:
https://photography-on-the.net/forum/showthre​ad.php?t=296528

Did that help you any?

BTW: I'm glad to see I'm not the only one who feels that there's something wrong with the Adobe forums... its not just that some people are bashing Adobe... its that when someone makes a simple comment or a feature request, they get sent through the ringer... Even when a newbie enters and asks a seemingly simple question they are made to feel like an idiot many times.

I'm glad to say I find this forum is much kinder and helpful IMO... even when I do ask stupid questions, or make comments without reading initial post thoroughly.

Well, with regard to the Adobe forum... it baffles me that companies with their capabilities still run these lame forum scripts that are difficult to work with. I run two forums with vBulletin and it is like someone tied one hand behind my back when I try to use the crappy ones that most big companies choose. Given that, the unbelievable bash-fest that goes on there makes it painful to even look there for anything... and their search is useless. Some guy asked a simple question... newbie on day one of use... and these two idiots went on a tyrant rampage about how the product stunk and Adobe was trying rip off everyone... completely ridiculous.

Anyways, back to the topic. I've not tried the compression yet... just not had time. Even if it works, as I'm sure it will, I'll still have way more images than can comfortably fit in one database at this point. I'm OK with that, if I can find a reasonable way to split things up... and I've got some ideas now.

Hopefully by next release of Lightroom, they'll have the database compression built in.

As a side note, Lightroom was one of the things that drove me to order a new 30" LCD from Dell today... can't wait to marry them up.

sorry to bring up an old thread, but how are people dealing with their huge catalogs? I'm asking more out of curiosity, as I don't see the point in having multiple catalogs myself. I use Aperture/Lightroom for indexing and editing basically. I never did like iview, and when Aperture came out, I jumped at it (albeit a bit too soon).

It comes down to performance... large catalogs bog down performance. I split my CATs by genre and then year. So I have Motorcycle Racing 2007 images, Motorcycle Racing 2006 images, Motorcycle Racing 2005 images, etcetera. I also maintain a master portfolio Catalog that contains all of my best images.

rcg wrote in post #4595778
It comes down to performance... large catalogs bog down performance. I split my CATs by genre and then year. So I have Motorcycle Racing 2007 images, Motorcycle Racing 2006 images, Motorcycle Racing 2005 images, etcetera. I also maintain a master portfolio Catalog that contains all of my best images.

Yeah, that's what I figured. Just curious if there's been any new development from v1.0->1.3.1.

I don't need all my images there, but ideally a thumbnail, keywords, and where it is/filename would be ideal. I've spent a lot of time cataloging, and want it all searchable.

Of course, yet another program might be better suited to do this, but I was looking for more all in one, so it'd be easier to flip between using OS X and Windows if needed... if that makes any sense, heh

How large are you guys talking?

Using the latest version of LR, I've got about 9,000 images in mine and see about the same performance as when I use a catalog with only 500.

(This is using a Core2duo 2.0ghz, with 2gb ram on Vista.)

If you are seeing performance issues, have you tried going to catalog settings and using relaunch and optimize?

I'm thinking long term over a few years... and I was just curious for the heavy users who have 20k+ in there.

My total cataloged images (all catalogs combined) is 116,000 images. Keeping all of those in one catalog is asking for trouble.

My Motorcycle 2007 cat. is 14,602 images.