Lightroom Catalog Question: Separate or Combined? How Big?

OK let me explain my question.

Like many of you, my collection of images is getting quite large. I have a catalog for personal images and one for clients. I've always been on the fence as to whether this was the best way to organize images.

It's nice that they are separate but an example of when it's not great would be this...

I have collections with rules established so I can quickly see images shot with a particular lens. I set up the rules in my personal image catalog. If I want to see the same type of collections in my client image library I have to export the rules from personal and import them to clients. It works but it also doesn't let me see them all together.

I still wonder whether have separate catalogs makes sense.

Another reason why I create separate catalogs was because I had read that once a catalog gets really huge it can slow down and become problematic.

I can understand that but I wonder if that's really true and at what point you need to be concerned.

I backup my images and my catalogs but I also really like to go back to older images or rework a client image or just browse and enjoy the images and not only is it a pain to have to close one catalog to open another but if I were to only have backups it would be a pain in the neck to go back and look at old images, so ideally I just want a library that always contains everything without the need to remove images or clean house.

How do others handle these issues? I'd love to hear some solutions or gain some insight on this topic.

Rob,

How much of a difference in performance you will see depends on your system, pretty much. My best advice would be to create a new catalog, import some folders, and try things out compared to your main catalog. If it makes a big difference, then, well, you have that info. Just know that to switch catalogs you will restart LR -- to switch back to another catalog you will again need to restart.

tonylong wrote in post #8872736
Rob,

How much of a difference in performance you will see depends on your system, pretty much. My best advice would be to create a new catalog, import some folders, and try things out compared to your main catalog. If it makes a big difference, then, well, you have that info. Just know that to switch catalogs you will restart LR -- to switch back to another catalog you will again need to restart.

Hi Tony,

I'm not really worried about speed or performance at this stage of the game. I was just curious at what point the problem supposedly kicks in.

I know that to switch catalogs makes LR restart and I don't really care for that and wanted to know how other handle this from a convenience standpoint. Do you keep personal and client images separate? Do you keep everything in one giant catalog and let the catalog grow or do you remove images or archive and eliminate what is not current.

This is a question that's a combination of technical and personal preference.

The catalogs are a database. Databases can become corrupt. The larger they are the most resources they require to keep up to speed.

TO keep things speedy and to minimize my risk of corruption, I keep my catalogs seperate per event type. My main catalog is Weddings, I have a Personal catalog, an Events, catalog, a commercial catalog, etc...

Another thing to consider is that if you ever have to move that catalog to another compute, you have to move all of the preview data as well, all those thumbnails. If everything is in one catalog, that file can be huge.

cory1848 wrote in post #8872980
The catalogs are a database. Databases can become corrupt. The larger they are the most resources they require to keep up to speed.

TO keep things speedy and to minimize my risk of corruption, I keep my catalogs seperate per event type. My main catalog is Weddings, I have a Personal catalog, an Events, catalog, a commercial catalog, etc...

Another thing to consider is that if you ever have to move that catalog to another compute, you have to move all of the preview data as well, all those thumbnails. If everything is in one catalog, that file can be huge.

Thanks Cory. That makes sense but it leaves me wondering why it's not like other database applications that let you open multiple databases at one time or give you the ability to search or create collections across databases.

So it seems there is no way to look at all the images shot with a particular lens in multiple databases without opening one catalog at a time. Correct?

TMR Design wrote in post #8873106
Thanks Cory. That makes sense but it leaves me wondering why it's not like other database applications that let you open multiple databases at one time or give you the ability to search or create collections across databases.

So it seems there is no way to look at all the images shot with a particular lens in multiple databases without opening one catalog at a time. Correct?

That would be correct -- I suspect a characteristic of the MySQL database design, which isn't as functional as top-tier databases.

tonylong wrote in post #8873122
That would be correct -- I suspect a characteristic of the MySQL database design, which isn't as functional as top-tier databases.

huh You're saying Facebook and Flickr run on a second-tier database?

That makes no sense and absolutely defeats the purpose of having a database.

I know several photographers with hundreds of thousands of images in their catalogs and Lightroom is still running well. The key is to regularly optimize your catalog and to test the integrity of the catalog to spot any problems early on.

cory1848 wrote in post #8872980
The catalogs are a database. Databases can become corrupt. The larger they are the most resources they require to keep up to speed.

TO keep things speedy and to minimize my risk of corruption, I keep my catalogs seperate per event type. My main catalog is Weddings, I have a Personal catalog, an Events, catalog, a commercial catalog, etc...

Another thing to consider is that if you ever have to move that catalog to another compute, you have to move all of the preview data as well, all those thumbnails. If everything is in one catalog, that file can be huge.

PixelMagic wrote in post #8873333
That makes no sense and absolutely defeats the purpose of having a database.

I know several photographers with hundreds of thousands of images in their catalogs and Lightroom is still running well. The key is to regularly optimize your catalog and to test the integrity of the catalog to spot any problems early on.

SO you're saying that as long as you backup, optimize and test the integrity of the catalog that I should be able to maintain a very large catalog with all my images, data, collections, etc.?

In your original post you said: "Another reason why I create separate catalogs was because I had read that once a catalog gets really huge it can slow down and become problematic."

Have you actually experienced this or are you relying on anecdotal information?

There are very practical reasons for keeping multiple catalogs but large catalogs slowing down is not one of them.

What many photographers I know do is keep a large master catalog with all their photos and several smaller catalogs related to specific jobs. As these jobs are finalized they export the smaller catalog into the master database. That strategy addresses any possible performance issues with large catalogs while maintaining a single searchable database.

TMR Design wrote in post #8873370
SO you're saying that as long as you backup, optimize and test the integrity of the catalog that I should be able to maintain a very large catalog with all my images, data, collections, etc.?

Yes.

TheHoff wrote in post #8873211
huh You're saying Facebook and Flickr run on a second-tier database?

Umm, I dunno, do they use MySQL?

I admit, it's been quite a while since I did any database development work, and MySQL was in its early stages and I never considered it for anything serious.

PixelMagic wrote in post #8873427
What many photographers I know do is keep a large master catalog with all their photos and several smaller catalogs related to specific jobs. As these jobs are finalized they export the smaller catalog into the master database. That strategy addresses any possible performance issues with large catalogs while maintaining a single searchable database.

Now that sounds pretty nifty!

PixelMagic wrote in post #8873427
Have you actually experienced this or are you relying on anecdotal information?

many photographers... keep a large master catalog with all their photos and several smaller catalogs related to specific jobs. As these jobs are finalized they export the smaller catalog into the master database. That strategy addresses any possible performance issues with large catalogs while maintaining a single searchable database.

Was about to post nearly the same exact thing!

Remember that if you only keep seperate catalogs without merging you pretty much lose any benefit of keywording.

Not sure, but I think the notes released with LR3 beta mentioned better support for larger catalogs... there obviously is an issue for some; and if you see that issue it makes sense to split. But quite possibly this point might be moot if Adobe does make the improvements they're intending to.

tonylong wrote in post #8873480
Umm, I dunno, do they use MySQL?

I admit, it's been quite a while since I did any database development work, and MySQL was in its early stages and I never considered it for anything serious.

Heh yea they both do. Facebook has even started plugging code back into the MySQL codebase that helps with concurrency... they are one of the very biggest and busiest sites in the world, right? Largest photosharing app on the planet... run on MySQL. And Flickr, too.

MySQL is quite robust now and since 4.x with query-caching it has been nearly the equal in performance to Oracle or any other db you put it up against. It still lacks some features like transaction-level locking in the MyISAM table format but the InnoDB tables have it... now it has stored procedures, views, and just about everything else.

and touche'... I didn't know Lightroom was using MySQL

PixelMagic wrote in post #8873427
In your original post you said: "Another reason why I create separate catalogs was because I had read that once a catalog gets really huge it can slow down and become problematic."

Have you actually experienced this or are you relying on anecdotal information?

There are very practical reasons for keeping multiple catalogs but large catalogs slowing down is not one of them.

What many photographers I know do is keep a large master catalog with all their photos and several smaller catalogs related to specific jobs. As these jobs are finalized they export the smaller catalog into the master database. That strategy addresses any possible performance issues with large catalogs while maintaining a single searchable database.
Yes.

I believe it was information provided by Adobe that led me to believe that there was some safe limit to the size of a catalog. Perhaps that was on earlier versions but that was the source.

I haven't actually experienced any slowdown in the catalog that isn't directly related to a computer slowdown due to lack of horsepower and RAMM. When those slowdowns occur it is system wide and not only in Lightroom so I can't say that I've seen any indication of slowdowns due to catalog size. Then again my largest catalog is only about 25,000 images so I don't know if that is considered to be large or not.

I like the idea of having separate catalogs and then a master catalog containing everything. How exactly do you go about doing that?

Right now I'm importing images to their respective catalog. What is the process of creating a large master catalog without duplicating the images? I assume that's what is done and there is only one copy of the image with only the data and location being copied or imported to the new catalog. Is that right?

TheHoff wrote in post #8873568
Heh yea they both do. Facebook has even started plugging code back into the MySQL codebase that helps with concurrency... they are one of the very biggest and busiest sites in the world, right? Largest photosharing app on the planet... run on MySQL. And Flickr, too.

MySQL is quite robust now and since 4.x with query-caching it has been nearly the equal in performance to Oracle or any other db you put it up against. It still lacks some features like transaction-level locking in the MyISAM table format but the InnoDB tables have it... now it has stored procedures, views, and just about everything else.

Well then, it sounds like LR is simply not utilizing the various features of MySQL, since those sites oviously enable things like multiple users and crossreferencing catalogs and whatnot. I suppose they figure now it's modeled only for individual photograhers, but I be there are plenty of folks who wish otherwise.