Every once in a while (a few times a day), the forums will appear to "hang" for 30 seconds or so. Sometimes, after it does so, it'll come back with a message saying the server is too busy. The responsiveness of the forums is usually excellent right before that happens, and a minute or so later it will appear to have recovered fully.
Here's where it gets technical and speculative.
The forum software is vBulletin. It uses MySQL as a backend.
MySQL has a number of table types it can use, the most prominent being MyISAM and InnoDB.
MyISAM is the default table type. It's also the one that's been around the longest. But most importantly, from a database perspective, it's also the one that has the most limitations.
MyISAM isn't very friendly in an environment involving a lot of concurrent reads and writes. It's friendly enough when all that's happening are reads with an occasional write thrown into the mix. But it tends to fall over when writes are thrown in. And it has no clue about transactions (which means that if an error occurs when writing to one of the tables, the client software, vBulletin in this case, has to go back to the tables it has already written to and undo whatever it did).
When a write occurs on a MyISAM table (say, for instance, when you post a message), MySQL will acquire a "full table lock" on the table it's currently writing to. The reason it does so is to prevent other readers and writers from reading inconsistent data from the table or, worse, writing conflicting data into the table such that the table would get corrupt. The lock prevents that kind of corruption. But it exacts a very heavy price on performance.
The lock is held for the duration of the write. If there's a lot of data to write to the table, the table will be inaccessible to everyone who is trying to read from it or write to it until the lock is released.
When the forum "locks up" this way and eventually comes back saying the server is "too busy", I believe it's because of the one or more full table locks that have been taken by the database.
InnoDB doesn't have any of these limitations. When a write occurs, the change is recorded in a transaction log and a new version of each of the rows involved is created. Old versions are eventually scrubbed, but won't be as long as there's a running transaction that the database knows has seen them. The end result is that when a write occurs, readers aren't blocked, only other writers, and then only for the rows that they are updating, not the entire table.
And so, after all this, the question of the day is: is the InnoDB table type being used for this forum? If it's not, can it be (it may require migration)? MySQL-based sites such as slashdot.org are using it to very good effect. I would be willing to donate some of my time towards a migration effort if it's desired.
