Well, things seem sort of normal again now. There is no definitive
answer as to what was going on. It seems likely to have had something
to do with the apc cache.
Brion has a very safe looking script to clear the apc cache, I see
nothing wrong with running it. But it seemed "stuck", and the site
was very unhappy, and when he kill -9'd it, the site seemed better.
But at roughly the same time, he disabled apc caching completely,
which is probably a negative for performance to some extent.
So, we're running again, but I'm not at all confident that we understand
the problem.
answer as to what was going on. It seems likely to have had something
to do with the apc cache.
Brion has a very safe looking script to clear the apc cache, I see
nothing wrong with running it. But it seemed "stuck", and the site
was very unhappy, and when he kill -9'd it, the site seemed better.
But at roughly the same time, he disabled apc caching completely,
which is probably a negative for performance to some extent.
So, we're running again, but I'm not at all confident that we understand
the problem.