Mailing List Archive

Cause for slowdowns: spiders?
Right now, I'm seeing nice and fast responses, except every
once in a while everything slows to a halt. If that's due to our
script, then there must be some really bad, really rare special
function somewhere. I doubt that.

Maybe the slowdowns are due to spiders that hit our site and request
several pages at once, in parallel, like many of these multithreaded
programs do. I read somewhere that everything2.com for this very
reason has disallowed spiders completely and doesn't even allow Google
to index their site anymore.

Maybe we should search the server logs for several rapid requests from
the same IP, and try to correlate those to load averages?

Axel
Re: Cause for slowdowns: spiders? [ In reply to ]
Axel Boldt <axel@uni-paderborn.de> writes:

> Right now, I'm seeing nice and fast responses, except every
> once in a while everything slows to a halt. If that's due to our
> script, then there must be some really bad, really rare special
> function somewhere. I doubt that.

Its a punt, and I've no real evidence ... but ...
when the 'pedia comes back after some slowtime, there often seems to be an

Wikipedia:Unsuccessful searches (date); [Unsuccessful search for foobar]

at the top of RecentChanges

I might be imagining it, I certainly can't reproduce it at will.
--
Gareth Owen
Re: Cause for slowdowns: spiders? [ In reply to ]
On Thursday 16 May 2002 19:43, Axel Boldt wrote:
> Right now, I'm seeing nice and fast responses, except every
> once in a while everything slows to a halt. If that's due to our
> script, then there must be some really bad, really rare special
> function somewhere. I doubt that.

How long do slowdowns last? I've been trying for at least half an hour to
submit a change to a page. I get nothing but timeouts. Sometimes I can't even
get a connect.

phma
Re: Cause for slowdowns: spiders? [ In reply to ]
On Fri, May 17, 2002 at 08:25:19AM -0400, Pierre Abbat wrote:
> On Thursday 16 May 2002 19:43, Axel Boldt wrote:
> > Right now, I'm seeing nice and fast responses, except every
> > once in a while everything slows to a halt. If that's due to our
> > script, then there must be some really bad, really rare special
> > function somewhere. I doubt that.

I'm not following this closely so maybe this is a stupid remark but I know
that Brion disabled his code that did LIKE queries when MATCH gave empty
results. Has this change already become effective for the actual Wikipedia?

-- Jan Hidders
Re: Cause for slowdowns: spiders? [ In reply to ]
On Friday 17 May 2002 08:25, Pierre Abbat wrote:
> How long do slowdowns last? I've been trying for at least half an hour to
> submit a change to a page. I get nothing but timeouts. Sometimes I can't
> even get a connect.

Okay, I got a time on this. It happened between 4:16 and 5:48 server time.
There were no unsuccessful searches between that time.

phma