Mailing List Archive: Raised number of max connections

Raised number of max connections

vibber at aludra

Apr 23, 2003, 7:58 AM

Post #1 of 8 (306 views)

I've been getting a lot of timeouts lately. A fair chunk of connections
are from the 'grub' distributed search engine spider, which connects too
often and doesn't play with robots.txt as nicely as I'd like. Some time
ago I'd put them in the 403 rejection list but, they don't get the hint
and keep on trying to connect.

This doesn't touch the database, but does eat up some apache connections.
Although I have now explicitly banned grub in robots.txt and filled out
their little update form, they're still connecting -- even to the banned-
for-all /w subdirectory. It's pissing me off.

I've upped the max connections on apache from 175 to 260, and on mysql
from 400 to 560. Lee or Magnus; if we get mysql too-many-connections
errors, either bring down the apache limit or move up the mysql limit.

-- brion vibber (brion @ pobox.com)

Re: Raised number of max connections [ In reply to ]

nick at twoevils

Apr 23, 2003, 8:06 AM

Post #2 of 8 (306 views)

On Wed, Apr 23, 2003 at 07:58:27AM -0700, Brion Vibber wrote:
> I've been getting a lot of timeouts lately. A fair chunk of connections
> are from the 'grub' distributed search engine spider, which connects too
> often and doesn't play with robots.txt as nicely as I'd like. Some time
> ago I'd put them in the 403 rejection list but, they don't get the hint
> and keep on trying to connect.
>
> This doesn't touch the database, but does eat up some apache connections.
> Although I have now explicitly banned grub in robots.txt and filled out
> their little update form, they're still connecting -- even to the banned-
> for-all /w subdirectory. It's pissing me off.
>
> I've upped the max connections on apache from 175 to 260, and on mysql
> from 400 to 560. Lee or Magnus; if we get mysql too-many-connections
> errors, either bring down the apache limit or move up the mysql limit.

Perhaps it would be best to put in an iptables rule to deny a TCP/IP
connection from their crawler (assuming it comes from a reasonably
limited set of IPs)?

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN

Re: Raised number of max connections [ In reply to ]

vibber at aludra

Apr 23, 2003, 8:13 AM

Post #3 of 8 (306 views)

On Wed, 23 Apr 2003, Nick Reinking wrote:
> Perhaps it would be best to put in an iptables rule to deny a TCP/IP
> connection from their crawler (assuming it comes from a reasonably
> limited set of IPs)?

No, see, it's a *distributed* crawler. So I don't know where the fuck
they're coming from. Like I said, it's really pissing me off.

See also:
http://www.grub.org/forums/viewtopic.php?t=124

-- brion vibber (brion @ pobox.com)

Re: Raised number of max connections [ In reply to ]

jwales at bomis

Apr 23, 2003, 8:32 AM

Post #4 of 8 (306 views)

I hadn't heard about this Grub project, but I'm not impressed. I'd be
impressed if they made the resulting data available via a free
license.

--Jimbo

Re: Raised number of max connections [ In reply to ]

nick at twoevils

Apr 23, 2003, 8:42 AM

Post #5 of 8 (310 views)

On Wed, Apr 23, 2003 at 08:32:49AM -0700, Jimmy Wales wrote:
> I hadn't heard about this Grub project, but I'm not impressed. I'd be
> impressed if they made the resulting data available via a free
> license.
>
> --Jimbo

I agree. I'm not impressed either way - the attitude of the developers
is, quite frankly, obnoxious.

--
Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN

Re: Raised number of max connections [ In reply to ]

Apr 23, 2003, 10:13 AM

Post #6 of 8 (304 views)

> > I hadn't heard about this Grub project, but I'm not impressed. I'd be
> > impressed if they made the resulting data available via a free
> > license.
> >
> > --Jimbo
>
> I agree. I'm not impressed either way - the attitude of the developers
> is, quite frankly, obnoxious.

They've gotten plenty of press lately (Slashdot, New Scientist, et al.)
but I have to say I'm not terribly impressed with their operation
either. Basically, they've got an interesting idea, but a lousy
execution, and not muc of a clear vision.

Maybe submitting a Slashdot story about their problems would be a
kick in the pants?

--
Lee Daniel Crocker <lee@piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC

Re: Raised number of max connections [ In reply to ]

jwales at bomis

Apr 23, 2003, 6:27 PM

Post #7 of 8 (304 views)

Kurt Jansson wrote:
> Lee Daniel Crocker schrieb:
> > Maybe submitting a Slashdot story about their problems would be a
> > kick in the pants?
> Or telling them that we're regarding it as a DDOS attac and sue them if
> they don't stop? (Just threaten to do it, of course.)

Worse than that, we'll write a NPOV article about their problems and
post it in the most authoritative encyclopedia on the net, Wikipedia.
Heh, that'll get 'em.

--Jimbo

Re: Raised number of max connections [ In reply to ]

Apr 23, 2003, 7:27 PM

Post #8 of 8 (305 views)

Lee Daniel Crocker schrieb:

> Maybe submitting a Slashdot story about their problems would be a
> kick in the pants?

Or telling them that we're regarding it as a DDOS attac and sue them if
they don't stop? (Just threaten to do it, of course.)

Kurt