Mailing List Archive

Last time, Rob Hartill uttered the following other thing:
>
>
> The latest fork-free patch is in /patches/for_Apache_0.6.2/
>
> Things we need to fix...
>
> There's a potential problem in the current setup, which I think
> the NCSA approach will suffer from too, it should be possible for
> a trouble maker to open N connections, and grab the attention of
> all N child processes. He can then hold these N connections open
> for TIMEOUT seconds, and thus block all other connections to the
> server.
>
> My first thought was, "Aha, reduce the TIMEOUT for receiving
> a request to something more realistice - say 20 seconds"
> ...but that's going to cause problems for people POSTing large
> amounts of information over slow links.
>
> The only real soultion I could come up with was to have a forking
> model as a fallback.

This is exactly what 1.4 does. If all of the children are in use, and
there are MaxServers of them, then it will fall back to 1.3 type behavior.

> The children can send signals (ALARM or whatever) to the parent to
> say "ready" or "busy". The parent only needs to know how many of
> the children are "ready", so that in the event of all of them being
> "busy", it can then spawn the forking model.

We just use an array to mark a child as "Busy" and when the child is done,
it sends a DONE message to the parent who then clears the busy flag.

> All this is based on the paranoia that someone will think it'll
> be a hoot to clog someone else's server. If I though of it, I'm
> sure someone else will too.

The main reason we added this was because of alpha tests on hoohoo and
www.acm.uiuc.edu. On hoohoo we have a really bad archie script that almost
never returns as it waits for the archie request to finish. This will tie
up the child until TIMEOUT usually. The problem with www.acm was Rob's
Multimedia Lab, which contains muli-megabyte files. During peak times,
30-40 connections would be likely, because of time to service was so high.

Brandon

--
Brandon Long (N9WUC) "I think, therefore, I am confused." -- RAW
Computer Engineering Run Linux 1.1.xxx It's that Easy.
University of Illinois blong@uiuc.edu http://www.uiuc.edu/ph/www/blong
Don't worry, these aren't even my views.

On Sat, 22 Apr 1995, Rob Hartill wrote:
> Things we need to fix...
>
> There's a potential problem in the current setup, which I think
> the NCSA approach will suffer from too, it should be possible for
> a trouble maker to open N connections, and grab the attention of
> all N child processes. He can then hold these N connections open
> for TIMEOUT seconds, and thus block all other connections to the
> server.

Rob-

There's no real way to stop anyone who's determined to commit a denial of
service attack against a TCP server... all they need to do is send a
stream of SYN packets with bogus source addresses, and never complete the
handshake. It isn't worth going to extreme lengths to try and stop
deliberate attacks; however, it might be worth putting in a forking
fail-over model to cope with those times when MCInet decides to stop
routing packets so they can shoot another commercial.

I can't remember whether the load-average code made it into the mainline
release; if it did, the best approach might be to only fork if the
load-average is below a given level.

Simon // This Message Composed By Voice

From: Rob Hartill <hartill@ooo.lanl.gov>
Date: Sat, 22 Apr 95 12:30:49 MDT

The latest fork-free patch is in /patches/for_Apache_0.6.2/

Things we need to fix...

There's a potential problem in the current setup, which I think
the NCSA approach will suffer from too, it should be possible for
a trouble maker to open N connections, and grab the attention of
all N child processes. He can then hold these N connections open
for TIMEOUT seconds, and thus block all other connections to the
server.

I wouldn't be so worried about deliberate denial of service attacks;
if an attacker wants to take down any current server (or a non-forking
server with a forking fallback state, for that matter), they can
always just bombard it with connections faster than it can keep up,
leading to a proliferation of processes which spend all of their time
fighting over the CPU, and none of it getting anything done. (When
the load average is 400, no one is getting served effectively).

The real problem is situations where this may occur by accident, some
of which Brandon outlined. Unfortunately, the forking fallback
position has problems of its own --- basically, when NCSA 1.4, or a
like-constructed server, is under heavy load, it actually gets slower.
This can lead to catastrophic failure when the load has sudden peaks
against a background of steadily high activity (which I saw a number
of times while testing my 1.4/Apache mix under load); note that real
Web traffic does turn out to be fairly peakish in practice (the AI
lab's peak load is roughly 10,000 transactions/hr, at about 4:00 PM
EDT, which is < 3 tr/sec, but we see peaks of up to 12-13 tr/sec when
we histogram transactions/calendar second on a daily basis).

rst

From: Rob Hartill <hartill@ooo.lanl.gov>
Date: Sat, 22 Apr 95 12:30:49 MDT

The latest fork-free patch is in /patches/for_Apache_0.6.2/

Things we need to fix...

There's a potential problem in the current setup, which I think
the NCSA approach will suffer from too, it should be possible for
a trouble maker to open N connections, and grab the attention of
all N child processes. He can then hold these N connections open
for TIMEOUT seconds, and thus block all other connections to the
server.

I wouldn't be so worried about deliberate denial of service attacks;
if an attacker wants to take down any current server (or a non-forking
server with a forking fallback state, for that matter), they can
always just bombard it with connections faster than it can keep up,
leading to a proliferation of processes which spend all of their time
fighting over the CPU, and none of it getting anything done. (When
the load average is 400, no one is getting served effectively).

The real problem is situations where this may occur by accident, some
of which Brandon outlined. Unfortunately, the forking fallback
position has problems of its own --- basically, when NCSA 1.4, or a
like-constructed server, is under heavy load, it actually gets slower.
This can lead to catastrophic failure when the load has sudden peaks
against a background of steadily high activity (which I saw a number
of times while testing my 1.4/Apache mix under load); note that real
Web traffic does turn out to be fairly peakish in practice (the AI
lab's peak load is roughly 10,000 transactions/hr, at about 4:00 PM
EDT, which is < 3 tr/sec, but we see peaks of up to 12-13 tr/sec when
we histogram transactions/calendar second on a daily basis).

rst

On Sat, 22 Apr 1995, Rob Hartill wrote:
> The latest fork-free patch is in /patches/for_Apache_0.6.2/
>
> Things we need to fix...
>
> There's a potential problem in the current setup, which I think
> the NCSA approach will suffer from too, it should be possible for
> a trouble maker to open N connections, and grab the attention of
> all N child processes. He can then hold these N connections open
> for TIMEOUT seconds, and thus block all other connections to the
> server.

At the very least, if all children are busy, there should be at least one
more process listening on the port that sends a 503 response.

My first thought was to try and devise some algorithm that checked for
attacks like this by seeing if the requests were coming from the same
host or something, but that could be defeated without too much trouble.

Ugh, Simon's probably right in that every algorithm we could devise could
be defeated by someone who also knew the algorithm. It seems this is
much more a TCP/IP issue than something we could control - maybe a
command the equivalent of "ftpwho" for FTP sites could be implemented.

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/

Last time, Brian Behlendorf uttered the following other thing:
>
> On Sat, 22 Apr 1995, Rob Hartill wrote:
> > The latest fork-free patch is in /patches/for_Apache_0.6.2/
> >
> > Things we need to fix...
> >
> > There's a potential problem in the current setup, which I think
> > the NCSA approach will suffer from too, it should be possible for
> > a trouble maker to open N connections, and grab the attention of
> > all N child processes. He can then hold these N connections open
> > for TIMEOUT seconds, and thus block all other connections to the
> > server.
>
> At the very least, if all children are busy, there should be at least one
> more process listening on the port that sends a 503 response.

The things we tried in alpha (in order):
have a special child that always returns a "we're busy" and hand
the connections over the limit to it
a queue for connections that have come in (and been accepted)
let the OS queue it by not accepting it
fallback to 1.3 behaviour

The first was a problem in general (scripts and the like could cause a
server to be perpetually busy without really being busy).
The second made response time unacceptable (and SunOS only allows 64
open file descriptors)
The third was no better than the second, plus this is one of the major
problems with the server in general (5 is the hard limit for the queue
on most systems (at least older systems))

Hence the reason for the fallback

My personal idea was to allow infinite number (or at least a large number)
of children be created, and then have them timeout and die after 5 minutes
if they haven't been used. This way a server "responds" to increased
demand, and yet relinquishes the resources when it slows down.
Unfortunately, I started coding in Unix about the same time I started
coding this, and didn't know how to do it, so the current version has
stuck. Maybe for 1.5 (2.0, 2000, whatever)

Brandon
--
Brandon Long (N9WUC) "I think, therefore, I am confused." -- RAW
Computer Engineering Run Linux 1.1.xxx It's that Easy.
University of Illinois blong@uiuc.edu http://www.uiuc.edu/ph/www/blong
Don't worry, these aren't even my views.

On Sun, 23 Apr 1995, Brandon Long wrote:

> The second made response time unacceptable (and SunOS only allows 64
> open file descriptors)

man getrlimit(2) for how to change this (but make sure you change the
limit back before you start something like sendmail, which tries to close
all 2^31 file descriptors before doing anything else.

Simon

In reply to Rob Hartill who said
>
>
> Having read about the potential for attacks at a lower level,
> I agree it's pointless trying to defend against it at the HTTP
> level.
>
> The other problem is, what if lots of slow connections cause a
> bottleneck.
>
> The NCSA approach is to start forking. This has the following
> pros and cons,
>
> pro: there will always be a process ready to accept a new connection
> cons: there will always be a process ready to accept a new connection
>
> If you sets up N child processes, you are effectively setting a
> maximum load you're willing to accept, and maybe that should be
> enough.
>

Won't this mean that when you use up your N connections the client will
simply fail to connect, with no explanation. I wouldn't like to run an
important http site where clients simply couldn't connect. It wouldn't
look good since there's no way of knowing from the client side that the
connection failed because the site is very busy, rather than just
unreliable or broken.

I'm not sure how to solve this, would there be a way to fork a limited
child that simply accepted the connection, sent some "we're busy" message
and then closed the connection. If so, you could have N children and when
these are all in use keep forking the limited children to answer further
requests until one of the N connections becomes free.

Hmm, if you get really clever, you could queue the pending connections
until one of the N connections close.

--
Paul Richards, FreeBSD core team member.
Internet: paul@FreeBSD.org, URL: http://isl.cf.ac.uk/~paul/
Phone: +44 1222 874000 x6646 (work), +44 1222 457651 (home)
Dept. Mechanical Engineering, University of Wales, College Cardiff.

Having read about the potential for attacks at a lower level,
I agree it's pointless trying to defend against it at the HTTP
level.

The other problem is, what if lots of slow connections cause a
bottleneck.

The NCSA approach is to start forking. This has the following
pros and cons,

pro: there will always be a process ready to accept a new connection
cons: there will always be a process ready to accept a new connection

If you sets up N child processes, you are effectively setting a
maximum load you're willing to accept, and maybe that should be
enough.

Brandon mentioned some archie script which would almost always
time-out, if so, that script should be rewritten or discarded as
being unsuitable for a HTTP server - isn't that what archive servers
and cleints are for ?

Other potential CGI bottlnecks are scripts which have a lot of work
to do. If, by means of a limited number of simultaneous active children,
these scripts hold up other connections, then this might not be a bad
thing after all. If you spawn more children, you just slow the scritps
down even futher.

For the moment, my +1 goes on the side of non-forking. Set up N children,
and if the load stays low even at peak times, then it's probably safe
to increase N a little, until you find a value of N which gives you the
maximum cpu load you are willing to live with.


robh

On Sun, 23 Apr 1995, Robert S. Thau wrote:
> I agree with this analysis for the moment. The only thing that gives me
> pause about it is that it may not be adequate for HTTP/1.1 (specifically,
> for persistent connections).

Hmm - my understanding was that the connection is only persistant for
some set collection of requests; i.e., fetching a page and its 7 inline
images would be 2 separate connections, and when the last inline image
came through then the connection would close until the user clicked
again. This is different from persistance in the FTP sense, and I don't
think it would mean the server processes would suffer; indeed, I think
the opposite.

I wish there was a browser we could play around with that did HTTP/1.1....

> The most straightforward way to implement
> these HTTP/1.1 keep-alives would be to allocate one member of the server
> pool to each client for the duration of its connection --- but this does
> mean that each client will be locking up a whole server process for a
> nontrivial amount of time (perhaps more than a minute for retrievals of
> pages with a *lot* of inlines --- latex2html output with gif-ized math,
> and Ellen Spertus' non-profit orgs page with 60-odd logos, both come to
> mind from around here).

We need a bake-off to compare that to one process making 60 requests in
terms of server efficiency and such - yes, effectively the requests can
be "interrupted" with a non-persistant connection, but to whose benefit?
Perhaps the server could have an option for a maximal amount of
persistant-connection requests, like say 10 (analogous to the number of
parallel connections Netscape lets you set).

Simon can probably talk about bake-off's with HTTP/1.1, as I think he has
implemented it (as well as someone at Digital). Or were those just
simulations?

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/

> Won't this mean that when you use up your N connections the client will
> simply fail to connect, with no explanation. I wouldn't like to run an
> important http site where clients simply couldn't connect. It wouldn't
> look good since there's no way of knowing from the client side that the
> connection failed because the site is very busy, rather than just
> unreliable or broken.


I don't think so. I think you'll just have to wait for a connection.
The client timeout setting will determine if you wait long enough.

I've hit 10 child processes from 25 cleints simultaneously without
ever failing to get a connection.

> New puzzling symptom ... I was running fork-free for a while, and noticed
> that the server was dropping some connections on the floor (accepting and
> then immediately closing them).

I've yet to see this in the >50,000+ connections I've hit it with...
but that's not to say it isn't happening for you.

I agree with this analysis for the moment. The only thing that gives me
pause about it is that it may not be adequate for HTTP/1.1 (specifically,
for persistent connections). The most straightforward way to implement
these HTTP/1.1 keep-alives would be to allocate one member of the server
pool to each client for the duration of its connection --- but this does
mean that each client will be locking up a whole server process for a
nontrivial amount of time (perhaps more than a minute for retrievals of
pages with a *lot* of inlines --- latex2html output with gif-ized math,
and Ellen Spertus' non-profit orgs page with 60-odd logos, both come to
mind from around here).

On the other hand, allocating a separate process for each such long-lived
connection might not be easy on swap space.

What would be best, if we can manage it, would be to somehow distribute
the individual requests coming in from a kept-alive connection around the
server pool... but that won't be easy (at least, not without creating an
NCSA-1.4-style bottleneck distributor process).

rst

New puzzling symptom ... I was running fork-free for a while, and noticed
that the server was dropping some connections on the floor (accepting and
then immediately closing them). The configuration was a bit of a witches'
brew, so I'll have to work on duplicating this later with Rob's exact code
(or not) --- in particular, I had a test nameserver cache in the mix ---
but it would be interesting for people to check to see if the same sort
of thing hits them...

(Sometime, I should do a paranoid webmonkey which checks the length of
the return, and reports discrepancies...).

rst

/*
* "Re: fork free" by Rob Hartill <hartill@ooo.lanl.gov>
* written Sun, 23 Apr 95 15:12:35 MDT
*
* Brandon mentioned some archie script which would almost always
* time-out, if so, that script should be rewritten or discarded as
* being unsuitable for a HTTP server - isn't that what archive
* servers and cleints are for ?
*
*/


AA - A /bin/sh and awk filter to
archie. http://hoohoo.ncsa.uiuc.edu/archie.html

It's a wonderful interface, much better than any of the others I'd
seen before. However, the fact that it's written as a Bourne shell
script is a huge loss. It's singlehandedly responsible for many of the
CGI brain death that made its way into 1.2 and 1.3. The bottom line is
that scripts like that should be Perl, C, or not written.

--Rob

Date: Sun, 23 Apr 1995 15:29:21 -0700 (PDT)
From: Brian Behlendorf <brian@organic.com>
Precedence: bulk
Reply-To: new-httpd@hyperreal.com

On Sun, 23 Apr 1995, Robert S. Thau wrote:
> I agree with this analysis for the moment. The only thing that gives me
> pause about it is that it may not be adequate for HTTP/1.1 (specifically,
> for persistent connections).

Hmm - my understanding was that the connection is only persistant for
some set collection of requests; i.e., fetching a page and its 7 inline
images would be 2 separate connections, and when the last inline image
came through then the connection would close until the user clicked
again. This is different from persistance in the FTP sense, and I don't
think it would mean the server processes would suffer; indeed, I think
the opposite.

That's probably right --- or at least, probably, the recommended way
to use the feature. However, there are pages with a *whole* lot of
inline images (Ellen's nonprofit orgs page has 60 gifs, some of
nontrivial size; latex2html output with -split 0 (everything into one
file) in a file with a lot of math can generate literally hundreds).

We need a bake-off to compare that to one process making 60 requests in
terms of server efficiency and such - yes, effectively the requests can
be "interrupted" with a non-persistant connection, but to whose benefit?
Perhaps the server could have an option for a maximal amount of
persistant-connection requests, like say 10 (analogous to the number of
parallel connections Netscape lets you set).

Another way to handle this might be to have the server drop any given
(multi-request) connection after completing transactions which took
more than a certain amount of wall-clock time. (That sort of policy
might yield more consistent results in cases where there are a lot of
pages with little GIFs, and a few with really large ones).

Of course, all of this assumes that a server can, in fact, drop the
connection after issuing its own first keep-alive...

rst

/*
* "Re: fork free" by Rob Hartill <hartill@ooo.lanl.gov>
* written Sun, 23 Apr 95 15:12:35 MDT
*
* Brandon mentioned some archie script which would almost always
* time-out, if so, that script should be rewritten or discarded as
* being unsuitable for a HTTP server - isn't that what archive
* servers and cleints are for ?
*
*/

Rob --- the probem here isn't with the script itself; it's with
Archie, which generally does just time out these days no matter how
you get to it...

rst

Last time, Rob McCool uttered the following other thing:
>
> /*
> * "Re: fork free" by Rob Hartill <hartill@ooo.lanl.gov>
> * written Sun, 23 Apr 95 15:12:35 MDT
> *
> * Brandon mentioned some archie script which would almost always
> * time-out, if so, that script should be rewritten or discarded as
> * being unsuitable for a HTTP server - isn't that what archive
> * servers and cleints are for ?
> *
> */
>
>
> AA - A /bin/sh and awk filter to
> archie. http://hoohoo.ncsa.uiuc.edu/archie.html
>
> It's a wonderful interface, much better than any of the others I'd
> seen before. However, the fact that it's written as a Bourne shell
> script is a huge loss. It's singlehandedly responsible for many of the
> CGI brain death that made its way into 1.2 and 1.3. The bottom line is
> that scripts like that should be Perl, C, or not written.

A great deal of the problem would also be fixed if we made it an nph
script (then it wouldn't hold up a httpd child). The only reason its
still around is that its useful for testing the server (under bad
cgi conditions )

Brandon

--
Brandon Long (N9WUC) "I think, therefore, I am confused." -- RAW
Computer Engineering Run Linux 1.1.xxx It's that Easy.
University of Illinois blong@uiuc.edu http://www.uiuc.edu/ph/www/blong
Don't worry, these aren't even my views.

> it could be that the best way to handle this is to advise
> people to write such things as nph- scripts, rather than attempt to
> have the server itself be robust against this particular form of
> misbehavior.

"nph" as it stands, sucks. (I'm getting the hang of this Yank speak).

How about if we devise a better method to identify scripts as being
of the nph variety.

Looking for a "nph" prefix is just plain ugly and almost impossible
to switch to once you have popular URLs. A mime type based on the
file extension would look better than a nph prefix, but it's just
as bad when it comes to switching an existing URL to be a "nph" script.

Now that we've sorted out the sleep 3/die problem, I think we can
change the behaviour of script handling to the following...

read the headers
output headers
if header contains some special directive, e.g. Httpd: full-speed-ahead
send the document body using nph technology
else
use existing setup.


One could then modify existing scripts to send headers ASAP, and then
talk directly with the client (at the expense of losing the content length
for the log). Apache could concentrate on the next request without
any delay.

If there are no holes in this proposal, I think it'd make a good
extension for the CGI spec.


robh

/*
* "Re: fork free" by rst@ai.mit.edu (Robert S. Thau)
* written Mon, 24 Apr 95 18:02:35 EDT
*
* So, I'm not sure I see how the server is supposed to put off the
* decision of whether or not to go the nph- route until after the
* script is actually sending output --- to do that, it has to see the
* output, which means the script can't be running under nph "rules of
* engagement".
*
*/

Well, there was a bug in 1.3 where the client socket descriptor was
left open for the CGI process. One (ugly) possibility would be to
duplicate that socket on a well-known fd (like stderr or #3 perhaps)
so that a script could output a stop-parsing magic string on stdout,
and then finish the response on the client socket.

--Rob

Date: Mon, 24 Apr 1995 15:27:48 -0500 (CDT)
From: Brandon Long <blong@uiuc.edu>

A great deal of the problem would also be fixed if we made it an nph
script (then it wouldn't hold up a httpd child). The only reason its
still around is that its useful for testing the server (under bad
cgi conditions )

Hmmm... it will always be the case that a sufficiently ill-behaved CGI
script can cause trouble (e.g., scripts which go into infinte loops,
or worse, do the moral equivalent of "while (1) fork{};"). Given
that, and given that (as Brandon points out), there is a way for
external gateways to long-lived services to avoid hogging a server
process, it could be that the best way to handle this is to advise
people to write such things as nph- scripts, rather than attempt to
have the server itself be robust against this particular form of
misbehavior.

(This is only a real problem for scripts which are likely to hang for
some *substantial* length of time --- several seconds at least).

The reason I'd rather avoid a forking fallback is that it has problems
of its own --- in particular, it can cause the server to handle
transient load peaks in a *very* ungraceful fashion (fork bombs). I
have actually seen that sort of thing happen while testing
server-pooled servers on my own machine under heavy load, with the
real (less busy, but quite peaky) web server in the background.

rst

From: Rob Hartill <hartill@ooo.lanl.gov>
Date: Mon, 24 Apr 95 15:24:23 MDT

Looking for a "nph" prefix is just plain ugly and almost impossible
to switch to once you have popular URLs. A mime type based on the
file extension would look better than a nph prefix, but it's just
as bad when it comes to switching an existing URL to be a "nph" script.

MultiViews can be used to fudge this issue.

Now that we've sorted out the sleep 3/die problem, I think we can
change the behaviour of script handling to the following...

read the headers
output headers
if header contains some special directive, e.g. Httpd: full-speed-ahead
send the document body using nph technology
else
use existing setup.

Ummm... I'm not *quite* sure I understand how this is supposed to work.

The two most important things about nph- these days seem to be that
the script doesn't get screwed up by buffering done by the server (for
Netscape-style "server push" animation), and for us, that a server
process doesn't have to watch the connection, meaning that the server
itself can go on to service another connection. Both properties
require that the socket be passed to the script, so that the server
doesn't have to do further I/O, and can forget about it. However,
that precludes anything like the server's usual processing of CGI
headers.

So, I'm not sure I see how the server is supposed to put off the
decision of whether or not to go the nph- route until after the script
is actually sending output --- to do that, it has to see the output,
which means the script can't be running under nph "rules of engagement".

rst

> Well, there was a bug in 1.3 where the client socket descriptor was
> left open for the CGI process. One (ugly) possibility would be to
> duplicate that socket on a well-known fd (like stderr or #3 perhaps)
> so that a script could output a stop-parsing magic string on stdout,
> and then finish the response on the client socket.

... you took the words out of my mouth.

I was think along the lines of letting the script send the headers
and body in different directions. Sending to stderr isn't particularly
attractive, but as long as both Apache and scripts can agree in advance
where the direct line will be, we should be ok.

robh

I've got some code now which lets you pass a file descriptor, for
a direct link to the client, to a CGI script.

*If* the script wants to use it to talk directly with the client, it
picks up the value from $ENV{CLIENT_FD} and sends back a special
header to the server...

e.g.

#!/usr/local/bin/perl

print "Http-intercept: direct\r\n";
open(NEW,">&$ENV{CLIENT_FD}") || die "Boo hoo";
select(NEW);
print "HTTP/1.0 200 OK\r\n";


Apache justs waits for the script to finish before continuing.

Now if I can only get the child to not wait, and just carry on
regardless (wasn't Sid James in that ?), we'd have something
really useful. It might be useful as it stands though.


robh