Mailing List Archive

> I've had a quick look at apache 0.7, and it sees to suffer from a major
> design flaw, namely that it always starts up a fixed number of processes.
>
> This would be a pretty major disincentive for a low-use site to run apache.
> Running tens of daemons is very wasteful on swap space and kernel resources,
> and will put off sys admins. But a 0.7 server with a small number of daemons
> (5, say) would probably provide an inferior service to a 0.6 server.
> All it would take is one netscape user to access a page with a few inline
> images, and your site is saturated.
>
> So I would suggest that is is essential that apache adapt the number of
> daemons to the varying connection load.

This was discussed before any 0.7 code was written. After some input from
serveral people, including Rob McCool (Netsite takes the same approach),
it was decided that, forking processes to meet demand wasn't a good idea.
I was worried that a fix number of processes (dozens of them) would be
a bottleneck.

The sysadmin at Cardiff setup a 2nd server the other week, and ran 0.6.x
on it because he didn't want to "waste" RAM on apache processes which were
sitting around doing nothing. I explained how the forking/preforking versions
compared, and he agreed that (in theory at least), that having "too many"
0.7.x apaches would be less overhead than 0.6.x

Ideas on how to fork new processes to meet additional demand were discussed,
but none made any real sense. To fork to meet demand seems to need a
centralised "accept" or child<->parent communication, which we have
made great efforts to avoid so far.

If there are any low overhead solutions to this problem, lets hear them.

rob
--
http://nqcd.lanl.gov/~hartill/

> Clearly you would need child -> parent communication, but not necessarily
> the reverse. Why is that a bad thing?

it's not a bad thing, just something that hasn't been needed so far,
and thus makes the model simpler.

If we move to a new logging system based on children offloading info as
and when they find it, there may well be some child->parent communication
if it is decided that the parent is a good place to do the logging.

But surely if a current user of Apache or 1.3 is worried about the wastage
you mentioned, then won't they already have hit meltdown point when it forks
unlimited numbers of children to talk to multiple mozillas each fetching
4 inlines at a time.
In the example David gave, of a 5 process apache being swamped by
a couple of Mozilla users, I'd guess that a forking server would
be unable to fork processes when needed, and the connections would just
be lost.


0.7 will have a fixed overhead and there's no way to exceed it. The
webmaster will configure Apache and if necessary swapspace to match
the allowable max concurrent requests.

I'd be interested to see any stats that have been collected which
map the number of children required to the daily load or hourly peeks.

I had 80 children running at Cardiff. Now there are 40, and the
responses are always instant... I suspect lots of people will overestimate
how many processes they actually need... 40 children on a 100k hits/day
server with lots of CGI seems to be very comfortable.

> The server needs a fast response to increasing load, with a slower falloff
> when the load drops. The latter can be done as a first-order Poisson process
> with the children killing themselves after a random (but constant mean)
> interval.

It's possible to SIGHUP at different times of the day, to reconfigure
the number of children.. have more during peek hours, less overnight.
It'd be nice (and not impossible) to use another signal to increase the
number of children, without killing running children.

> The parent needs to know how often no child is 'accepting' on the socket
> in order to determine when more children are needed. Maybe it could try
> selecting on the socket itself, to determine if the queue of incoming
> connections stays non-empty.

there's also the chance that if you wait another millisecond, a dozen
children will be ready to accept again... there's a danger of overkill,
spawning so many processes that they make it impossible for each other
to do any work. This was one of the big pluses for a fixed process number.

> Or alternatively, the children could send
> it messages each time the call and return from accept().
> The latter could also be used as keep-alives for the parent to notice
> dead children, and allow it to bury them. You should be doing this sort
> of communication anyway, for the hostname cache.

At the moment the hostname cache isn't being used. We've yet to see if
it is better to leave that to a logging process and/or to the CGI scripts...
I bet 99% of CGI scripts don't even look at the client's hostname.

> Of course, on those machines where you are forced to put a (shared-memory)
> lock around accept(), you get low-overhead child <-> parent IPC for free.


rob
--
http://nqcd.lanl.gov/~hartill/

On Mon, 26 Jun 1995, David Robinson wrote:
> So I would suggest that is is essential that apache adapt the number of
> daemons to the varying connection load.

Then what's the difference between that and 0.6.5?

One suggestion has been (and I think this is what NCSA 1.4 and Netsite do) is
to start with "x" servers, and when all servers are being used and a new
request comes in fork a short-lived one-request-service child, up to "y"
possible children total. I don't know if that's possible without serializing
requests though.... what's a *real* solution is to distinguish between
short-lived responses and long-lived responses, so that 10 people downloading
a 6 meg MPEG movie from your site over a 14.4 modem don't swamp your
10-child server. Any ideas?

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/

On Mon, 26 Jun 1995, Rob Hartill wrote:
> > what's a *real* solution is to distinguish between
> > short-lived responses and long-lived responses, so that 10 people downloading
> > a 6 meg MPEG movie from your site over a 14.4 modem don't swamp your
> > 10-child server. Any ideas?
>
> It should be possible for the child to signal the parent to effectively
> say, "hey, I've just been asked to send this file which is bigger than X.
> Start a new child to replace me, and I promise to exit when I'm done"
>
> However, if X is chosen to be too small, this degenerates into a
> forking model.

I would rather base it on transfer time than size - i.e. the child says
"hey, I've been delivering this object for X seconds and I'm only
halfway done. Start a new child to replace me, and I promise to exit when
I'm done" One could experiment with X to get the forking/non-forking
balance they want - for a heavy site where 5% of the downloads are large
objects that people over 14.4 links want, I might set that to 30 seconds.

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/

>
> On Mon, 26 Jun 1995, David Robinson wrote:
> > So I would suggest that is is essential that apache adapt the number of
> > daemons to the varying connection load.
>
> Then what's the difference between that and 0.6.5?
>
> One suggestion has been (and I think this is what NCSA 1.4 and Netsite do) is
> to start with "x" servers, and when all servers are being used and a new
> request comes in fork a short-lived one-request-service child, up to "y"
> possible children total. I don't know if that's possible without serializing
> requests though.... what's a *real* solution is to distinguish between
> short-lived responses and long-lived responses, so that 10 people downloading
> a 6 meg MPEG movie from your site over a 14.4 modem don't swamp your
> 10-child server. Any ideas?

It should be possible for the child to signal the parent to effectively
say, "hey, I've just been asked to send this file which is bigger than X.
Start a new child to replace me, and I promise to exit when I'm done"

However, if X is chosen to be too small, this degenerates into a
forking model.



--
http://nqcd.lanl.gov/~hartill/

> I would rather base it on transfer time than size - i.e. the child says
> "hey, I've been delivering this object for X seconds and I'm only
> halfway done. Start a new child to replace me, and I promise to exit when
> I'm done" One could experiment with X to get the forking/non-forking
> balance they want - for a heavy site where 5% of the downloads are large
> objects that people over 14.4 links want, I might set that to 30 seconds.
>
> Brian

Okay, how about 2 config setting, X, Y

if after X seconds of writing to the client and at the current rate
it looks like taking > Y seconds to send the rest, then signal the parent
with a "replace me".

I'd set X and Y to 15 seconds as a first attempt.


rob
--
http://nqcd.lanl.gov/~hartill/

Clearly you would need child -> parent communication, but not necessarily
the reverse. Why is that a bad thing?

The server needs a fast response to increasing load, with a slower falloff
when the load drops. The latter can be done as a first-order Poisson process
with the children killing themselves after a random (but constant mean)
interval.

The parent needs to know how often no child is 'accepting' on the socket
in order to determine when more children are needed. Maybe it could try
selecting on the socket itself, to determine if the queue of incoming
connections stays non-empty. Or alternatively, the children could send
it messages each time the call and return from accept().
The latter could also be used as keep-alives for the parent to notice
dead children, and allow it to bury them. You should be doing this sort
of communication anyway, for the hostname cache.

Of course, on those machines where you are forced to put a (shared-memory)
lock around accept(), you get low-overhead child <-> parent IPC for free.

David.

/*
* "Re: Apache 0.7: too many processes" by rst@ai.mit.edu (Robert S. Thau)
* written Mon, 26 Jun 95 20:42:43 EDT
*
** One suggestion has been (and I think this is what NCSA 1.4 and
** Netsite do) is to start with "x" servers, and when all servers are
** being used and a new request comes in fork a short-lived
** one-request-service child, up to "y" possible children total.
* Hmmm... I believe NCSA 1.4 does this, but I don't think Netsite
* does.
*/

That's correct. Netsite also implements the lockf() around accept()
that Cliff's friend suggested after we made a stink about this with
Sun last fall.

--Rob

Date: Mon, 26 Jun 95 15:29 BST
From: drtr@ast.cam.ac.uk (David Robinson)
Precedence: bulk
Reply-To: new-httpd@hyperreal.com

This would be a pretty major disincentive for a low-use site to run apache.
Running tens of daemons is very wasteful on swap space and kernel resources,
and will put off sys admins.

Is that so? As you note immediately after saying this:

But a 0.7 server with a small number of daemons
(5, say) would probably provide an inferior service to a 0.6 server.
All it would take is one netscape user to access a page with a few inline
images, and your site is saturated.

In other words, under peak conditions you *need* to be able to run
tens of daemons simultaneously or performance will be terrible.
What's more, since peaks may arise without warning, you need those
swap space and kernel resources to be available to the web server all
the time, even if there isn't an actual process pre-forked and serving
as a placeholder.

(In any case, the amount of resources actually consumed by having the
pre-forked children around is not that great; my process pool
currently averages about 300K/process for 20, or about 6 megabytes
total (printout below) --- that's not much these days for a dedicated
server. NB this is Shambhala, but I don't think whatever difference
there is between it and 0.7.x is enough to make a major impact).

BTW, I do think there's an issue which can reasonably be addressed
here, which is trying to find the right value for StartServers
adaptively, rather than making the webmaster guess. However, as the
experience at Cardiff suggests, I think the main issue here is making
sure that you have enough of them that they don't all get tied up
talking to slow clients. At any rate, any strategy for adaptively
adjusting child server lifetimes should probably keep that in mind, as
well as the connection arrival rate...

rst

PS: as promised, here's ps alx output for my server pool (these
processes have all been running since 3:00 AM, so I don't expect them
to get much bigger than this):

F UID PID PPID CP PRI NI SZ RSS WCHAN STAT TT TIME COMMAND
80488000 0 3623 1 6 5 0 104 0 child IW ? 0:00 ./shambhala
488001 8 5092 3623 0 1 0 360 496 socket S ? 2:55 ./shambhala
488001 8 5093 3623 1 1 0 276 400 socket S ? 3:10 ./shambhala
488000 8 5094 3623 0 1 0 324 0 socket IW ? 3:33 ./shambhala
488001 8 5095 3623 0 1 0 276 424 socket S ? 3:10 ./shambhala
488000 8 5096 3623 0 1 0 312 0 socket IW ? 2:48 ./shambhala
488001 8 5097 3623 2 1 0 300 468 socket S ? 3:13 ./shambhala
488001 8 5098 3623 0 1 0 312 464 socket S ? 2:47 ./shambhala
488001 8 5099 3623 1 1 0 324 476 socket S ? 3:01 ./shambhala
488000 8 5100 3623 0 1 0 336 0 socket IW ? 3:13 ./shambhala
488000 8 5101 3623 0 1 0 324 0 socket IW ? 2:40 ./shambhala
488001 8 5102 3623 0 1 0 276 468 socket S ? 3:03 ./shambhala
488001 8 5103 3623 4 1 0 312 488 socket S ? 3:01 ./shambhala
488001 8 5104 3623 1 1 0 352 488 socket S ? 3:00 ./shambhala
488001 8 5105 3623 0 1 0 308 76 socket S ? 3:13 ./shambhala
488000 8 5106 3623 1 1 0 300 0 socket IW ? 3:12 ./shambhala
488001 8 5107 3623 1 1 0 312 484 socket S ? 2:43 ./shambhala
488001 8 5108 3623 1 1 0 300 472 socket S ? 2:43 ./shambhala
488001 8 5109 3623 0 1 0 324 456 socket S ? 3:01 ./shambhala
488001 8 5110 3623 0 1 0 344 484 socket S ? 3:12 ./shambhala
488001 8 5111 3623 2 1 0 276 448 socket S ? 2:56 ./shambhala

Date: Mon, 26 Jun 1995 11:13:38 -0700 (PDT)
From: Brian Behlendorf <brian@organic.com>

One suggestion has been (and I think this is what NCSA 1.4 and
Netsite do) is to start with "x" servers, and when all servers are
being used and a new request comes in fork a short-lived
one-request-service child, up to "y" possible children total.

Hmmm... I believe NCSA 1.4 does this, but I don't think Netsite does.

rst

David writes,

> This does allow the number of servers to temporarily grow, but it doesn't
> really allow for automatic setting of the number of daemons (or X and Y).
>
> Let connection rate be r, connection time t

where do these come from ? I can't follow the rest of the algorithm
without understanding what these meand, and how they are computed
(presumably at regular intervals)

> A simple dynamical system:
> create processes at a rate of 1/N * (r + 1/t )
> processes die after N*t

they die based on how long they've been alive rather than how many
requests they've processed ?

The only reason for children to die at the moment is to limit the
possibility of meltdown due to leaks.. ideally they should run forever.

> Or, a similar system with possibly a better initial response time:
> if no processes are blocked in accept, then create a new process, up to
> a maximum creation rate cmax
> processes die after N*t

I could live with that, maybe with "spare" children just not being replaced
once they've processed N requests (that's easy to fit into the current model)

So is it possible for the parent to monitor the accept status without
pulling requests out of the queue ?

> Both these should have a typical number of processes of rt + 1, with each
> process typically handling N connections.
> The second process limits the number of processes to N.cmax.t; so cmax could
> be set to nmax/(Nt)

where's my calculator ? :-)

rob
--
http://nqcd.lanl.gov/~hartill/

>Okay, how about 2 config setting, X, Y
>
>if after X seconds of writing to the client and at the current rate
>it looks like taking > Y seconds to send the rest, then signal the parent
>with a "replace me".
>
>I'd set X and Y to 15 seconds as a first attempt.
>
>rob

This does allow the number of servers to temporarily grow, but it doesn't
really allow for automatic setting of the number of daemons (or X and Y).

Let connection rate be r, connection time t

A simple dynamical system:
create processes at a rate of 1/N * (r + 1/t )
processes die after N*t

Or, a similar system with possibly a better initial response time:
if no processes are blocked in accept, then create a new process, up to
a maximum creation rate cmax
processes die after N*t

Both these should have a typical number of processes of rt + 1, with each
process typically handling N connections.
The second process limits the number of processes to N.cmax.t; so cmax could
be set to nmax/(Nt)

David.