Mailing List Archive: performance tuning - smoothing out the spikes

performance tuning - smoothing out the spikes

Feb 11, 1997, 9:23 PM

Post #1 of 12 (3644 views)

So I've managed to set up qmail as purely a delivery agent for mailing
list mail, as opposed to the only MTA on the system. It'll probably be
handling ~150K deliveries a day (on a good day), as a P5-150, 96M ram,
BSDI 2.1 machine. I figure once I know my way around this setup
transferring to full use of qmail will be a manageable task.

Because I have a lot of traffic and a lot of deliveries to distant lands
and slow connections, I've set concurrencyremote to 100. When I was using
sendmail I'd average 100-150 simultaneous connections during the day so
this made sense to me.

The main problem I have with my setup is that when a message comes in for
a 500-person list, and qmail-remotes are fired up to handle the traffic,
the jump in number of processes in such a short time (within 3 seconds)
swamps my machine. It'll be idling there around 10 -remotes and then fork
off another 90. This is a pretty big performance hit - and since they're
all started at once they hit DNS and then the network at the same time.
So for a good 10 seconds the machine is literally at 100% utilization.
This is bad only so far as this machine is performing lots of other tasks
(web server, telnet server, lots of DNS primary hosting) and getting
swamped for 10 seconds every 5 minutes is not a Good Thing. Though some
will of course beg to differ.

One way to ameliorate this swamping is to set limits on the speed at which
qmail-remotes may be launched. If it could be limited to 5 per second,
say, or 3, the process initialization, DNS, and network hits would be
staggered.

I was going to give this a try by putting a simple usleep() command
somewhere in qmail-send, but I was wondering what other folks thought.
Surely if I turned down concurrencyremote to 20 there would be less
impact, but then mail would be delivered more slowly too.

Down the road Dan (or whomever else contributes) may want to think about
what we did for Apache, where a pool of servers are constantly running,
adjusting themselves in number depending on the number of spare resources
they have. Web traffic is a little more constant than email traffic is
so that might not be the right model, but who knows.

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@hyperreal.com http://www.apache.org http://www.organic.com/jobs

Re: performance tuning - smoothing out the spikes [ In reply to ]

ira at ard

Feb 16, 1997, 3:03 AM

Post #2 of 12 (3613 views)

Permalink

On Tue, 11 Feb 1997, Brian Behlendorf wrote:

> So for a good 10 seconds the machine is literally at 100% utilization.
> This is bad only so far as this machine is performing lots of other tasks
> (web server, telnet server, lots of DNS primary hosting) and getting
> swamped for 10seconds every 5 minutes is not a Good Thing. Though some
> will of course beg to differ.

> I was going to give this a try by putting a simple usleep() command
> somewhere in qmail-send, but I was wondering what other folks thought.
> Surely if I turned down concurrencyremote to 20 there would be less
> impact, but then mail would be delivered more slowly too.

in that case, it's time to face reality, and admit your hardware can't
carry the load, and move it to another relay on your network to be
exploded there. mixing mail, telnet, web, dns and news on the same machine
could be fine for some sites, but not when you run 500-name lists and
websites for several virtual domains... at some point you'll have to
release some bottlenecks. if you can't shell out the money for an extra
machine, or explode it at some uplink, then limit the con. to 20 like you
sugested...

--------------------------------------------------------------
Ira Abramov, Sysadmin <ira@scso.com> <ira@ard.huji.ac.il>
Webmaster at the Hebrew University's Authority For R & D
http://ard.huji.ac.il 02-65-86696 / beeper 91826 at 058-212121

Re: performance tuning - smoothing out the spikes [ In reply to ]

bunbun at reptile

Feb 16, 1997, 3:19 PM

Post #3 of 12 (3620 views)

Permalink

On Sun, 16 Feb 1997, Ira Abramov (at work) wrote:

> On Tue, 11 Feb 1997, Brian Behlendorf wrote:

> > So for a good 10 seconds the machine is literally at 100% utilization.

> release some bottlenecks. if you can't shell out the money for an extra
> machine, or explode it at some uplink, then limit the con. to 20 like you
> sugested...

I dunno... there is really no need for mailing lists to be delivered at
top speed... why not have a definable delay? Or configure your qmail to
send only two, three messages at the same time. Or run the q* processor
at the lowest priority?

*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*-=-*
Devilbunny <bunbun@reptile.rug.ac.be>

"Be there now blown clear the Human Race what use is there of figures then
wrote wordless Statistic Names and Numbers now seem out of place
a paper representing Life blowing down the Street"
-- NIVEK OGRE

Re: performance tuning - smoothing out the spikes [ In reply to ]

brian at hyperreal

Feb 16, 1997, 3:29 PM

Post #4 of 12 (3612 views)

Permalink

On Sun, 16 Feb 1997, Ira Abramov (at work) wrote:
> in that case, it's time to face reality, and admit your hardware can't
> carry the load, and move it to another relay on your network to be
> exploded there.

Er, it can "handle" the load fine, by my definition of handle at any
rate... it's the uneven nature of the process model, the fork-and-die
model, which is causing the problems with load, as fork() is expensive,
and having 30 forks in one second is really bad. It's precisely why the
NCSA 1.3 web server was so resource-intensive, for every transaction it
forked a copy of the main server, served the request, and died, whereas
Apache and Netscape's server and others obviated that by having persistant
processes.

Since the requirements for time delay for mail delivery is less than that
for web accesses, it would be fine if qmail simply forked off processes at
a configurable rate, say 3 a second. It would also be better if in queue
runs or large list delivery runs a qmail-remote process didn't die if
there were more addresses in the job left to deliver to, as sendmail does.

All in all qmail is a marvelous piece of work so I don't mean to sound
like an ingrate. I'd contribute code but all my uncompensated development
time is dedicated to Apache. :)

> mixing mail, telnet, web, dns and news on the same machine
> could be fine for some sites, but not when you run 500-name lists and
> websites for several virtual domains...

You'd be surprised.

> at some point you'll have to
> release some bottlenecks. if you can't shell out the money for an extra
> machine, or explode it at some uplink, then limit the con. to 20 like you
> sugested...

Yeah, I've limited it to 50, and in a roundabout way that's probably
functionally equivalent to my suggestion of limiting the spawn rate.

What do folks think about multithreading?

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@hyperreal.com http://www.apache.org http://www.organic.com/jobs

Re: performance tuning - smoothing out the spikes [ In reply to ]

schwartz at cse

Feb 16, 1997, 3:58 PM

Post #5 of 12 (3619 views)

Permalink

>... as fork() is expensive, and having 30 forks in one second is really bad.

Only if your OS sucks. Linux or Plan 9 can do about 300 forks
per second on a Pentium/100, no sweat.

>What do folks think about multithreading?

"If people understood fork, we wouldn't have threads" -- Rob Pike

Re: performance tuning - smoothing out the spikes [ In reply to ]

pgf at foxharp

Feb 16, 1997, 4:09 PM

Post #6 of 12 (3617 views)

Permalink

> >... as fork() is expensive, and having 30 forks in one second is really bad.
>
> Only if your OS sucks. Linux or Plan 9 can do about 300 forks
> per second on a Pentium/100, no sweat.
and how many thread_create()'s can the same processor do? quite a few
more, i'll warrant. and the relative difference will especially be felt on
systems that run on pc-class architectures with only a single process
context.

>
> >What do folks think about multithreading?
>
> "If people understood fork, we wouldn't have threads" -- Rob Pike

i agree completely that fork performance has been sadly neglected over
the years, but the fact is that it has been. on many systems forking is
very expensive -- on those systems letting qmail fork as fast as possible
may well be sub-optimal. (i don't know -- i don't have one of those systems.
i run linux. :-) on the other hand, threading certainly isn't the answer
if you're trying to minimize system complexity.

(btw, i'd be surprised if limiting the qmail's fork rate was a complicated
change.)

paul
---------------------
paul fox, pgf@foxharp.boston.ma.us (arlington, ma)

Re: performance tuning - smoothing out the spikes [ In reply to ]

schwartz at cse

Feb 16, 1997, 5:36 PM

Post #7 of 12 (3613 views)

Permalink

>> Only if your OS sucks. Linux or Plan 9 can do about 300 forks
>> per second on a Pentium/100, no sweat.
>
>and how many thread_create()'s can the same processor do? quite a few
>more, i'll warrant. and the relative difference will especially be felt on
>systems that run on pc-class architectures with only a single process
>context.

My measurements aren't up to lmbench standards, but it looks to me like
Solaris' thr_create() on a Sparc 5/85 is slower than Plan 9's fork() on
a P100. It's even slower if you force thr_create to actually make a new
LWP for each call, which is a fairer comparison to what Plan 9 does.

Re: performance tuning - smoothing out the spikes [ In reply to ]

dgaudet-list-qmail at arctic

Feb 16, 1997, 6:10 PM

Post #8 of 12 (3618 views)

Permalink

Oh give me a break. Having hundreds of forks per second is *not good*
unless the machine is a dedicated mail server. Theoretical maximums don't
mean squat. qmail is not intended to run only on dedicated mail servers,
and so the suggestion is far from pointless. It should limit forks per
second by default. But I, like Brian, put all my spare coding time into
Apache.

Your statement belongs in an advocacy group, and is not really
constructive. I happen to have an SMP linux box that can do hundreds of
forks per second. But if it did, then my mud, and my quake server, and my
webserver, not to mention my own development would all grind to a halt.

Dean

On Sun, 16 Feb 1997, Scott Schwartz wrote:

> >... as fork() is expensive, and having 30 forks in one second is really bad.
>
> Only if your OS sucks. Linux or Plan 9 can do about 300 forks
> per second on a Pentium/100, no sweat.
>
> >What do folks think about multithreading?
>
> "If people understood fork, we wouldn't have threads" -- Rob Pike
>
>

Re: performance tuning - smoothing out the spikes [ In reply to ]

djb at koobera

Feb 16, 1997, 6:58 PM

Post #9 of 12 (3613 views)

Permalink

> Having hundreds of forks per second is *not good*
> unless the machine is a dedicated mail server.

Is your mail server doing thirty million remote deliveries per day?

> It should limit forks per second by default.

Why? What resource is this protecting?

The OS already offers a configurable scheduling mechanism. You can
control how it parcels out CPU time. If you don't want mail deliveries
to interfere with other programs, run them with a lower priority. RTFM.

---Dan
Put an end to unauthorized mail relaying. http://pobox.com/~djb/qmail.html

Re: performance tuning - smoothing out the spikes [ In reply to ]

dgaudet-list-qmail at arctic

Feb 16, 1997, 7:20 PM

Post #10 of 12 (3618 views)

Permalink

Uh, excuse me? Yeah sure I can lower the priority. But that has zero
effect on the effort the kernel is going to expend scheduling 100s of
processes.

The subject is "smoothing out the spikes". I know the machine that Brian
Behlendorf is trying to tune. It handles several really large mailing
lists plus a couple dozen webservers. When a message is set for delivery
to one of the lists there is a bloody huge spike in cpu usage. Spawning
hundreds of active children only blows apart your L2 cache which makes the
cpu spend extra cycles waiting for the L2 to fill as it switches. The
other side-effect is a large number of processes in the Runnable state,
which increases the time the kernel has to bother scheduling things.

He can lower the maximum number of children, but that affects the large
degree of parallelism due to network latencies. It's just the initial
startup time when all children, regardless of where they're delivering to,
are busy. After that the effect of having hundreds live is smoothed by
network latency.

As Brian hinted at, this is a problem that has been examined in detail and
"solved" in the Apache webserver (and others, but that just happens to be
the one which I work with frequently). At some point we improved
performance by rate limiting fork().

Dean

On 17 Feb 1997, D. J. Bernstein wrote:

> > Having hundreds of forks per second is *not good*
> > unless the machine is a dedicated mail server.
>
> Is your mail server doing thirty million remote deliveries per day?
>
> > It should limit forks per second by default.
>
> Why? What resource is this protecting?
>
> The OS already offers a configurable scheduling mechanism. You can
> control how it parcels out CPU time. If you don't want mail deliveries
> to interfere with other programs, run them with a lower priority. RTFM.
>
> ---Dan
> Put an end to unauthorized mail relaying. http://pobox.com/~djb/qmail.html
>

Re: performance tuning - smoothing out the spikes [ In reply to ]

djb at koobera

Feb 16, 1997, 8:49 PM

Post #11 of 12 (3622 views)

Permalink

> Spawning hundreds of active children only blows apart your L2 cache

Unless your whole system's working set fits into the L2 cache---which is
simply not possible for a big server---you will pay for cache refills.
There's no way around this.

If you want the cost of mail delivery to be spread out over time, run
qmail-start (or the name server, if that's where the real problem is)
with a lower priority.

If you want the cost of mail delivery to be reduced, you should start by
finding out where the time is actually going. Maybe it's the networking
stack. Maybe it's shared libraries. Some problems are easier to fix than
others.

> The subject is "smoothing out the spikes".

It is the kernel's job to select quanta that account for overhead.

> a large number of processes in the Runnable state,
> which increases the time the kernel has to bother scheduling things.

The overhead of a few hundred processes should be negligible. If you
measure it and find that it's not, talk to the kernel designers.

---Dan
Put an end to unauthorized mail relaying. http://pobox.com/~djb/qmail.html

Re: performance tuning - smoothing out the spikes [ In reply to ]

jcring at switch

Feb 17, 1997, 7:19 AM

Post #12 of 12 (3606 views)

Permalink

At 05:58 PM 2/16/97 EST, Scott Schwartz wrote:
>>... as fork() is expensive, and having 30 forks in one second is really bad.
>
>Only if your OS sucks. Linux or Plan 9 can do about 300 forks
>per second on a Pentium/100, no sweat.

Appearently there is zero consideration of ever porting QMAIL to a different
OS where the cost of forks is even higher, such as OpenVMS or NT. Not a
gripe, only an observation.
-------------------------
John C. Ring, Jr.
jcring@switch.com
Network Specialist
Union Switch & Signal Inc.