Mailing List Archive

----------
> From: Jon Kay <jkay@cs.utexas.edu>
> To: djb-qmail@koobera.math.uic.edu
> Subject: infinite connections (was: Features)
> Date: Wednesday, March 05, 1997 2:20 PM
>
> I think you guys misunderstand jhawk. The problems here are the
> *default* concurrency setting and the apparently widespread belief
> within the qmail community that there are no adverse effects
> associated with maximizing concurrency.

I won't argue any of your points except to say
that this parameter *is* configurable. If you don't
like lots of simulateneous deliveries, why not
set the concurrency limit to *1*. Then you get
sendmail behavior.

Personally, mail is the single most important
Internet service for me. Far more important
that my web surfing. I'm waiting on
some critical documents to arrive right now.

Daryl
Daryl L. Biberdorf darylb@superserve.com

D. L. Biberdorf writes:
> I won't argue any of your points except to say
> that this parameter *is* configurable. If you don't
> like lots of simulateneous deliveries, why not
> set the concurrency limit to *1*. Then you get
> sendmail behavior.

Not quite. Sendmail aggregates deliveries to the same host. Instead
of multiple TCP sessions for the same piece of mail, it gives you one.
Setting the concurrency limit just affects whether the sessions occur
in parallel or serially.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

Jon Kay writes:
> I think you guys misunderstand jhawk. The problems here are the
> *default* concurrency setting

No. The problem is that a TCP session "learns" what is the proper
retransmit time for a connection. It takes a few back-and-forth
packets for this to occur, however, so if you have a short-lived
connection, you will be using an inappropriate timeout.

sendmail's solution (MX everything; sort; and aggregate deliveries)
reportedly uses *more* total traffic, some of which uses UDP with a
fixed timeout algorithm.

Frankly, I doubt that anyone who objects to single-rcpt deliveries is
actually looking up the MX records. Everyone who has reported counts
of actual users on the same host has simply examined the host name.
qmail could do the same thing.

> The bottom line here is that right now you are killing your own users
> by using qmail. Qmail is almost certainly using up the network so
> thoroughly when it's running that other people trying to use the
> network have to be pretty much scrogged.

Not very likely. Look at the traffic stats in terms of volume. SMTP
is way down there on the list.

If you have client machines surfing the web, then by definition you
don't care about the cost of SMTP.

> Running this number of concurrent TCP connections effectively
> disables all TCP's fairness algorithms, which are what keep the
> network usable for everybody.

Nope. It's "running this number of TCP connections." Who cares if
they occur at the same time, or different times?

> A default maximum, at least for network connections, on the order of
> two, would seem more appropriate.

Not for a mailing list server.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

> > I think you guys misunderstand jhawk. The problems here are the
> > *default* concurrency setting
>
> No. The problem is that a TCP session "learns" what is the proper
> retransmit time for a connection.

This is not a problem. This is a feature. "retransmit time" is not
the issue -- it's size of the transmission window.

> It takes a few back-and-forth packets for this to occur, however, so
> if you have a short-lived connection, you will be using an
> inappropriate timeout.

Again, not the concern. If you send 10 simultaneous TCP SYNs and 5
get dropped, those five connections don't back off or get throttled,
they'll just fire again 6 seconds later and, effectively not learning
from their mistakes. If some fraction of the TCP sessions gets
penalized due to congestion, the rest may not see this congestion and
back off appropriately. This is wasteful of network bandwidth.

> sendmail's solution (MX everything; sort; and aggregate deliveries)
> reportedly uses *more* total traffic, some of which uses UDP with a
> fixed timeout algorithm.

Please justify your statement, it does not make sense.

> Frankly, I doubt that anyone who objects to single-rcpt deliveries is
> actually looking up the MX records. Everyone who has reported counts
> of actual users on the same host has simply examined the host name.
> qmail could do the same thing.

This would be an optimization, and clearly deals with the obvious case
of multiple TCP sessions to the same host. So would a multiplexing
implementation.

The trickier issue is determining how much of a problem concurrent
delivery to multiple hosts is. I suppose this would be best addressed
by some simulations.

> > The bottom line here is that right now you are killing your own users
> > by using qmail. Qmail is almost certainly using up the network so
> > thoroughly when it's running that other people trying to use the
> > network have to be pretty much scrogged.
>
> Not very likely. Look at the traffic stats in terms of volume. SMTP
> is way down there on the list.

I'm not sure where you're getting your data from, but I don't think it's
realistic.

In some stats for the past few weeks from one of my boxes at MAE WEST<
SMTP is #4:

Protocol Bytes/Sec
-------- ---------
TCP-WWW 3611200
TCP-FTPD 281338
UDP-other 227001
TCP-SMTP 159556
UDP-DNS 123066
TCP-Telnet 23048.2
GRE 20147.4
ICMP 17776.5
UDP-Frag 8037.2
TCP-FTP 4495
TCP-X 3879.3
UDP-NTP 1383.2
TCP-BGP 847.3
TCP-Frag 746.4
IPINIP 653.8
IP-other 471.2
IGMP 345.6
UDP-TFTP 94
TCP-other 1.25944e+06
Total: 5.73994e+06

> If you have client machines surfing the web, then by definition you
> don't care about the cost of SMTP.

I'm afraid I am not addressing your individual concerns.
I'm concerned about the health of the network, not of individual
users.


> > Running this number of concurrent TCP connections effectively
> > disables all TCP's fairness algorithms, which are what keep the
> > network usable for everybody.
>
> Nope. It's "running this number of TCP connections." Who cares if
> they occur at the same time, or different times?

People who care about the transport layer.
Running lots of TCP connections at the same time cuts down on the
effectiveness of congestion control, and makes the Internet's data
behave more more like constant bit rate trafffic that does not
respond to congestion control. This is one of the phenomena the
Web is responsible for bringing around.

> > A default maximum, at least for network connections, on the order of
> > two, would seem more appropriate.
>
> Not for a mailing list server.

All TCP applications need to be well-behaved. Or at least, such is my
thesis. You may disagree, but at that point having the discussion with
you is rather fruitless.

The network is not a magic black box, it takes a bit of proper care
and feeding.

--jhawk

> I won't argue any of your points except to say
> that this parameter *is* configurable.

Very true. And helpful.

But that doesn't help with the jillions of other qmail sites.

> Personally, mail is the single most important
> Internet service for me. Far more important
> that my web surfing.

I said nothing about *importance.*

I talked about *interactivity.* There is a subtle, but important
difference.

Dunno about you guys, but between dyslexic typing, mouse jitter, NFS
slowness, and a godawful huge mail queue, I can't generally get to
read a new piece of mail in under a couple of seconds. And, even
going through junk mail, it's pretty hard for me to go through more
than one piece of mail a second. In general, I'm pretty happy if mail
arrives in under a minute (which is more quickly than I generally
notice mail arrival anyway). Yes, it had better get there eventually,
or I will be REAL unhappy.

Telnet is a different story. When I type a keystroke into telnet, I
am pretty unhappy if the character doesn't get there in a small
fraction of a second. And if I keep typing, the characters better
keep coming through at a pretty good clip.

Yes, I am often unhappy about my telnet performance. The Internet is
in poor shape, and programs that fail to exercise reasonable flow and
congestion control policies only make things worse.

Jon

On 5 Mar 1997, Russell Nelson wrote:

> > by using qmail. Qmail is almost certainly using up the network so
> > thoroughly when it's running that other people trying to use the
> > network have to be pretty much scrogged.
>
> Not very likely. Look at the traffic stats in terms of volume. SMTP
> is way down there on the list.

I think such generalizations are rather dangerous. I can easily imagine a
site (though I've never managed one, personally) where SMTP volume is not
"way down there". How about a standalone server, with a low-bandwidth
Internet connection, which manages several mailing lists and also
accepts anonymous FTP requests? If the anonymous FTP requests require
good responsiveness, but mailing list requests cause brief periods of
poor responsiveness, then there could be complaints.

Of course, the proper course of action in this scenario would be to
reduce the parallelism of qmail. But blanket statements that SMTP
bandwidth is negligible aren't useful.

> If you have client machines surfing the web, then by definition you
> don't care about the cost of SMTP.

Again, I don't see that this is necessarily true in all cases. Perhaps
a slight rewording would help get the point across.

------------ Greg Wooledge -------------
------- <wooledge@kellnet.com> -------
--- <http://kellnet.com/wooledge/main.html> ---

On Wed, 5 Mar 1997, Greg Wooledge wrote:
[stuff deleted]
> "way down there". How about a standalone server, with a low-bandwidth
> Internet connection, which manages several mailing lists and also
> accepts anonymous FTP requests? If the anonymous FTP requests require
> good responsiveness, but mailing list requests cause brief periods of
> poor responsiveness, then there could be complaints.
> Of course, the proper course of action in this scenario would be to
> reduce the parallelism of qmail. [ stuff deleted]

Reducing concurrency is the only feasable answer. In this situation
continuing to maintain a high volume mailing list as well as even a
moderate volume ftp site leaves you (the admin) with 3 choices:

1) Make the mailing list less responsive, and the ftp service spiffy.
Result: ftp users are happy, but the subscribers to the list(s) receive
their list mail absymally slowly. (Abysmally, based on the fact that
if there's a high volume of mail, then there has to be both a lot of
activity and a lot of recipients). Your site becomes a favorite as an ftp
archive, but the list maintainers will probably look for another ISP.

2) Throttle the ftp service, allow mail to go quickly; in this case your
list subscribers will be happy, but whomever is running the archive will
probably seek another ISP.

3) Let them fight it out. If the lists are small, the burst of activity
won't even really interrupt ftp transfers, or affect speed. If the lists
are large though, then ftp speed will really suffer.

Unfortunately the very compelling argument against throttling the # of
qmail remotes is that if a site has a dead network connection, or a broken
mail server, then you're stuck doing the sendmail thing: manually removing
messages from the queue; and having all remote deliveries wait until that
site times out each time the queue is run.

The only answer that keeps both groups happy in a scenario like this is to
get more bandwidth. Neither throttling scenario provides for all groups
being adequately taken care of.

-Peter

John Hawkinson writes:
> > sendmail's solution (MX everything; sort; and aggregate deliveries)
> > reportedly uses *more* total traffic, some of which uses UDP with a
> > fixed timeout algorithm.
>
> Please justify your statement, it does not make sense.

According to Dan's measurements, at three sites, the total traffic
generated by the MTA is lower with qmail than with sendmail. I don't
know if he measured traffic by packet or byte counts. Sendmail, when
presented with multiple recipients for the same piece of mail, MXes
all of the addresses, sorts by destination machine, then aggregates
deliveries to the same machine. This results in more DNS queries than
qmail uses. DNS queries use UDP, not TCP, so the whole TCP
congestance avoidance scheme goes out the window. For the sake of
truth, I haven't checked to see how often bind retries requests, but
given how quickly it times out, it can't be using an algorithm that
takes network congestion very much into account.

> > Not very likely. Look at the traffic stats in terms of volume. SMTP
> > is way down there on the list.
>
> In some stats for the past few weeks from one of my boxes at MAE WEST<
> SMTP is #4:

You ought to read Huff's _How to Lie with Statistics_. I said "in
terms of volume", not in terms of protocols being used. I've added
another column to *your own figures*, which proves my point. SMTP
traffic is WAY down on the list -- only 3 % of the total.

> Protocol Bytes/Sec % of total
> -------- --------- -------
> TCP-WWW 3611200 63
> TCP-other 1.25944e+06 22
> TCP-FTPD 281338 5
> UDP-other 227001 4
> TCP-SMTP 159556 3
> Total: 5.73994e+06
>
> > > Running this number of concurrent TCP connections effectively
> > > disables all TCP's fairness algorithms, which are what keep the
> > > network usable for everybody.
> >
> > Nope. It's "running this number of TCP connections." Who cares if
> > they occur at the same time, or different times?
>
> Running lots of TCP connections at the same time cuts down on the
> effectiveness of congestion control, and makes the Internet's data
> behave more more like constant bit rate trafffic that does not
> respond to congestion control.

That's really only because most TCP stacks don't share retransmit
timer information between TCP connections. It's an implementation
issue, not a design or protocol issue.

And remember the application here -- concurrent deliveries to the same
host generated by a posting to a mailing list. There is very, very
little difference between five TCP sessions, to the same host, started
at about the same time (which would ONLY happen if the users at the
same host are in:
head -`cat /var/qmail/control/concurrencyremote` .qmail-LIST
and five TCP sessions to the same host that don't overlap in time. If
they're not, then the connections to the same host are very, very
unlikely to be concurrent. In the face of congestion, the net is a
very, very good random number generator. An MTA would be extremely
unlikely to start the identical set of SMTP sessions past the first
concurrencyremote addresses, because the first sessions will complete
in random amounts of time, so the next sessions will start at random
times after that.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

Dean Gaudet writes:
> I have absolutely no proof that any of this is a win. But I think
> the numbers speak for themselves. 5803 single recipient deliveries
> will necessarily take more of my network and machine resources than a
> single 5803-recipient delivery.

You mean 58 hundred-recipient, and one three-recipient deliveries.
SMTP RFC mandates 100 or less RCPTs per delivery, and sendmail only
uses up to 20, so it's likely that some SMTP's have a rcpts[20]
somewhere in their innards.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

The heuristic I've been considering is to push MX-grouping responsibility
into the MLM (and doing "rev | sort | rev" is what I'd call sufficient).
Then tweaking the remote delivery system to group sequential addresses
with the same MX into a single streamed connection. My gut feeling is
that this is good enough to get a win for little coding effort.

I'm not talking about doing the multi-rcpt thing, because I like RFCVERP
for the most part. I'm just talking about streaming multiple messages on
a single TCP connection.

RFCVERP breaks down when delivering to brain dead lotus notes gateways,
msmail gateways, compuserve, and some mainframe gateways -- all of which
I've seen send "bounces" back to the Reply-To: or the From: addresses
instead of the envelope sender. But that's not something easy to work
around.

One list I manage has 173521 subscribers, here's the domains with
more than 1000 recipients:

5803 aol.com
5097 ix.netcom.com
4156 compuserve.com
2723 msn.com
2093 earthlink.net
1969 hotmail.com
1372 worldnet.att.net
1195 prodigy.com

That's a small enough number that I'd be happy doing RFCVERP for the
rest of the list, but doing something special for those. For those
multi-recipient delivery and a bounce parser would work fine.

I have absolutely no proof that any of this is a win. But I think
the numbers speak for themselves. 5803 single recipient deliveries
will necessarily take more of my network and machine resources than a
single 5803-recipient delivery. How it affects AOL's resources I won't
speculate on, since they do have some godawfully large number of MX hosts.

Dean

On Wed, 5 Mar 1997, John Hawkinson wrote:

> > > I think you guys misunderstand jhawk. The problems here are the
> > > *default* concurrency setting
> >
> > No. The problem is that a TCP session "learns" what is the proper
> > retransmit time for a connection.
>
> This is not a problem. This is a feature. "retransmit time" is not
> the issue -- it's size of the transmission window.
>
> > It takes a few back-and-forth packets for this to occur, however, so
> > if you have a short-lived connection, you will be using an
> > inappropriate timeout.
>
> Again, not the concern. If you send 10 simultaneous TCP SYNs and 5
> get dropped, those five connections don't back off or get throttled,
> they'll just fire again 6 seconds later and, effectively not learning
> from their mistakes. If some fraction of the TCP sessions gets
> penalized due to congestion, the rest may not see this congestion and
> back off appropriately. This is wasteful of network bandwidth.
>
> > sendmail's solution (MX everything; sort; and aggregate deliveries)
> > reportedly uses *more* total traffic, some of which uses UDP with a
> > fixed timeout algorithm.
>
> Please justify your statement, it does not make sense.
>
> > Frankly, I doubt that anyone who objects to single-rcpt deliveries is
> > actually looking up the MX records. Everyone who has reported counts
> > of actual users on the same host has simply examined the host name.
> > qmail could do the same thing.
>
> This would be an optimization, and clearly deals with the obvious case
> of multiple TCP sessions to the same host. So would a multiplexing
> implementation.
>
> The trickier issue is determining how much of a problem concurrent
> delivery to multiple hosts is. I suppose this would be best addressed
> by some simulations.
>
> > > The bottom line here is that right now you are killing your own users
> > > by using qmail. Qmail is almost certainly using up the network so
> > > thoroughly when it's running that other people trying to use the
> > > network have to be pretty much scrogged.
> >
> > Not very likely. Look at the traffic stats in terms of volume. SMTP
> > is way down there on the list.
>
> I'm not sure where you're getting your data from, but I don't think it's
> realistic.
>
> In some stats for the past few weeks from one of my boxes at MAE WEST<
> SMTP is #4:
>
> Protocol Bytes/Sec
> -------- ---------
> TCP-WWW 3611200
> TCP-FTPD 281338
> UDP-other 227001
> TCP-SMTP 159556
> UDP-DNS 123066
> TCP-Telnet 23048.2
> GRE 20147.4
> ICMP 17776.5
> UDP-Frag 8037.2
> TCP-FTP 4495
> TCP-X 3879.3
> UDP-NTP 1383.2
> TCP-BGP 847.3
> TCP-Frag 746.4
> IPINIP 653.8
> IP-other 471.2
> IGMP 345.6
> UDP-TFTP 94
> TCP-other 1.25944e+06
> Total: 5.73994e+06
>
> > If you have client machines surfing the web, then by definition you
> > don't care about the cost of SMTP.
>
> I'm afraid I am not addressing your individual concerns.
> I'm concerned about the health of the network, not of individual
> users.
>
>
> > > Running this number of concurrent TCP connections effectively
> > > disables all TCP's fairness algorithms, which are what keep the
> > > network usable for everybody.
> >
> > Nope. It's "running this number of TCP connections." Who cares if
> > they occur at the same time, or different times?
>
> People who care about the transport layer.
> Running lots of TCP connections at the same time cuts down on the
> effectiveness of congestion control, and makes the Internet's data
> behave more more like constant bit rate trafffic that does not
> respond to congestion control. This is one of the phenomena the
> Web is responsible for bringing around.
>
> > > A default maximum, at least for network connections, on the order of
> > > two, would seem more appropriate.
> >
> > Not for a mailing list server.
>
> All TCP applications need to be well-behaved. Or at least, such is my
> thesis. You may disagree, but at that point having the discussion with
> you is rather fruitless.
>
> The network is not a magic black box, it takes a bit of proper care
> and feeding.
>
> --jhawk
>

Ah bleh, there's a fact that I would have liked to hear a while ago. This
gives a lot more credibility to yours and Dan's arguments that multi-rcpt
isn't a win. Thanks for the data. At 20 per delivery I can see lots of
other factors making the results less clear.

Dean

On 6 Mar 1997, Russell Nelson wrote:

> Dean Gaudet writes:
> > I have absolutely no proof that any of this is a win. But I think
> > the numbers speak for themselves. 5803 single recipient deliveries
> > will necessarily take more of my network and machine resources than a
> > single 5803-recipient delivery.
>
> You mean 58 hundred-recipient, and one three-recipient deliveries.
> SMTP RFC mandates 100 or less RCPTs per delivery, and sendmail only
> uses up to 20, so it's likely that some SMTP's have a rcpts[20]
> somewhere in their innards.
>
> --
> -russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
> Crynwr Software sells network driver support | PGP ok
> 521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
> Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)
>

Dean Gaudet:
> RFCVERP breaks down when delivering to brain dead lotus notes gateways,
> msmail gateways, compuserve, and some mainframe gateways -- all of which
> I've seen send "bounces" back to the Reply-To: or the From: addresses
> instead of the envelope sender. But that's not something easy to work
> around.

It is possible to work around, in principle:

(1) maintain a database of such sites (e.g. LDAP).
(2) Munge From: and Reply-To: lines going to such sites so that they
have envelope return address.
(3) Include at top of message a short paragraph explaining why this
measure has been taken and what address to reply to list at.
(4) periodically test SMTP session signature of site, to see if it
changes (e.g. initial 220 line and response to EHLO or HELO).

"It's just software."

--
Raul

> qmail uses. DNS queries use UDP, not TCP, so the whole TCP
> congestance avoidance scheme goes out the window. For the sake of
> truth, I haven't checked to see how often bind retries requests, but
> given how quickly it times out, it can't be using an algorithm that
> takes network congestion very much into account.

The algorithm for the client resolver is "first timeout X seconds,
double every time, try N times". N is fixed at 4. X=max(4, 5/number of
servers), i.e. 5 for one server and 4 for more than one. Of course,
this applies to every query made by every process - new queries always
start with the minimum timeout.

The server itself, when qerying other servers, starts at max(4, 2*RTT
estimate) and limits the maximum timeout to 45 seconds.

RFC1536 describes these as "good" algorithms, mainly because they use
exponential backoff and therefore don't cause congestion. Many users
disagree and curse the long timeouts.

olaf

Raul Miller <rdm@rdm.legislate.com> writes:
>Dean Gaudet:
>> RFCVERP breaks down when delivering to brain dead lotus notes gateways,
>> msmail gateways, compuserve, and some mainframe gateways -- all of which
>> I've seen send "bounces" back to the Reply-To: or the From: addresses
>> instead of the envelope sender. But that's not something easy to work
>> around.
>
>It is possible to work around, in principle:
>
>(1) maintain a database of such sites (e.g. LDAP).
>(2) Munge From: and Reply-To: lines going to such sites so that they
>have envelope return address.
>(3) Include at top of message a short paragraph explaining why this
>measure has been taken and what address to reply to list at.
>(4) periodically test SMTP session signature of site, to see if it
>changes (e.g. initial 220 line and response to EHLO or HELO).
>

Hmmm... Something occurred to me while reading this. I imagine
the folks who've written list managers have already thought of
this and discarded it, but here goes:

If you have to munge each outgoing message (or a selected subset
of them) to compensate for gateways that lose the VERP'ed envelope
sender address, why not put the VERP'ed address into the Message-ID?

When the bounce comes back to the list owner, the message-id in the
enclosed message can show the failing address. If the gateway doesn't
include the original address or a reference to its id, then you're no
worse off than before.

-Greg
--
Greg Andrews West Coast Online
Unix System Administrator 5800 Redwood Drive
gerg@wco.com Rohnert Park CA 94928
(yes, 'greg' backwards) 1-800-WCO-INTERNET

John Hawkinson <jhawk@bbnplanet.com> writes on 5 March 1997 at 18:20:24 -0500
> > sendmail's solution (MX everything; sort; and aggregate deliveries)
> > reportedly uses *more* total traffic, some of which uses UDP with a
> > fixed timeout algorithm.
>
> Please justify your statement, it does not make sense.

I've been wondering about this myself. Clearly qmail does have to
look up the mx record at some point. So where does qmail win here?

I understand the sendmail sequence to look like this:

For each recipient, do DNS lookup to find mx.
Sort on MX
For each mx,
Do dns lookup to find system
Make connect to system and send multiple rcpts mail

And the qmail sequence is:
For each recipient (in parallel):
Do dns lookup to find mx
Do dns lookup on mx
make connect and deliver message

This actually seems to give sendmail a win on total network
transactions, plus it spreads them over more time :-), thus being
"gentler" on the network.

Now, I know perfectly well that Dan and Russell and Dave Sill, for
example, know this stuff better than me, so at least one of the obove
sequences of actions is presumably WRONG.

What does reality look like?

Russell Nelson <nelson@crynwr.com>
> According to Dan's measurements, at three sites, the total traffic
> generated by the MTA is lower with qmail than with sendmail. I don't
> know if he measured traffic by packet or byte counts. Sendmail, when
> presented with multiple recipients for the same piece of mail, MXes
> all of the addresses, sorts by destination machine, then aggregates
> deliveries to the same machine. This results in more DNS queries than
> qmail uses.

But then comes the next question: Does the number of DNS queries
relate to generated WAN traffic?

> DNS queries use UDP, not TCP, so the whole TCP
> congestance avoidance scheme goes out the window. For the sake of
> truth, I haven't checked to see how often bind retries requests, but
> given how quickly it times out, it can't be using an algorithm that
> takes network congestion very much into account.

There's BIND, and there's BIND. BIND-the-resolver is often quite bad
at backing off, but it generally talks only to local name servers (the
ones in resolv.conf, on UNIX). BIND-the-nameserver, which _does_ talk
to remote name server, is much better at timing out. It also caches
the results, so you can do one, three or even ten MX lookups in the
course of message processing and BIND will generate the same amount of
WAN traffic.

--Arnt

>I've been wondering about this myself. Clearly qmail does have to
>look up the mx record at some point. So where does qmail win here?
>
>I understand the sendmail sequence to look like this:
>
> For each recipient, do DNS lookup to find mx.
> Sort on MX
> For each mx,
> Do dns lookup to find system
> Make connect to system and send multiple rcpts mail
>
>And the qmail sequence is:
> For each recipient (in parallel):
> Do dns lookup to find mx
> Do dns lookup on mx
> make connect and deliver message
>
>This actually seems to give sendmail a win on total network
>transactions, plus it spreads them over more time :-), thus being
>"gentler" on the network.
>
>Now, I know perfectly well that Dan and Russell and Dave Sill, for
>example, know this stuff better than me,

Whoa, I never claimed to be an expert on qmail or sendmail internals.

>so at least one of the obove
>sequences of actions is presumably WRONG.

My understanding is that sendmail does more than two DNS lookups per
delivery. Can't remember how many he said it did, though. Stephen van
den Berg, author of procmail, claimed to have a sendmail.cf that did
only two DNS lookups, but he never produced a copy.

Dan was also the one who claimed recently that he measured and
compared bandwidth on three sites and found that qmail used less. I
don't think he's lying, but I also don't know the details.

>What does reality look like?

I wish I knew. :-)

-Dave

David Dyer-Bennet <ddb@gw.ddb.com> writes:
>
>I've been wondering about this myself. Clearly qmail does have to
>look up the mx record at some point. So where does qmail win here?
>

As I understand it, qmail performs fewer lookups on non-recipient
addresses. I'm not an authority on how Sendmail does its lookups,
but based on what I've seen here and in comp.mail.sendmail, I've
made additions to your sequence below:

>
>I understand the sendmail sequence to look like this:
>

For the sender address
do DNS lookup of host/domain in address
rewrite address with canonical fqdn
For each address in From: header
sort by domain, remove duplicates
do DNS lookup of host/domain in address
rewrite address with canonical fqdn
For each address in To: header
sort by domain, remove duplicates
do DNS lookup of host/domain in address
rewrite address with canonical fqdn
For each address in Cc: header
sort by domain, remove duplicates
do DNS lookup of host/domain in address
rewrite address with canonical fqdn
add address to recipient list
For each address in Bcc: header (if any)
sort by domain, remove duplicates
do DNS lookup of host/domain in address
rewrite address with canonical fqdn
add address to recipient list, remove from header
> For each recipient, do DNS lookup to find mx.
> Sort on MX
> For each mx,
> Do dns lookup to find system
> Make connect to system and send multiple rcpts mail
>
>
>And the qmail sequence is:
> For each recipient (in parallel):
> Do dns lookup to find mx
> Do dns lookup on mx
> make connect and deliver message
>

So every time it handles a message, Sendmail attempts to rewrite
the envelope and header addresses via CNAME lookups. If Sendmail
merely queues the message (because of -odq argument or high load
average), I'm pretty sure it doesn't perform the MX lookups.

This means Sendmail repeats the lookups necessary for address rewriting
for each delivery attempt. Qmail does not. If delivery was deferred,
qmail performs only the MX lookups on each delivery attempt. Sendmail
performs the MX and the CNAME lookups on each delivery attempt.


This is comparing the out-of-the-box configuration of qmail and
Sendmail. I've been told that Sendmail can be tweaked to perform
fewer lookups, but still not as few as qmail.


-Greg
--
Greg Andrews West Coast Online
Unix System Administrator 5800 Redwood Drive
gerg@wco.com Rohnert Park CA 94928
(yes, 'greg' backwards) 1-800-WCO-INTERNET

D. J. Bernstein wrote:
> If a burst in mail traffic slows down web browsing, the problem is
> certainly not bandwidth. The problem is your CPU. The solution, as
> documented in the man pages, is to run qmail-start with a low priority.

I deliver my big list (40K) users with concurrencyremote = 255 and
qmail-start at nice 4. Works very well. It's great watching qmail fly
open so many sessions. Now, if it could fly open 500+ connections, I'd
be all set :-)

This is running on BSD/OS 3.0, a single Barracuda 2 GB fast/wide with
tagged queuing, on a Pentium 100, 128 MB. It's amazing watching qmail
zoom along on this relatively mundane system versus how sendmail would
putter along.

Evan

> Qmail is almost certainly using up the network so
> thoroughly when it's running that other people trying to use the
> network have to be pretty much scrogged.

Nonsense.

Let's say you're running at concurrency 255 (maximum), with messages of
size 3K (larger than average), at 10 seconds per SMTP connection
(smaller than average).

That's not even half of a T1 line.

If a burst in mail traffic slows down web browsing, the problem is
certainly not bandwidth. The problem is your CPU. The solution, as
documented in the man pages, is to run qmail-start with a low priority.

> Running this number of
> concurrent TCP connections effectively disables all TCP's fairness
> algorithms, which are what keep the network usable for everybody.

Running concurrent TCP connections _improves_ the situation, since it
provides helpful information to the network layer. The only problem is
to implement cross-connection congestion control. SMOP. There's an I-D
on this topic.

---Dan
Let your users manage their own mailing lists. http://pobox.com/~djb/qmail.html

> Nonsense. The core of the network doesn't take the slightest notice of
> your tiny bumps in traffic. You might be able to overload your local
> router if you have dozens of mailing list machines running full blast,
> but in the real world you would have succumbed to HTTP months earlier.
>
> Practically every outage is caused by the hosts, not by the network.

These are some astonishing assertions. They contradict my experience.

Note that HTTP is not as bad as qmail because there are only four
connections per user. That is probably already causing problems, but
there is a world of difference between four and twenty. An
application trying to run four connections over a slow link will stop
much more easily than one trying to run *twenty* (much less 255).

> Let's say you're running at concurrency 255 (maximum), with messages of
> size 3K (larger than average),

OK.

> at 10 seconds per SMTP connection (smaller than average).

10 seconds? This seems surprising, but looking at old flow data, I
see you're right. Does that include hung connections? What's the
median length when using qmail? Since the slow guys will be isolated
in their own qmail subprocesses, average doesn't say quite enough.

> That's not even half of a T1 line.

1) Not everybody has a T1 line. When I arrived at my last employer,
they only had switched-56. Many sites only have 28.8.

2) There are many sites going into "the core." E.g., 100 qmails
on a transatlantic T3 would be unpleasant.

3) The problem is not isolated to qmail; there are other applications
that strain or avoid congestion control. I'm working on the other
guys, too, fair *is* fair....

> Running concurrent TCP connections _improves_ the situation, since it
> provides helpful information to the network layer.

Running concurrent TCP connections provides *misleading* information
to the transport layer. TCP's unit of scheduling is the single TCP
session; there is a basic assumption in TCP design that applications
will use a small number of TCP sessions. If every application used a
single session and TCP worked perfectly (which, of course, it does
not), applications would see roughly equal service from bottleneck
links.

> The only problem isto implement cross-connection congestion
> control. SMOP.

If you implemented either full congestion control or some plausible
way of making TCP congestion control work with parallelism (gotta be
gobs of ways - just to throw one out, only spawning off things that
require more than a certain amount of time, up to
concurrencyparallel?), I would shut up.

> There's an I-D on this topic.

Which one? There are a large number, and none of the current obvious
choices - the ones with your name on them - seem to talk about this.

Jon

Jon Kay writes:
> TCP's unit of scheduling is the single TCP session;

This is an implementation issue, not a protocol issue.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)