Mailing List Archive: Reusing open delivery connections ?

Reusing open delivery connections ?

Mar 5, 1997, 1:02 PM

Post #1 of 11 (1979 views)

I was flipping through the manual for exim, an MTA from which qmail could
take a few hints, and noticed that they do a nice optimization by reusing
open SMTP connections. That is, after delivering a message, if there's
another message going the same place, it sends an RSET and uses the same
open connection.

Before you flame me, this is NOT the same thing as sending the same copy
of a message to multiple recipients, which I happen to agree is a bad idea
if only because it breaks the wonderful automatic list cleaning made
possible by the per-receipient sender addresses.

This is two separate deliveries which may or may not be the same message,
but instead of tearing down a virtual circuit and immediately setting up a
new one to the same place, it reuses the one that's already open. It
looks to me like this is clearly a win, since you avoid the teardown and
setup packets, and RSET/250 is one round trip, whereas QUIT/221 and then
220/EHLO/250 is two and half.

Is there any structural reason why qmail couldn't do this? Exim looks at
the outgoing queue and if it sees a suitable message, re-execs itself
with a flag so it knows its connection is already open.

Regards,
John Levine, johnl@iecc.com, http://iecc.com/johnl, Trumansburg NY
Primary Perpetrator of "The Internet for Dummies"
and Information Superhighwayman wanna-be

Re: Reusing open delivery connections ? [ In reply to ]

Mar 5, 1997, 1:20 PM

Post #2 of 11 (1969 views)

Hi,

> This is two separate deliveries which may or may not be the same message,
> but instead of tearing down a virtual circuit and immediately setting up a
> new one to the same place, it reuses the one that's already open. It
> looks to me like this is clearly a win, since you avoid the teardown and
> setup packets, and RSET/250 is one round trip, whereas QUIT/221 and then
> 220/EHLO/250 is two and half.

Whats the Reason for the RSET?

Greetings
Bernd
--
(OO) -- Bernd_Eckenfels@Wittumstrasse13.76646Bruchsal.de --
( .. ) ecki@{inka.de,linux.de,debian.org} http://home.pages.de/~eckes/
o--o *plush* 2048/93600EFD eckes@irc +4972573817 BE5-RIPE
(O____O) If privacy is outlawed only Outlaws have privacy

Re: Reusing open delivery connections ? [ In reply to ]

Mar 5, 1997, 2:19 PM

Post #3 of 11 (1954 views)

>I was flipping through the manual for exim, an MTA from which qmail could
>take a few hints,

And vice versa...

>and noticed that they do a nice optimization by reusing
>open SMTP connections. That is, after delivering a message, if there's
>another message going the same place, it sends an RSET and uses the same
>open connection.

And how much does this optimization cost in terms of increased code
size, overhead and complexity, and is the benefit worth the price?
Offhand, I'd be surprised if it was really a win.

>This is two separate deliveries which may or may not be the same message,
>but instead of tearing down a virtual circuit and immediately setting up a
>new one to the same place, it reuses the one that's already open. It
>looks to me like this is clearly a win, since you avoid the teardown and
>setup packets, and RSET/250 is one round trip, whereas QUIT/221 and then
>220/EHLO/250 is two and half.

It's not clear to me. For example, how does one scan the queue for
messages going to the same MX? How often will connections be reused?
Often enough to pay the cost of reusing them? Sendmail spends a lot of
time looking up MX's and sorting by MX and it's not at all clear that
that actually makes it faster than qmail's unsorted parallel
delivery--in fact, by every comparison I've seen, it's slower.

-Dave

Re: Reusing open delivery connections ? [ In reply to ]

Mar 6, 1997, 12:31 PM

Post #4 of 11 (1958 views)

John R. Levine <johnl@iecc.com> writes on 5 March 1997 at 15:02:06 -0500

> This is two separate deliveries which may or may not be the same message,
> but instead of tearing down a virtual circuit and immediately setting up a
> new one to the same place, it reuses the one that's already open. It
> looks to me like this is clearly a win, since you avoid the teardown and
> setup packets, and RSET/250 is one round trip, whereas QUIT/221 and then
> 220/EHLO/250 is two and half.
>
> Is there any structural reason why qmail couldn't do this? Exim looks at
> the outgoing queue and if it sees a suitable message, re-execs itself
> with a flag so it knows its connection is already open.

Since nobody seems to have answered your specific question, here
goes. Yes, there is a structural reason this is difficult. Qmail
spawns off independent tasks for each delivery. The spawner and all
the other tasks depend on that independence. If you wish to have the
delivery tasks rescan the queue (expensive locally) and start sending
another message, there has to be a coordination mechanism between
them, which is currently completely absent. I'd call that a
structural reason.

Re: Reusing open delivery connections ? [ In reply to ]

nelson at crynwr

Mar 6, 1997, 9:16 PM

Post #5 of 11 (1956 views)

David Dyer-Bennet writes:
> > Is there any structural reason why qmail couldn't do this? Exim
> > looks at the outgoing queue and if it sees a suitable message,
> > re-execs itself with a flag so it knows its connection is
> > already open.
>
> Since nobody seems to have answered your specific question, here
> goes. Yes, there is a structural reason this is difficult. Qmail
> spawns off independent tasks for each delivery. The spawner and all
> the other tasks depend on that independence. If you wish to have the
> delivery tasks rescan the queue (expensive locally) and start sending
> another message,

I looked at it. There's no structural reason why qmail-remote
couldn't be given several deliveries to the same host. I don't have
the time right now to suggest somewhere to change qmail-send.c and
qmail-[rl]spawn.c, but from my casual inspection, it's possible to
tell them to accept multiple messages. The mailing list would have to
be sorted in hostname order, so that qmail-send.c could simply peek at
the next addressee in the queue/remote file to see if the host matches
the current addressee.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

Re: Reusing open delivery connections ? [ In reply to ]

Mar 7, 1997, 9:13 AM

Post #6 of 11 (1956 views)

At 07:31 PM 3/6/97 -0000, David Dyer-Bennet wrote:
>John R. Levine <johnl@iecc.com> writes on 5 March 1997 at 15:02:06 -0500
>
> > Is there any structural reason why qmail couldn't do this? Exim looks at
> > the outgoing queue and if it sees a suitable message, re-execs itself
> > with a flag so it knows its connection is already open.
>
>Since nobody seems to have answered your specific question, here
>goes. Yes, there is a structural reason this is difficult. Qmail

But note in THOUGHTS around line 276 that the changes alluded to are
described as "easy".

Regards.

Re: Reusing open delivery connections ? [ In reply to ]

Mar 12, 1997, 4:06 PM

Post #7 of 11 (1950 views)

> That is, after delivering a message, if there's
> another message going the same place, it sends an RSET and uses the same
> open connection.

Translation: Next time AOL goes down, flood it with traffic as soon as
you notice it's back up. Watch it go down again. Neat.

I think I'll stick to a regular retransmission schedule.

---Dan
Let your users manage their own mailing lists. http://pobox.com/~djb/qmail.html

Re: Reusing open delivery connections ? [ In reply to ]

Mar 12, 1997, 10:23 PM

Post #8 of 11 (1951 views)

> > That is, after delivering a message, if there's
> > another message going the same place, it sends an RSET and uses the same
> > open connection.
>
> Translation: Next time AOL goes down, flood it with traffic as soon as
> you notice it's back up. Watch it go down again. Neat.

Uh, maybe I'm unusually dense today, but how does sending sequential
messages down a single connection hammer the receiver any worse than
sticking a bunch of closes and opens in between them?

Regards,
John Levine, johnl@iecc.com, http://iecc.com/johnl, Trumansburg NY
Primary Perpetrator of "The Internet for Dummies"
and Information Superhighwayman wanna-be

Re: Reusing open delivery connections ? [ In reply to ]

qmail at timothy

Mar 13, 1997, 4:14 AM

Post #9 of 11 (1956 views)

> Uh, maybe I'm unusually dense today, but how does sending sequential
> messages down a single connection hammer the receiver any worse than
> sticking a bunch of closes and opens in between them?

SIngle connection: open the connection, blast data, machine load gets high,
machine falls over.

multiple connections: blast single messages, machineload gets high, starts
refusing mail, mail backs off.

I think ;o)

Timothy

Re: Reusing open delivery connections ? [ In reply to ]

qmail at timothy

Mar 13, 1997, 6:57 AM

Post #10 of 11 (1950 views)

"Justin M. Streiner" <streiner@co.iup.edu> wrote:
>
> > SIngle connection: open the connection, blast data, machine load gets high,
> > machine falls over.
> > multiple connections: blast single messages, machineload gets high, starts
> > refusing mail, mail backs off.
>
> The point that was being made was that there's a limit to the amount of
> data you can shoot across one TCP connection. It's very unlikely for one
> SMTP connection to bring a machine to its knees.

What makes you so special that you are the only one with mail waiting
for that single host?

Timothy

Re: Reusing open delivery connections ? [ In reply to ]

agulbra at troll

Mar 13, 1997, 8:38 AM

Post #11 of 11 (1958 views)

Timothy Hunt <qmail@timothy.org.uk>
> What makes you so special that you are the only one with mail waiting
> for that single host?

The best secondary MX is, to a good approximation, the only host with
mail waiting.

--Arnt