Mailing List Archive

Re: pipelining vs. single-message delivery
> qmail's current behaviour is similar to netscape's behaviour where it
> opens multiple tcp connections to suck down multiple images at the same time.

In both cases, parallel connections produce results for the user as
quickly as possible.

> But as has been shown by the W3C
> <http://www.w3.org/pub/WWW/Protocols/HTTP/Performance/Pipeline.html>,
> pipelined connections are far more efficient on bandwidth.

But, gee, most servers don't support pipelined connections.

Nor compression, which is what saves most of the bandwidth.

> Pipelining is available as an extension to SMTP.

So is compression, with XLZW. So what? Are you trying to deliver mail in
a fantasy world, or in the real world?

Anyway, SMTP ``pipelining'' is nowhere near as effective as HTTP
pipelining---it doesn't do anything about the per-message RTTs. Try
QMTP, or UUCP, if you really want to reduce latency.

> So it's reasonable to think that
> it wouldn't be hard to put pipelining into the major MTAs

sendmail forks after MAIL FROM. This prevents pipelining.

> At any rate, every time this topic comes up the claim is made that qmail's
> approach is superior regardless of the waste of bandwidth.

The bandwidth loss from parallel connections is wiped out by bandwidth
gains in other areas.

Anyway, here's a specific latency figure: qmail reportedly completed
28000 deliveries around the world in half an hour from a single PC.

> A simplistic approach such as sorting the recipient list by domain and using
> pipelining for recipients at the same domain is likely to be faster than
> what qmail does now.

I'm really not interested in listening to further uninformed speculation
on this topic.

---Dan
Let your users manage their own mailing lists. http://pobox.com/~djb/qmail.html
Re: pipelining vs. single-message delivery [ In reply to ]
On 23 Feb 1997, D. J. Bernstein wrote:

> > qmail's current behaviour is similar to netscape's behaviour where it
> > opens multiple tcp connections to suck down multiple images at the same time.
>
> In both cases, parallel connections produce results for the user as
> quickly as possible.
>
> > But as has been shown by the W3C
> > <http://www.w3.org/pub/WWW/Protocols/HTTP/Performance/Pipeline.html>,
> > pipelined connections are far more efficient on bandwidth.
>
> But, gee, most servers don't support pipelined connections.

Read the link, parallel connections do not produce results for the
user as quickly as pipelined connections. "most servers" are apache,
and 1.2b7 released as of today has the optimized pipelinig support.
Of course this all depends on how one defines "results". Certainly
with pipelined HTTP/1.1 connections you'll get all the data faster,
but partial image loads are sometimes more useful and you'll get those
faster without pipelining.

> sendmail forks after MAIL FROM. This prevents pipelining.

Not really, it just hinders multiple messages on the same connection.
Pipelining is all about when to flush output. It certainly doesn't hinder
multiple recipients of the same message (which right now get handled in
lock-step request-response).

> The bandwidth loss from parallel connections is wiped out by bandwidth
> gains in other areas.

Such as...? This does not mean that parallel connections should be
excluded from experimental research.

> Anyway, here's a specific latency figure: qmail reportedly completed
> 28000 deliveries around the world in half an hour from a single PC.

This is hardly a useful datapoint for making conclusions. You are just
as guilty of baseless speculation.

The only point I am attempting to make here is that I have seen no data
that shows the case is closed as far as parallel connections vs. multiple
recipients goes. In any event, the VERP stuff makes parallel connections
very useful for mailing lists, so the argument for multi-rcpt has to be
really strong.

Dan, if you're not interested in this then don't bother replying.
As you have pointed out, we're all free to patch qmail as we see fit.
If someone else with the time believes my crock of shit more than yours
or just wants to experiment, they'll do it. But so far there's no good
data that supports your view or my speculation.

Dean
Re: pipelining vs. single-message delivery [ In reply to ]
> > But, gee, most servers don't support pipelined connections.
> Read the link, parallel connections do not produce results for the
> user as quickly as pipelined connections.

Pay attention.

``Most servers don't support pipelined connections.''

Clients can't take advantage of a feature that servers don't support.

Note that, even with pipelining, parallel connections still boost speed
when the amount of data transferred exceeds the window size.

> > sendmail forks after MAIL FROM. This prevents pipelining.
> Not really,

Yes, really. The stdio buffers in the parent could contain RCPT and DATA
that are read by the child---or they could contain RSET and then a new
MAIL that has to be read by the parent. The parent has no idea how far
the child has read when the child exits.

> > Anyway, here's a specific latency figure: qmail reportedly completed
> > 28000 deliveries around the world in half an hour from a single PC.
> This is hardly a useful datapoint for making conclusions.

I'm not drawing conclusions. I'm telling you to put up or shut up. You
_haven't_ reduced latency, so don't claim that you _can_ reduce latency.

Declarations of ignorance are uninteresting in performance programming.
If you can do X in time under T, prove it. If you can't, keep quiet.

---Dan
Let your users manage their own mailing lists. http://pobox.com/~djb/qmail.html
Re: pipelining vs. single-message delivery [ In reply to ]
On 24 Feb 1997, D. J. Bernstein wrote:

> > > But, gee, most servers don't support pipelined connections.
> > Read the link, parallel connections do not produce results for the
> > user as quickly as pipelined connections.
>
> Pay attention.
>
> ``Most servers don't support pipelined connections.''
>
> Clients can't take advantage of a feature that servers don't support.

Most servers don't support QMTP. By your logic I can't imagine why you
implemented it.

Most MTAs don't even support pipelining as a client, why did you implement
that?

These things have to start somewhere. I'm talking about experimentation
here. I'm not saying "Dan go write this code now" (although other
people have been). I'm saying "I would be really interested in seeing
experimental results showing that qmail's present scheme is in fact
about the best we can do with the mixture of MTAs, link qualities, and
end-hosts out there today". You seem to think it is, I'm not convinced.

Dean
Re: pipelining vs. single-message delivery [ In reply to ]
> Most servers don't support QMTP.

Putting a QMTP server into the package didn't slow the mailer down.
Tossing a QMTP client into qmail-remote would be a different story.

> Most MTAs don't even support pipelining as a client,

Adding pipelining to the server was very cheap: some extra bytes of
data (not enough to push the data segment past one page) and one extra
line of code. The biggest cost is the 16 extra bytes per connection to
say 250-PIPELINING\r\n to the other side.

---Dan
Let your users manage their own mailing lists. http://pobox.com/~djb/qmail.html