Mailing List Archive

slow starting ( was Re: infinite connections)
-----BEGIN PGP SIGNED MESSAGE-----

On Sat, 8 Mar 1997, I wrote:

> how about a slow-start mechanism? a SMTP transfer goes though a number of
> states: connect, HELO, MAIL, RCPT, DATA, QUIT, close.
> qmail could be attempting to start /remoteconcurrency/ simultaneous
> transfers instantly to one machine. By not attempting to connect to a
> remote MTA if you're already attempting to connect to the same remote
> MTA one might achieve soft-starting. If the remote end can only cope
> with n simultaneous connections then n+1 should cause a deferment.
>
> I'll look and see if this is possible as a hack on my machines here and
> see what the effect is.

okay, I've made an attempt at hacking this into qmail to see what the
effect is. This is a real hack; the code's never going to leave here since
it's only coded for one remote-machine and I use strcat/strcpy etc.

my setup: illuin - this machine: 486/33 , 12MB, linux 1.2.13
ormal - compute server: Dual-pentium 133, 64MB, linux 2.0.28

sending 1700 messages from the compute server to this machine:
- inetd decided that 100 connections at once was too much and closed
the smtp port. fix: inetd under linux has an opitional parameter to
nowait -- the number of connections per minute inetd will accept.
smtp stream tcp nowait.1000 qmaild /usr/sbin/tcpd /var/qmail/bin/tcp-env
/var/qmail/bin/qmail-smtpd

- the load average of the laptop went into double-digits and the system
entered thrash city with 100 inbound smtp transfers. not nice. not nice at
all.

Implementing soft-start where ormal does not make another connection if
there is another qmail-remote waiting for the connection to complete.
this was indistinugishable from the first one in terms of effect on this
machine. basically the effect of the tcp wrappers, tcp-env, and the startup
of qmail-smtpd is quite a long period of time, during which more
connections come in.

implementing soft-start where ormal does not make another connection if
there is another qmail-remote waiting for the HELO to complete. This was
quite interesting since ormal never appeared to run more than one instance
of qmail-remote at once.

testing mail in the opposite direction was delayed whilst I rebuilt the
binaries because the uid/gids aren't syncronised between machines.
#insert plea for non-compiled in values.
In this case I had multiple qmail-remotes going to the faster machine,
which is what I probbaly want.

HOWEVER <<< always one of these.
what I've hacked up proves the concept, the point though is this should
posibly be in the scheduler rather than in the qmail-rspawn/qmail-remote
programs where I've put them. In this way a differet MTA could be
attempted instead of waiting for a delivery to the current MTA.
this is leading to complexity, like PP has, in the scheduler and I'm
desparately trying to avoid that.

RjL
+----------------------------+
| richard@illuin.demon.co.uk | Aut viam inveniam aut faciam
+----------------------------+


-----BEGIN PGP SIGNATURE-----
Version: 2.6

iQCVAgUBMyNYSp6bDk8vHTn1AQGgBwP+O1wd+XnH9EoevYFV2KQtqJMs9Z4jpZc3
cNJSlMJFBTxJSKKa1JeVzDLUbPR7lpU4OyKJEqWZTroY7Y+beZtjbDEDkUu3MuD3
BVMAu7garuSgS95qDTX1h5nyU/bts0W8cIYfyrHDBQCHqoOdY11l73xUZQ/3Yj4d
64exlzGLQ0Y=
=Cl7X
-----END PGP SIGNATURE-----
Re: slow starting ( was Re: infinite connections) [ In reply to ]
> - inetd decided that 100 connections at once was too much and closed
> the smtp port.

Don't use inetd for high loads.

> /usr/sbin/tcpd

Don't use tcpd for high loads.

> - the load average of the laptop went into double-digits

Don't use inetd for high loads.

tcpserver has a concurrency limit; the default limit of 40 should fit
inside your 12MB machine. It's also much faster than inetd+tcp-env. If
you need connection control, use tcpcontrol, which is much faster than
tcpd.

With tcpserver+qmail-smtpd, your 486/33 should be able to keep up with
the Pentium running full blast.

---Dan
Let your users manage their own mailing lists. http://pobox.com/~djb/qmail.html
Re: slow starting ( was Re: infinite connections) [ In reply to ]
On 12 Mar 1997, D. J. Bernstein wrote:

> Don't use inetd for high loads.
> Don't use tcpd for high loads.
> Don't use inetd for high loads.

> tcpserver has a concurrency limit; the default limit of 40 should fit
> inside your 12MB machine. It's also much faster than inetd+tcp-env. If
> you need connection control, use tcpcontrol, which is much faster than
> tcpd.
>
> With tcpserver+qmail-smtpd, your 486/33 should be able to keep up with
> the Pentium running full blast.

I think you're mis-understanding what I was trying to do.
I was trying to see what effect modifying qmail's sending algorithm
would have on a poorly configured machine with limited resources.
ie. this laptop with wrappers.

The important part of the test was not to optimise the receiving
machine, but to look at what I could do on the sending machine to make
life easier for the receiving machine. the first part of the test that
caused inetd to disable the incoming channel has caught me out in the past
and undoubtedly will catch other people in future.

If the[my] mail-relays at work start killing off sendmail and NOTES servers
around campus I'll not be very popular, and since I won't be able to just
dish out binaries because of the compiled-in UID/GID issue I'll have to
consider other solutions.

the biggest bottleneck on this machine is the IDE hard disk, and my
Mailbox format mailboxes.

RjL
+----------------------------+
| richard@illuin.demon.co.uk | Aut viam inveniam aut faciam
+----------------------------+