Mailing List Archive

Hi,

I use qmsmac for virtual hosts. (I converted because it's simpler).
To make something automated work with is a joke as my /etc/aliases (a
symlink) looks like this:
user1: blah
user1@foo.bar: blah2
user3@foo2.bar: blah@blah.com
You get the idea. It's very simple. But you'll have to do carefull planning
as converting to qmail with such a large amount of users won't be that simple.
Especially converting your tuned sendmail.cf and /etc/aliases (which in
qmail can't have a |). If you don't give shell accounts on your pop server
(except for staff of course) you should be ok.

Andi
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Andi Gutmans - Computer Science, Technion

Email: andi@il.eu.org andi@vipe.technion.ac.il
Web: http://andi.rifkin.technion.ac.il

PGP public key: finger andi@vipe.technion.ac.il

Mark Delany writes:
> At 03:23 AM 3/9/97 -0600, gkm-qmlist@moe.gawd.mb.ca wrote:
> >How much more efficent is Qmail's popper over Qualicom's popper?
> > (I'm guessing ALOT better, but would be nice to have stats)
>
> In general terms you have an additional fork/exec or two for
> authentication, but qmail-pop3d does not have to do any mailbox
> manipulation at all. qpopper re-writes mailboxes, inserts UIDLs,
> writes a Status: line, merges new mail and so on. qmail-pop3d
> doesn't need to do any of that.

It's hard to say whether qmail-pop3d is more efficient in toto. It
does a lot less, and is (I presume) a lot smaller than qpopper. Using
Maildir saves a lot of code that would otherwise be required. But you
have the usual mh/mbox mailbox comparison to make. Jeff Hayward
<J.Hayward@utexas.edu> made the following analysis sent to me in
private mail which I hope he will forgive for my publishing it:

In a typical POP transaction, the entire mailbox is read once and
truncated to zero-length. For a Mailbox file, this amounts to the i/o
to open and read the file, then update the inode. For a maildir
mailbox, it involves open, read, update inode, update directory *for
each message*. The inode and directory updates really hurt, as
they're synchronous writes. I also have to worry about the
pathological cases (I get users who keep 2000 messages in their inbox,
set "retrieve new mail, keep old on server", and poll every 5
minutes). So I don't think maildir is a win for POP.

> The biggest boon is that you get rid of the .lock problem as there
> is no need for a pop server to lock with Maildir.

This *is* a serious improvement. I had one site tearing their hair
out over lost pop locks. However, you might get the same piece of
mail delivered twice. That would only happen if you had deleted some
mail, then failed to issue the quit command. Mail is not deleted if
the daemon times out.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

> Mark Delany writes:
> > At 03:23 AM 3/9/97 -0600, gkm-qmlist@moe.gawd.mb.ca wrote:
> > >How much more efficent is Qmail's popper over Qualicom's popper?
> > > (I'm guessing ALOT better, but would be nice to have stats)
> >
> > In general terms you have an additional fork/exec or two for
> > authentication, but qmail-pop3d does not have to do any mailbox
> > manipulation at all. qpopper re-writes mailboxes, inserts UIDLs,
> > writes a Status: line, merges new mail and so on. qmail-pop3d
> > doesn't need to do any of that.
>
> It's hard to say whether qmail-pop3d is more efficient in toto. It
> does a lot less, and is (I presume) a lot smaller than qpopper. Using
> Maildir saves a lot of code that would otherwise be required. But you
> have the usual mh/mbox mailbox comparison to make. Jeff Hayward
> <J.Hayward@utexas.edu> made the following analysis sent to me in
> private mail which I hope he will forgive for my publishing it:
>
> In a typical POP transaction, the entire mailbox is read once and
> truncated to zero-length. For a Mailbox file, this amounts to the i/o
> to open and read the file, then update the inode. For a maildir
> mailbox, it involves open, read, update inode, update directory *for
> each message*. The inode and directory updates really hurt, as
> they're synchronous writes.
Not in my case, we're using Sun Ultras for mail, with a fiber channel RAID
array. Writes are nice and fast.(the RAID has a battery onboard to hold
cached data till it gets written)
> I also have to worry about the
> pathological cases (I get users who keep 2000 messages in their inbox,
> set "retrieve new mail, keep old on server", and poll every 5
> minutes). So I don't think maildir is a win for POP.
Yes, we have alot of those users as well. What we find the popper spending
most of it's time doing is scanning the mailbox, looking for the word "From "
and the status so it can say there's no new messages. What does qm's pop do?
Just scan the directory? Or does it have to do more?
I see the normal state of things being alot of Emails in the cur directory, a
few in the new directory(or none), and nothing in tmp. The user will be
coming in very regular(anywhere from 30 seconds to 5 minute) to check for new
mail.
Anyone know what the mail client actually does to check for new mail?
Well, I just answered my own question. I put in some syslogging into qmail's
popper and this is what Netscape does:
Logs in, then does a stat, then a quit. As long as there's no new mail,
that's all it does. If it detects new mail (or at least different) it does a
list and a uidl on top of the stat. So far, I don't think the popper has to
open any files, if it doesn't want to, just some directory scans and filesize
stats.
When netscape is told to actually get the mail, it does:
Mar 9 14:26:13 wormwood popper: doing (STAT)
Mar 9 14:26:13 wormwood popper: doing (LIST)
Mar 9 14:26:13 wormwood popper: doing (UIDL)
Mar 9 14:26:13 wormwood popper: doing (RETR 2)
Mar 9 14:26:13 wormwood popper: doing (RETR 3)
Mar 9 14:26:13 wormwood popper: doing (QUIT)
Leaving the old Emails completely alone.
Then it does a stat, list, uidl, quit, I guess to get it's bearing again and
after that it's back to stat, quit.

Now, I haven't gone though the actual popper code to see what each of these
commands involve, but an opening of the files shouldn't be needed, I don't
think.
>
>
Comments, everyone?

Greg
--
__ Where's the kaboom? There was
/\_| supposed to be an earth shattering kaboom!!
(/:.--
/_...-"-.
( )_...._/
|'( o o|
\ \ _.'\ Greg K. Moeller
/ :\/[___]==-
(.'_|-8'
<'8||\> Fax: 204/783-7523
|::|__ gkm@gawd.mb.ca
|:.__)
(_)

gkm-qmlist@moe.gawd.mb.ca writes:

> Yes, we have alot of those users as well. What we find the popper
> spending most of it's time doing is scanning the mailbox, looking
> for the word "From " and the status so it can say there's no new
> messages. What does qm's pop do? Just scan the directory?

It reads both new and cur.

> Now, I haven't gone though the actual popper code to see what each of these
> commands involve, but an opening of the files shouldn't be needed, I don't
> think.

Right. The UIDL information comes from the filename.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

At 03:23 AM 3/9/97 -0600, gkm-qmlist@moe.gawd.mb.ca wrote:
>
>We're running a rather large sendmail sendmail site that I'd LOVE to get
rid of, both for security reasons, but also for pop efficency
reasons.(during prime-time, we can have over 11000 pop accesses per hour)

Generally you'd expect the total system load to decline - especially if you
use Maildir for pop boxes and the average user does not retain 1000s of
emails on the server.

>First, the stats:
>over 900 virtual domains.
>around 2.6 Gig in Email in over 25,000 mailboxes
>over 11,000 pop accesses per hour during prime-time.
>
>Now, the concerns:
>Can Qmail be convinced to do virtual domains as easily (or close) as
sendmail?
> Ie: Just MX record, and mailmapping's all in one file.
> (Was thinking maybe qmsmac could do it?)

It's actually easier than that. A one line entry in
/var/qmail/control/virtualdomains
and (optionally) .qmail file. An example? Say I want all of gawd.mb.ca
to go into the pop box of user mbca. Then I'd need this:

echo gawd.mb.ca:mbca >>/var/qmail/control/virtualdomains

Done.


>How much more efficent is Qmail's popper over Qualicom's popper?
> (I'm guessing ALOT better, but would be nice to have stats)

In general terms you have an additional fork/exec or two for authentication,
but qmail-pop3d does not have to do any mailbox manipulation at all. qpopper
re-writes mailboxes, inserts UIDLs, writes a Status: line, merges new mail
and so on. qmail-pop3d doesn't need to do any of that. The biggest boon is
that you get rid of the .lock problem as there is no need for a pop server
to lock with Maildir.

>None of the users will have control of the virtual domains, they don't
want it, what we have now is a WWW based robot that edits the sendmail.db
file and reruns the command to remake the database.

If they also have no access to their home (or .forward) then life is
especially pleasant for you.


Regards.

>> I also have to worry about the
>> pathological cases (I get users who keep 2000 messages in their inbox,
>> set "retrieve new mail, keep old on server", and poll every 5
>> minutes). So I don't think maildir is a win for POP.
>Yes, we have alot of those users as well. What we find the popper spending
>most of it's time doing is scanning the mailbox, looking for the word "From "
>and the status so it can say there's no new messages. What does qm's pop do?
>Just scan the directory? Or does it have to do more?

It does a readdir() then a stat() call for each file (ie mail).

(As an aside, it's sort of a pity about doing the stat() as it only needs it
for sorting and size. If performance were really an issue, the size could be
encoded into the filename and the filname already has a timestamp. This
encoding could occur after a RETR that moves the file to cur)

>I see the normal state of things being alot of Emails in the cur
directory, a
>few in the new directory(or none), and nothing in tmp. The user will be
>coming in very regular(anywhere from 30 seconds to 5 minute) to check for
new
>mail.
>Anyone know what the mail client actually does to check for new mail?

Most do a LIST or UIDL.

>When netscape is told to actually get the mail, it does:
>Mar 9 14:26:13 wormwood popper: doing (STAT)
>Mar 9 14:26:13 wormwood popper: doing (LIST)
>Mar 9 14:26:13 wormwood popper: doing (UIDL)
>Mar 9 14:26:13 wormwood popper: doing (RETR 2)
>Mar 9 14:26:13 wormwood popper: doing (RETR 3)
>Mar 9 14:26:13 wormwood popper: doing (QUIT)
>Leaving the old Emails completely alone.
>Then it does a stat, list, uidl, quit, I guess to get it's bearing again and
>after that it's back to stat, quit.
>
>Now, I haven't gone though the actual popper code to see what each of these
>commands involve, but an opening of the files shouldn't be needed, I don't
>think.

Both LIST and UIDL require a readir() stat() pass. The biggest costs is the
likely to be the stat() calls which start to add up after a while. But it
probably
requires real tests to see what it means on your system.


Regards.

> I also have to worry about the pathological cases (I get users who
> keep 2000 messages in their inbox, set "retrieve new mail, keep old
> on server", and poll every 5 minutes). So I don't think maildir is
> a win for POP.

Personally, I think I'd like maildir to have some kind of partitioning
scheme for current mail. I'm not sure if this should be a has
(perhaps a chronological partition would be good?)

---
Raul