Mailing List Archive

On Mon, March 10 1997, gkm-qmlist@moe.gawd.mb.ca wrote:

>As an ISP we have occastion to send an Email to all the users...
>Having over 20,000 users, this would be quite the Emailing, so we use the
>POPbulletin feature of Qualicom's popper.
>Three solutions come to mind:
>1. LOTS of hard links to a single Email.
>2. Lots of Emails.
>3. Recreate how Qpop's bulletin feature works.
>
>Anyone made a patch for #3?
>And has anyone tried #1, would the OS even allow that many hard links?

inode data type for these different OS's:

Solaris 5.5 short /usr/include/sys/fs/ufs_inode.h
SunOS 4.1.4 short /usr/include/ufs/inode.h
linux 2.0.27 ushort twisted path (if I grok this):
/usr/include/linux/fs.h ->
/usr/include/linux/types.h ->
/usr/include/asm/posix_types.h

If you have a different system, poke around in /usr/include. If you run
a Sun box then it looks like hard links are a bad way to go.

Bill

> >As an ISP we have occastion to send an Email to all the users...
> >Having over 20,000 users, this would be quite the Emailing, so we use the
> >POPbulletin feature of Qualicom's popper.
> >Three solutions come to mind:
> >1. LOTS of hard links to a single Email.
> >2. Lots of Emails.
> >3. Recreate how Qpop's bulletin feature works.
> >
> >Anyone made a patch for #3?
> >And has anyone tried #1, would the OS even allow that many hard links?

On BSD/OS, which is derived from 4.4BSD, the link limit is 32767, a
parameter called LINK_MAX, so you could do it, but it sure would be ugly.

I concur that #3 is the best idea. Qpop's implementation is pretty
simple, you could probably scavenge some code.

Regards,
John Levine, johnl@iecc.com, http://iecc.com/johnl, Trumansburg NY
Primary Perpetrator of "The Internet for Dummies"
and Information Superhighwayman wanna-be

Mark Lillywhite writes:

> Hmm... I don't think I'd like to do 20,000 hard links, or even 1000 links
> to 20 files - you'll run out of inodes eventually, anyway, and it's not
> a very flexible solution. Also, is it OK just to stick some random message
> ID in the 'new' directory?

Actually, you're only using 1, and 20 inodes respectively. Hard links
just duplicate directory entries. Symbolic links create an inode.
And yes, you can stick pretty much anything you want, although it's
wise to use date.$$.site-wide-unique-string.

> this case, though, I would like a bulletin system that's a bit more
> flexible - ie, I'd like to be able to set up a bulletin that specifies
>
> Date it's valid from
> Date it's valid to
> People it should go to (ie, not EVERYONE)

For the date it's valid from, you simply create the links on that
date. For the date it's valid to, you simply delete any remaining
links on that date. And for people it should go to, you make a list
of them in a file.

> I'm not familiar with qualcomm's pop bulletin facility so if I'm about
> to duplicate something, please forgive me. But I have a need for just
> such a facility, so unless someone else does it first, I might have a go
> at it some time in the near future. I guess the easiest thing to do
> would be to have a separate program which is invoked before qmail-pop3
> which checks to see if a bulletin should be sent to the user, and if so
> to add that file to the 'new' directory. Then start qmail-pop.

Sigh. If you must complicate things, stuff the mail into the maildir
in the /bin/checkpassword program. Make a copy, make a link, whatever.

> Has anyone looked at IMAP support for qmail?

Yes. Could be done. I have little need for it myself, although I'd
be happy to program it for money.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

Mark Lillywhite writes:
> Sure - all that's pretty obvious. What I'm talking about is making it
> easy to manage. I still wonder about making 20,000 links; it's still
> going to hit the server really hard really instantaneously,

"really instantaneously"? I wonder what you mean by that.

> and in my
> experience of those 20,000 customers there is going to be a goodly
> number who never check e-mail. It also means you can't withdraw a
> bulletin very easily (you could use 'find' to find the links to the
> inode of the original bulletin etc etc).

Delete all unread bulletins older than 30 days:

find /home -name \*.bulletin -mtime +30 | grep /Maildir/ | xargs rm

How much easier than that do you want it to be?

> That way it's easy to withdraw a bulletin, and you don't wipe out
> an already busy machine with 20,000 hard link commands in one hit.

If you're concerned about the load on the machine, enter your
bulletins at night, or else ``nice'' the program that creates them.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

> I agree that the FS is a great database - for some applications. (just
> like an RDBMS is a horrible choice for some applications). I think in
> this case, though, I would like a bulletin system that's a bit more
> flexible - ie, I'd like to be able to set up a bulletin that specifies
>
> Date it's valid from
> Date it's valid to
> People it should go to (ie, not EVERYONE)
>
> I'm not familiar with qualcomm's pop bulletin facility so if I'm about
> to duplicate something, please forgive me.

Qpopper has a very simple scheme -- bulletins are numbered sequentially
and put in /var/bulletins with names like 1-crash, 2-reboot,
3-phone_problems, and so forth. (Anything in the name after the digit
string doesn't matter.) Every POP user has a file called something like
.qpopper which lists the highest numbered bulletin that the user has read.
When a POP session starts, qpopper looks at .qpopper and at the bulletin
directory, treats unread bulletins like new mail, and updates .qpopper
when done.

This way, when there's a new bulletin, you just dump it in the bulletin
directory with an appropriate name. When the bulletin's out of date, you
delete it.

Advantages over treating as mail messages:

-- much faster to create bulletins, don't need 20K links and 20K
directory entries
-- obsolete bulletins can be deleted easily
-- inactive accounts don't fill up with stale bulletins

If you want multiple bulletin categories, I'd suggest making multiple
bulletin directories, with symlinks from the Maildirs to the bulletin
directories to mark which users are interested in which bulletin
categories. On the other hand, as soon as you get that complicated,
I wonder whether local newsgroups wouldn't be more appropriate.

Regards,
John Levine, johnl@iecc.com, http://iecc.com/johnl, Trumansburg NY
Primary Perpetrator of "The Internet for Dummies"
and Information Superhighwayman wanna-be

> > and in my
> > experience of those 20,000 customers there is going to be a goodly
> > number who never check e-mail. It also means you can't withdraw a
> > bulletin very easily (you could use 'find' to find the links to the
> > inode of the original bulletin etc etc).
>
>Delete all unread bulletins older than 30 days:
>
>find /home -name \*.bulletin -mtime +30 | grep /Maildir/ | xargs rm
>
>How much easier than that do you want it to be?
>
> > That way it's easy to withdraw a bulletin, and you don't wipe out
> > an already busy machine with 20,000 hard link commands in one hit.
>
>If you're concerned about the load on the machine, enter your
>bulletins at night, or else ``nice'' the program that creates them.

The filesystem is a wonderful database. I've written many little
applications that make use of it. And none of them scaled too well.
The link-per-bulletin-per-user scheme works fine for maybe up to 1000
users. But:
a) link(2) is an expensive system call, because metadata writes
are synchronous on most filesystems.
b) YMMV, but every FS has a static limit on the number of links
to an inode. On linux (ext2) it's 32,000. I believe
on UFS/FFS it's 32767, but I don't have that source
in front of me.
c) I don't want to contemplate what programs such as fsck, cpio,
tar, etc. will do when they find an inode, or several
inodes (one for each bulletin) with such link counts.

In short, let's call a kluge a kluge. Just because something can be
easily put together using existing tools doesn't mean you shouldn't
stop an examine the overall engineering of the system.

> b) YMMV, but every FS has a static limit on the number of links
> to an inode. On linux (ext2) it's 32,000. I believe
> on UFS/FFS it's 32767, but I don't have that source
> in front of me.

#!/usr/bin/perl
# create bulletins from stdin in the current directory, which is
# presumed to be on the same filesystem as every user in /etc/passwd.
@bulletin = <STDIN>;
$time = time;
$i = -1;
$j = 500;
while(scalar(@user = getpwent())) {
if ($j >= 500) {
$j = 0;
$i++;
open(BULL, ">bulletin.$i");
print BULL @bulletin;
close(BULL);
}
next if !-d "$user[7]/Maildir";
link("bulletin.$i","$user[7]/Maildir/new/$time.$$.bull");
}

Don't optimize too early. If the above code turns out to require too
much resources, THEN create a bulletin system. Really, folks, this
principle is laid out in EVERY software engineering book.

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

John R. Levine writes:
> > Delete all unread bulletins older than 30 days:
> >
> > find /home -name \*.bulletin -mtime +30 | grep /Maildir/ | xargs rm
> >
> > How much easier than that do you want it to be?
>
> You forgot the line:
>
> (wait three hours to crawl through 20,000 Maildirs and 60,000 subdirs)

Sure. The three hours from 3:00 AM to 6:00 AM. Who cares except for
a few night owls?

> > If you're concerned about the load on the machine, enter your
> > bulletins at night, or else ``nice'' the program that creates them.
>
> Or else, consider a different technique that doesn't require 60,000
> synchronous disk writes to install a bulletin. (Link updates the directory
> contents, the directory's inode, and the linked inode). At a brisk 40
> operations per second, that's 25 minutes. Maybe I'm impatient, but I
> think that's slow enough to be worth inventing a different mechanism.

Try the script I just wrote THEN complain when it doesn't work. Don't
complain before you haven't tried it. It just might work!

--
-russ <nelson@crynwr.com> http://www.crynwr.com/~nelson
Crynwr Software sells network driver support | PGP ok
521 Pleasant Valley Rd. | +1 315 268 1925 voice | Peace, Justice, Freedom:
Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | pick two (only mostly true)

> Delete all unread bulletins older than 30 days:
>
> find /home -name \*.bulletin -mtime +30 | grep /Maildir/ | xargs rm
>
> How much easier than that do you want it to be?

You forgot the line:

(wait three hours to crawl through 20,000 Maildirs and 60,000 subdirs)

> If you're concerned about the load on the machine, enter your
> bulletins at night, or else ``nice'' the program that creates them.

Or else, consider a different technique that doesn't require 60,000
synchronous disk writes to install a bulletin. (Link updates the directory
contents, the directory's inode, and the linked inode). At a brisk 40
operations per second, that's 25 minutes. Maybe I'm impatient, but I
think that's slow enough to be worth inventing a different mechanism.

Regards,
John Levine, johnl@iecc.com, http://iecc.com/johnl, Trumansburg NY
Primary Perpetrator of "The Internet for Dummies"
and Information Superhighwayman wanna-be

>I want the flexibility to post short-term bulletins. Some bulletins will
>be valid for only 20 minutes. Eg, "News server is down for upgrade,
>sorry about that". I certainly don't want that message sent to 20,000
>people when it's likely to be valid for a very short time. At present I
>can't make small announcements like that 'cos it's going to hit the
>server too hard.
>
>I also want to be able to post a bulletin and forget about it. The 'find'
>command is way more restrictive in the setting of bulletin timings than
>is appropriate.
>
>Seems like I'll be making a lot of compromises to keep everything in the
>filesystem, rather than doing a few simple things (like qpopper seems to
>do) to make it all sweet. If I was running a database and I was told I'd
>have to do a sequential search of 20,000 user directories, and each
>directory had, say, 20 directory entries, well I'd say "that's a search
>of 400,000 directory entries to delete a single bulletin, you gotta be
>outta your mind".
>
>I think the qualcomm qpopper as described in another message is probably

Certainly the way that the qualcomm popper handles bulletins is nice. That
is, only incorporate the bulletins into the users list when that user
fetches mail.

That method could be applied to qmail-pop3d. Namely, when qmail-pop3d starts
for a given user, invoke the code that determines wether that user has seen
the
latest bulletins and if not, create the links just for that user.

Typically it'll mean that the links are only made for users who collect mail
while the bulletin is active and they disappear shortly as they are
deleted from the users directory. No finds should be necessary either.

I'd sort of be inclined to use symlinks and a bit of symlink detection
technology in qmail-pop3d.


Regards.

Russell Nelson wrote:
> Yes. Could be done. I have little need for it myself, although I'd
> be happy to program it for money.

Where it would be nice (IMHO) is so you could use pine on a qmail system
with Maildir delivery. I sort of get around that by running popclient
to move my mail from a Maildir to a Mailbox if I'm reading on the UNIX
boxes, otherwise I read from my mail reader at home.

Evan

Mark Lillywhite:
> I want the flexibility to post short-term bulletins. Some bulletins
> will be valid for only 20 minutes. Eg, "News server is down for
> upgrade, sorry about that". I certainly don't want that message sent
> to 20,000 people when it's likely to be valid for a very short
> time.

In other words, you want something like /etc/motd for pop mail
clients?

That's something very different from a bulletin -- you probably only
want one of these to exist at any time. [.Or you really expect people
to grovel through a whole batch of temporary messages during that
brief of a period of time?]

--
Raul

RM> == Raul Miller <rdm@rdm.legislate.com>

RM> That's something very different from a bulletin -- you probably only
RM> want one of these to exist at any time. [.Or you really expect people
RM> to grovel through a whole batch of temporary messages during that
RM> brief of a period of time?]

I think he wants a series of "motd-ish" items, which can be added to or
subtracted from independently, and which will show up in the user's
mailbox at most once per user.

These might include bulletins like:

1-wwwupgrade (We will be upgrading the WWW server on March 30.)
2-shelldowntime (Shell server will be down Sunday March 15 0400-0600.)
3-newpricing (New pricing plans available, see [url])
4-newsdown (We know news is down, we're working on it.)
5-tshirts (T-shirts with our logo now available)

When news is fixed, delete 4-newsdown. When March 15 has come and gone,
delete 2-shelldowntime. After the WWW server is upgraded, delete
1-wwwupgrade. Assuming the T-shirts don't run out, you could keep the
other two around for a while.

New accounts may get their "bulletinrc" initialized so that all previous
bulletins are considered "read", or not; it depends.

On Mon, 10 Mar 1997, John R Levine wrote:

: Qpopper has a very simple scheme -- bulletins are numbered sequentially
: and put in /var/bulletins with names like 1-crash, 2-reboot,
: 3-phone_problems, and so forth. (Anything in the name after the digit
: string doesn't matter.) Every POP user has a file called something like
: .qpopper which lists the highest numbered bulletin that the user has read.
: When a POP session starts, qpopper looks at .qpopper and at the bulletin
: directory, treats unread bulletins like new mail, and updates .qpopper
: when done.

Oh, yes, it's a nice feature, if qpopper don't segfault for you... my
experience with qpopper (in Linux) says that it's a very good popper,
but *whitout* any feature or option... all of them make it segfault :(

Regards,
--
Roberto Lumbreras Pastor * mailto:rover@lander.es
Lander Internet - Spain * http://www.lander.es/
Tel +34 1 556.28.83 * Fax +34 1 556.30.01