Mailing List Archive

ideas on filtering e-mail at the site-wide level
[.I posted this to the procmail list, but I think the SA-related, and mail
admin. related issues are more germane to this list, so am re-posting here.]

Recently, I developed a /etc/procmailrc (at the end of this message) script,
as a
prototype for a friend who runs a small not-for-profit group that supports
50/so
e-mail users. The users run the gamut in terms of computer technology
sophistication.
This is the first time I've tried writing a global /etc/procmail script, and
I'm looking for tips to improve the script, and to get some ideas on other
ways
to do things.

The script filters mail as follows:
/var/mail_log/${LOGNAME}.log - procmail log of recent transactions
/var/virus_trap/${LOGNAME} - place where likely viruses are dumped
/var/spam_trap/${LOGNAME} - place where likely spam is dumped
$HOME/spam.mbox - place where marginal spam is dumped

The script also trims the log files as it goes. Although the log files are
useful for debugging when VERBOSE=on, their primary function is to give
a brief audit trail of where recent mail was delivered (via a the
"log abstract" form).

The motivation for creating a global virus trap and spam trap was to lower
the
load on the users. The sysadmin can later put together a cron job that
deletes old virus and spam e-mails, or this could be done on an on-demand
basis in script at some point.

But depositing the logs, virus mail, and definite spam into a system-wide
location presented some difficulties. Mainly, I wanted to make sure those
files are readable by the user who was the intended recipient (and not
by others).

This led to some complications in the /etc/procmailrc script. For example,

#
# If spam found, and above threshold of 10
# dump it in the spam trap.
#
:0
* ^X-Spam-Status: Yes
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*
{
LOCKFILE=${SPAM_TRAP}$LOCKEXT
:0 Wc
* $ ! ? test -f $SPAM_TRAP
| touch $SPAM_TRAP && chown ${LOGNAME}.mail $SPAM_TRAP && chmod 640
$SPAM_TRAP

:0
$SPAM_TRAP
LOCKFILE # release lock
}

Above, two main things are going on:
1. a regional lock file is used, because the script wants to first check
whether
it needs to create the spam destination folder, and then to deposit it
there.
Although the touch/chown/chmod combo can probably be run all the time
without
creating any harmful race conditions, this approach of locking first,
testing
for the mailbox, and delivering seemed more efficient.
2. the touch/chown/chmod combo is needed to ensure that the user has read
access
to the spam mailbox file. At least I think so ....

Questions:

- if the delivery part of this recipe were changed simply to:

:0:
$SPAM_TRAP

can I depend upon procmail to create that file with the recipient as owner
of the file? (the containing directory is owner=root,group=mail and is
writeable
only by owner/group and rx everyone). Or would procmail create it on behalf
of the current user, which is root (or mail)?

- in this section of the recipe:

#
# Invoke spamasassin to check for spam.
#
:0 fw:$SA_LOCK
* $ ? test -x $SA
| su $LOGNAME -c "$SA"

I wanted to make sure that spamassassin is run under the user-id of
the recipient, so that SA can pick up local user preferences, local
Bayes database and so on. The SA_LOCK is just to serialize things on
a per-user basis to lower overall system load a bit. Still, this
all seemed a little clunky.

- The bigger question I have is: Is keeping the logs, the virus mail,
the almost-certain spam mail in a global location worth the fuss?
The script would've been a _lot_ simpler, if DROPPRIVS took effect
at the beginning and all these files were just deposited in the recipient's
$HOME directory.

- I'm concerned that the users may have some trouble navigating
the spam folder in /var/spam_trap/$LOGNAME, and may even require a link from
their home directory back to the spam_trap folder to easily access it from
their mail client application.

- A downside to using this /etc/procmailrc approach is that some decisions
are being made for the user: where to put the virus mail, almost-certain
spam,
marginal spam, and logs. If the user had their own filtering/filing rules
for
spam (one user did, but won't mind the change), they've just been pre-empted
by the sitewide procmailrc.

- My other concern is how will the users interact with the two categories of
spam mail folders? How will whitelisting be implemented (will they be shown
how to edit $HOME/spamassassin/user_prefs in order to add from_whitelist
and from_whitelist_rcvd directives? Many users won't know how to do that.
Is there a simple GUI that someone has come up with for that?

Any tips, or methods for improving this script would be appreciated.

- Gary

------------------
/etc/procmailrc ---------------------------------------------------
##### if we aren't the mail_test user, just exit. #####
:0
* ! LOGNAME ?? ^^mail_test^^
{ SWITCHRC }

# save old shell, make sure we're using 'sh'.
OLD_SHELL=$SHELL
SHELL=/bin/sh

#
# helper variables
#
NL="
"

#
# config. variables
#
SA=/usr/bin/spamassassin
USER_LOGFILE=/var/mail_log/${LOGNAME}.log
SPAM_TRAP=/var/spam_trap/$LOGNAME
VIRUS_TRAP=/var/virus_trap/$LOGNAME
SA_LOCK=$HOME/spam.lock
SPAM_MBOX=$HOME/spam.mbox

#
# set up per user procmail logs
#
:0
{
LOCKFILE=${USER_LOGFILE}$LOCKEXT
:0 Wc
* $ ! ? test -f $USER_LOGFILE
| touch $USER_LOGFILE && chown ${LOGNAME}.mail $USER_LOGFILE && chmod 640
$USER_LOGFILE
#
# if log has more than 6,000 lines, cut it back to 3000
#
:0 EWc
* $ ? test `wc -l < $USER_LOGFILE` -gt 6000
| cp -p $USER_LOGFILE ${USER_LOGFILE}.tmp ; tail -3000 ${USER_LOGFILE}.tmp >
$USER_LOGFILE ; rm -f ${USER_LOGFILE}.tmp
LOCKFILE # release lock
}

LOGFILE=$USER_LOGFILE

VERBOSE=yes

# add delivery infomation to the log
TRAP='formail -I "From " -a "From " -I Return-Path: -I Sender: -I
Errors-To: -l "$LASTFOLDER"'

#
# invoke the virus scanner, tell it not to deliver.
#
NONDEL=TRUE
INCLUDERC=/etc/procmailrcs/virus_scan.rc
#
# If either VIR_A or VIR_B are true, a possible virus
# was seen. Dump it into the virus trap.
#
:0
* $ 1^0 TRUE ?? ^^$VIR_A^^
* $ 1^0 TRUE ?? ^^$VIR_B^^
{ VIRUS_FOUND = TRUE }

# Will set VIRUS_FOUND=TRUE if mydoom virus signature found.
INCLUDERC=/etc/procmailrcs/virus_scan_mydoom.rc

:0
* VIRUS_FOUND ?? ^^TRUE^^
{
LOCKFILE=${VIRUS_TRAP}$LOCKEXT
:0 Wc
* $ ! ? test -f $VIRUS_TRAP
| touch $VIRUS_TRAP && chown ${LOGNAME}.mail $VIRUS_TRAP && chmod 640
$VIRUS_TRAP

:0
$VIRUS_TRAP
LOCKFILE # release lock
}


#
# Invoke spamasassin to check for spam.
#
:0 fw:$SA_LOCK
* $ ? test -x $SA
| su $LOGNAME -c "$SA"

#
# If spam found, and above threshold of 10
# dump it in the spam trap.
#
:0
* ^X-Spam-Status: Yes
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*
{
LOCKFILE=${SPAM_TRAP}$LOCKEXT
:0 Wc
* $ ! ? test -f $SPAM_TRAP
| touch $SPAM_TRAP && chown ${LOGNAME}.mail $SPAM_TRAP && chmod 640
$SPAM_TRAP

:0
$SPAM_TRAP
LOCKFILE # release lock
}

VERBOSE=off
LOGABSTRACT=no

# restore user's shell
SHELL=$OLD_SHELL
#

# Switch from root/mail to $LOGNAME
#
DROPPRIVS=yes

#
# Marginal spam delivered into user's spam.mbox
#
:0:
* ^X-Spam-Status: Yes
$SPAM_MBOX
Re: ideas on filtering e-mail at the site-wide level [ In reply to ]
On Tue, 3 Feb 2004, Gary Funck wrote:

> [.I posted this to the procmail list, but I think the SA-related, and
> mail admin. related issues are more germane to this list, so am
> re-posting here.]

I got it on both lists, but I'm only replying on sa-users; propagate as
you will.

> LOCKFILE=${SPAM_TRAP}$LOCKEXT
> :0 Wc
> * $ ! ? test -f $SPAM_TRAP
> | touch $SPAM_TRAP && chown ${LOGNAME}.mail $SPAM_TRAP && chmod 640 $SPAM_TRAP
>
> :0
> $SPAM_TRAP
> LOCKFILE # release lock

Why run the 'touch' as a pipe?

:0:
* $ 9876543210^0 ? test -f $SPAM_TRAP
* $ 1^0 ? touch $SPAM_TRAP && ...
$SPAM_TRAP

You could also assign to UMASK so as to avoid the chmod.

> Questions:
>
> - if the delivery part of this recipe were changed simply to:
>
> :0:
> $SPAM_TRAP
>
> can I depend upon procmail to create that file with the recipient as
> owner of the file?

Not if you haven't dropped privileges, no.

> (the containing directory is owner=root,group=mail and is writeable only
> by owner/group and rx everyone).

If you want to continue using a systemwide directory like that, then
you'll have to continue with the chmod.

> Or would procmail create it on behalf of the current user, which is root
> (or mail)?

Yes.

> - in this section of the recipe:
>
> #
> # Invoke spamasassin to check for spam.
> #
> :0 fw:$SA_LOCK
> * $ ? test -x $SA
> | su $LOGNAME -c "$SA"
>
> I wanted to make sure that spamassassin is run under the user-id of
> the recipient

Is there a reason you aren't running spamd and using "spamc -u $LOGNAME"?

> - The bigger question I have is: Is keeping the logs, the virus mail,
> the almost-certain spam mail in a global location worth the fuss?

Ordinarily I wouldn't think so, but it probably made your periodic cleanup
scripts simpler, so ...

> - I'm concerned that the users may have some trouble navigating the spam
> folder in /var/spam_trap/$LOGNAME, and may even require a link from
> their home directory back to the spam_trap folder to easily access it
> from their mail client application.

This is probable.

> - A downside to using this /etc/procmailrc approach is that some
> decisions are being made for the user: where to put the virus mail,
> almost-certain spam, marginal spam, and logs. If the user had their own
> filtering/filing rules for spam (one user did, but won't mind the
> change), they've just been pre-empted by the sitewide procmailrc.

A technique that's been suggested before on the procmail list is to test
for the presence of a $HOME/.procmailrc file and only do the global
processing if the user doesn't have one.
RE: ideas on filtering e-mail at the site-wide level [ In reply to ]
Bart, thanks for the well thought-out suggestions.

> -----Original Message-----
> From: Bart Schaefer [mailto:schaefer@zanshin.com]
> Sent: Tuesday, February 03, 2004 9:24 AM
[...]
>
> > LOCKFILE=${SPAM_TRAP}$LOCKEXT
> > :0 Wc
> > * $ ! ? test -f $SPAM_TRAP
> > | touch $SPAM_TRAP && chown ${LOGNAME}.mail $SPAM_TRAP && chmod
> 640 $SPAM_TRAP
> >
> > :0
> > $SPAM_TRAP
> > LOCKFILE # release lock
>
> Why run the 'touch' as a pipe?
>

An evolutionary artifact. I didn't have that touch/chown/chmod stuff in at
first, and then when I added it later, didn't think of your simpler solution
...

> :0:
> * $ 9876543210^0 ? test -f $SPAM_TRAP
> * $ 1^0 ? touch $SPAM_TRAP && ...
> $SPAM_TRAP
>

Looks good. I'm a little iffy on how the chain of touch, chmod, chown might
set the status codes, but have that problem in my formulation. This might
come up for example, the intended recipient is 'root' and the script is
being
run by 'root'. Thus the chown is redundant. I'm sure there's ways to work
around that, with a little research.

> You could also assign to UMASK so as to avoid the chmod.
>

Yup. thanks.

> > Questions:
> >
> > - if the delivery part of this recipe were changed simply to:
> >
> > :0:
> > $SPAM_TRAP
> >
> > can I depend upon procmail to create that file with the recipient as
> > owner of the file?
>
> Not if you haven't dropped privileges, no.
>
> > (the containing directory is owner=root,group=mail and is writeable only
> > by owner/group and rx everyone).
>
> If you want to continue using a systemwide directory like that, then
> you'll have to continue with the chmod.
>
> > Or would procmail create it on behalf of the current user, which is root
> > (or mail)?
>
> Yes.
>
> > - in this section of the recipe:
> >
> > #
> > # Invoke spamasassin to check for spam.
> > #
> > :0 fw:$SA_LOCK
> > * $ ? test -x $SA
> > | su $LOGNAME -c "$SA"
> >
> > I wanted to make sure that spamassassin is run under the user-id of
> > the recipient
>
> Is there a reason you aren't running spamd and using "spamc -u $LOGNAME"?
>

Yes. this prototype was developed just to test for a single user, and
it was easier just to run SA directly for now. Spamc/spamd would be in
the production set up. Your point that spamc will switch users without
the need for an explicit 'su' is well-taken, however.

> > - The bigger question I have is: Is keeping the logs, the virus mail,
> > the almost-certain spam mail in a global location worth the fuss?
>
> Ordinarily I wouldn't think so, but it probably made your periodic cleanup
> scripts simpler, so ...
>

Possibly, if mail filtering were done on a mail server, and mail delivery
done
at one/more separate hosts. In this case, the scavenger scripts could just
as
likely cruise the users' home directories.

> > - I'm concerned that the users may have some trouble navigating the spam
> > folder in /var/spam_trap/$LOGNAME, and may even require a link from
> > their home directory back to the spam_trap folder to easily access it
> > from their mail client application.
>
> This is probable.
>

Yeah, I'm waiting for feedback on that point.

> > - A downside to using this /etc/procmailrc approach is that some
> > decisions are being made for the user: where to put the virus mail,
> > almost-certain spam, marginal spam, and logs. If the user had their own
> > filtering/filing rules for spam (one user did, but won't mind the
> > change), they've just been pre-empted by the sitewide procmailrc.
>
> A technique that's been suggested before on the procmail list is to test
> for the presence of a $HOME/.procmailrc file and only do the global
> processing if the user doesn't have one.
>

Might be a good way to go, thanks.