Mailing List Archive

With the lack of response on this I get the feeling that this topic is not one that anyone wishes to discuss. Perhaps there are some other mailing lists that someone might suggest that is actively trying to stop spam?

Thanks

-----Original Message-----
From: Jeffrey Wheat
Sent: Wednesday, July 17, 2002 12:13 PM
To: exim-users@exim.org
Subject: [Exim] SPAM filtering


Hello. I have been reading the archives and floowing discussions on this topic for some time now. I have looked at Spam Assassin and tried to use it but generated far too many false positives. I am trying to find a solution that I can implement for my users to try and limit the amount of crap that they receive. Email is by far one of the most critical applications that my customers use. I have seen ads for earthlink that claim spam-free services and other companies as well. I realize that battling spam is a very difficult task. There are client packages such as mail washer out there but the customers do not wish to use external services nor do they wish to have to setup rules and filters themselves. Can we possibly get a discussion started on how others are using exim to combat spam for their customers? I feel this would be a great thread.

Regards,
Jeffrey

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.377 / Virus Database: 211 - Release Date: 7/15/2002


--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##


---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.377 / Virus Database: 211 - Release Date: 7/15/2002


---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.377 / Virus Database: 211 - Release Date: 7/15/2002

--
On Thu, Jul 18, 2002 at 09:39:07AM -0400, Jeffrey Wheat wrote:
| With the lack of response on this I get the feeling that this topic
| is not one that anyone wishes to discuss.

I think you killed the discussion from the beginning by saying you
don't want to use SA -- lots of people (myself included) use SA and
really like it.

| Perhaps there are some other mailing lists that someone might
| suggest that is actively trying to stop spam?

How about spamassassin-talk@lists.sourceforge.net? The SA developers
do try to effectively identify spam without tagging non-spam. The
wider variety of users that contribute, the better the rules and the
corpus will be. You can also tailor the scores and rules to your
liking.

-D

--
The lot is cast into the lap,
but its every decision is from the Lord.
Proverbs 16:33

http://dman.ddts.net/~dman/
--
[ Content of type application/pgp-signature deleted ]
--

Derrick,

Spam Assassin has cost me a large number of headaches. We have tried to use it a number of times which only resulted in customers getting extremely irate about how we are altering their emails. It causes a busy server to drive cpu load up through the roof, resulting in poor performance and lost email when the server begins to reject mail due to system loads.

Most importantly, the idea is to STOP spam, not just put a tag on it telling my customers that the mail they are looking at is possibly spam. They do not want to see that. They do not want to have to download hundreds of emails over a modem line (yes, we still have many thousands of modem users) so their email client can delete it based on one rule alone. This is where the problem in fact lies. Not simply making an already obvious observation that an email is spam and tagging as such.

Tools like exiscan do exactly what a virus tool should do. It stops it. It doesn't put a tag on it telling a customer that "hey you got a virus". It stops it and the customer never sees it. It doesn't kill my mail server by spawning perl processes for each mail arriving. Tools for spam should do exactly that as well. Integrated into exim using local_scan, stop spam instead of simply tagging it.

I am sure that spam assassin is a great solution for a number of sites, but for a busy ISP, it is simply too resource hungry and results in far too many complaints from customers.

Jeff

-----Original Message-----
From: Derrick 'dman' Hudson [mailto:dsh8290@rit.edu]
Sent: Thursday, July 18, 2002 10:07 AM
To: exim-users@exim.org
Subject: [Exim] Re: SPAM filtering


--
On Thu, Jul 18, 2002 at 09:39:07AM -0400, Jeffrey Wheat wrote:
| With the lack of response on this I get the feeling that this topic is
| not one that anyone wishes to discuss.

I think you killed the discussion from the beginning by saying you don't want to use SA -- lots of people (myself included) use SA and really like it.

| Perhaps there are some other mailing lists that someone might suggest
| that is actively trying to stop spam?

How about spamassassin-talk@lists.sourceforge.net? The SA developers do try to effectively identify spam without tagging non-spam. The wider variety of users that contribute, the better the rules and the corpus will be. You can also tailor the scores and rules to your liking.

-D

--
The lot is cast into the lap,
but its every decision is from the Lord.
Proverbs 16:33

http://dman.ddts.net/~dman/
--
[ Content of type application/pgp-signature deleted ]
--

--

## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##


---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.377 / Virus Database: 211 - Release Date: 7/15/2002


---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.377 / Virus Database: 211 - Release Date: 7/15/2002

On Thu, Jul 18, 2002 at 10:17:29AM -0400, Jeffrey Wheat wrote:
[rewrapped]
> Tools like exiscan do exactly what a virus tool should do. It stops
> it. It doesn't put a tag on it telling a customer that "hey you got a
> virus". It stops it and the customer never sees it. It doesn't kill my
> mail server by spawning perl processes for each mail arriving. Tools for
> spam should do exactly that as well. Integrated into exim using
> local_scan, stop spam instead of simply tagging it.

This sounds like a recipe for losing mail. Whatever happened to reliable
mail delivery?

It sounds like you basically want your own rules of netblocks and dns-based
blacklists. There isn't really any other way. You will almost certainly have
to have a custom solution.

MBM

--
Matthew Byng-Maddick <mbm@colondot.net> http://colondot.net/

On Thu, 18 Jul 2002, Jeffrey Wheat wrote:

> With the lack of response on this I get the feeling that this topic is
> not one that anyone wishes to discuss. Perhaps there are some other

This list has been discussing ways to limit and stop spam every month for
at least a couple years now. (It has been quite useful.)

> -----Original Message-----

(I've always disliked mua's that do that and then only use one ">" because
then it becomes confusing to know what message you are replying to. So
the following I am replying to the previous message.)

> critical applications that my customers use. I have seen ads for
> earthlink that claim spam-free services and other companies as well. I

Maybe these companies manuall ready every one of their customers' emails
to check for spams? (Just joking.) Anyways, no one can guarantee spam-free
service.

As for a few servers I admin: I use a few dnsbl lists, I use the exim
filter to block certain keywords, block certain X-Mailer's, invalid
message-ids, etc. Plus, I also have my own small blackhole lists of sender
addresses and sender hosts.

Good luck,

Jeremy C. Reed
...................................................
BSD software, documentation, resources, news...
http://bsd.reedmedia.net/

On Thu, 18 Jul 2002, Jeffrey Wheat wrote:

> Most importantly, the idea is to STOP spam, not just put a tag
> on it telling my customers that the mail they are looking at is

So simply filter it out on the server after it was tagged.

> got a virus". It stops it and the customer never sees it. It doesn't
> kill my mail server by spawning perl processes for each mail arriving.

I don't know, but I think it can be configured to have a always running
daemon. Maybe there is a way to have a light-weight client be spawned
that talks to the already running spam-checking daemon.

[many, many lines of unrelated text deleted]

Jeremy C. Reed
...................................................
BSD software, documentation, resources, news...
http://bsd.reedmedia.net/

Matthew,

> This sounds like a recipe for losing mail. Whatever happened
> to reliable mail delivery?

There is always this risk. There is always the potential to lose a valid email. This is of course a catch twenty-two. What to do...Deliver all email and risk having customers move to competition because they claim to be spam free...Or take a chance of dropping a valid email here and there.

>
> It sounds like you basically want your own rules of netblocks
> and dns-based blacklists. There isn't really any other way.

This isn't about wanting or using dns-based lists or custom netblock filters as much as it is about trying to reduce the ways that spammers exploit our mail servers. By starting there, we can begin to reduce the amount of spam. I personally think blacklisting an entire domain because of a few bad eggs is a bad thing.

Jeff

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.377 / Virus Database: 211 - Release Date: 7/15/2002

Hmm. I don't say much on this list, but this one piqued me. Indulge me.
Disclaimer: I don't use SpamAssassin.

On Thu, 18 Jul 2002, Jeffrey Wheat wrote:

> Most importantly, the idea is to STOP spam, not just put a tag
> on it telling my customers that the mail they are looking at is
> possibly spam.

*Your* idea, maybe. Other people want to do other things with stuff
identified as spam. Some of those other people might be your customers
(the quiet ones you don't hear from much, likely). They might quite like
the facility to have their spam tagged, and then _they_ decide whether
they are going to delete it, file it, or do so under certain other of
their own conditions. Some might always want to receive spam, for
amusement value, for research, because they are paranoid any "spam
stopper" will drop legitimate mail (which sooner or later you should
accept it will) ... many reasons.

SpamAssassin gives you the freedom to make these decisions. Depending on
how you implement with your MTA etc, it gives YOU as the operator of the
mail server the freedom to delete stuff SpamAssassin has identified as
spam, if that's the policy you want to operate. Others have commented on
the wisdom of that.

> They do not want to see that. They do not want to have
> to download hundreds of emails over a modem line (yes, we still have
> many thousands of modem users) so their email client can delete it
> based on one rule alone. This is where the problem in fact lies. Not
> simply making an already obvious observation that an email is spam and
> tagging as such.

So provide them with tools that let *them* choose what they want to
receive. Give them a web interface that lets them set their own SA
preferences. For the easy approach, let them toggle something that says
"don't deliver me mail with an SA score above XX".

> Tools like exiscan do exactly what a virus tool should do. It
> stops it. It doesn't put a tag on it telling a customer that "hey you
> got a virus". It stops it and the customer never sees it. It doesn't
> kill my mail server by spawning perl processes for each mail arriving.
> Tools for spam should do exactly that as well. Integrated into exim
> using local_scan, stop spam instead of simply tagging it.

> I am sure that spam assassin is a great solution for a number of
> sites, but for a busy ISP, it is simply too resource hungry and
> results in far too many complaints from customers.

Well that would rather depend on your implementation, and the resources
you give it. If you don't resource it enough, you will hit performance
problems. Your system is doing more work: sooner or later you're going to
ask it to do more work than it has capacity to do.

The point I'm making, is that just bunging SA on there and expecting it to
somehow know what you want it to do isn't enough. Systems don't work like
that I'm afraid, they aren't all just plug-and-play. You have to put some
effort in to get the results you require. SA and Exim and procmail and
RBL and MailScanner and exiscan and so on are (or provide) tools you can
use to help achieve your goal. It's YOUR job to make them work together
to provide something you (and your customers) want. If they don't work
the way you want: configure them, add new bits, or even re-write them the
way you want, if you desire.

Exim provides wonderful facilities to let you deal with mail under certain
conditions, presence and/or content of headers, running external programs,
local_scan, etc etc. Use them.

Jethro.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Jethro R Binks Computing Officer, IT Services
Mailmaster, Listmaster, Webmaster, University Of Strathclyde, Glasgow, UK
Cachemaster jethro.binks@strath.ac.uk

On Thu, 2002-07-18 at 14:39, Jeffrey Wheat wrote:
> With the lack of response on this I get the feeling that this topic
> is not one that anyone wishes to discuss.

First its been done very frequently already.
Second I almost never (OK this is an exception) to stuff that requires
me to do significant work setting up the quotes from the original in the
reply - your first message was all on one line.

About the only ways of keeping a mailbox spam free at present are:-

1. Use a completely blocked domain - ie all MX and A records
point to 127.0.0.1

2. Use TMDA - http://tmda.net/

In both cases the solution may be worse than the problem.

Nigel.
--
[ Nigel Metheringham Nigel.Metheringham@InTechnology.co.uk ]
[. - Comments in this message are my own and not ITO opinion/policy - ]

On Thu, Jul 18, 2002 at 10:46:51AM -0400, Jeffrey Wheat wrote:
[>I wrote:]
[rewrapped again]
> > This sounds like a recipe for losing mail. Whatever happened
> > to reliable mail delivery?
> There is always this risk. There is always the potential to lose a
> valid email. This is of course a catch twenty-two. What to do...Deliver
> all email and risk having customers move to competition because they
> claim to be spam free...Or take a chance of dropping a valid email here
> and there.

Erm, in what way is that a "catch-22". What I meant by "reliable mail
delivery" is, of course, that you don't drop something on the floor. That
the sending user gets a "your mail looked like spam" bounce. I mean, what
happens if that is the message that showed massive credit-card fraud on
my card. I'd be really very fucked off with you if you dropped the mail on
the floor.

> > It sounds like you basically want your own rules of netblocks
> > and dns-based blacklists. There isn't really any other way.
> This isn't about wanting or using dns-based lists or custom
> netblock filters as much as it is about trying to reduce the ways
> that spammers exploit our mail servers. By starting there, we can

Can you explain what you mean here. This doesn't seem to follow from
what you said above. If you have exploitable servers, then perhaps you
should fix them. If you are suffering from lots of spam whose final
destination is your customers, then I'm claiming that you probably need
a custom solution for what you're attempting to do.

> begin to reduce the amount of spam. I personally think blacklisting
> an entire domain because of a few bad eggs is a bad thing.

Where did I say anything about blacklisting entire domains? I agree with
this. This is unrelated to what you say above. You sound rather confused.

MBM

--
Matthew Byng-Maddick <mbm@colondot.net> http://colondot.net/

On Thu, Jul 18, 2002 at 10:17:29AM -0400, Jeffrey Wheat wrote:
> Derrick,
>
> Spam Assassin has cost me a large number of headaches. We have tried to
> use it a number of times which only resulted in customers getting
> extremely irate about how we are altering their emails. It causes a busy

You didn't configure SA correctly. Did you read its documentation?
You can turn the alterings off

> server to drive cpu load up through the roof, resulting in poor
> performance and lost email when the server begins to reject mail due to
> system loads.

That does not result in lost mail, just delayed mail.

> Most importantly, the idea is to STOP spam, not just put a tag on it
> telling my customers that the mail they are looking at is possibly

http://marc.merlins.org/linux/exim/sa.html
You complain about false positives?
Well, setup sa-exim to not reject mail before an SA score of 10 or 12. That
will give you very very few false positives.

That said, SA isn't set and forget for an ISP, you have to prepare yourself
to be very familiar with SA and to know how to tweak its scores.

Please send followups to the sa-talk or sa-exim lists.

Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key

On Thu, Jul 18, 2002 at 10:17:29AM -0400, Jeffrey Wheat wrote:
> Derrick,
>
> Spam Assassin has cost me a large number of headaches. We have tried to
> use it a number of times which only resulted in customers getting
> extremely irate about how we are altering their emails. It causes a busy

You didn't configure SA correctly. Did you read its documentation?
You can turn the alterings off

> server to drive cpu load up through the roof, resulting in poor
> performance and lost email when the server begins to reject mail due to
> system loads.

That does not result in lost mail, just delayed mail.

> Most importantly, the idea is to STOP spam, not just put a tag on it
> telling my customers that the mail they are looking at is possibly

http://marc.merlins.org/linux/exim/sa.html
You complain about false positives?
Well, setup sa-exim to not reject mail before an SA score of 10 or 12. That
will give you very very few false positives.

That said, SA isn't set and forget for an ISP, you have to prepare yourself
to be very familiar with SA and to know how to tweak its scores.

Please send followups to the sa-talk or sa-exim lists.

Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | Finger marc_f@merlins.org for PGP key

On Thu, Jul 18, 2002 at 10:17:29AM -0400, Jeffrey Wheat wrote:
[...]
> Tools like exiscan do exactly what a virus tool should do. It stops it. It doesn't put a tag on it telling a customer that "hey you got a virus". It stops it and the customer never sees it. It doesn't kill my mail server by spawning perl processes for each mail arriving. Tools for spam should do exactly that as well. Integrated into exim using local_scan, stop spam instead of simply tagging it.
1. You can run SA without forking Perl processes for each mail arriving.
You can greatly reduce system load by using spamd and spamc.
2. You can drop mail when it has been tagged.
3. If I were your customer and you did 2, I'd quit.
4. smtp-callback is beautiful.

--
CU,
Patrick.

>Most importantly, the idea is to STOP spam, not just put a tag on it
telling my customers that the mail they are looking at is >possibly
spam. They do not want to see that. They do not want to have to download
hundreds of emails over a modem line >(yes, we still have many thousands
of modem users) so their email client can delete it based on one rule
alone. This is where >the problem in fact lies. Not simply making an
already obvious observation that an email is spam and tagging as such.
>
>Tools like exiscan do exactly what a virus tool should do. It stops it.
It doesn't put a tag on it telling a customer that "hey you >got a
virus". It stops it and the customer never sees it. It doesn't kill my
mail server by spawning perl processes for each mail >arriving. Tools
for spam should do exactly that as well. Integrated into exim using
local_scan, stop spam instead of simply >tagging it.
>
>I am sure that spam assassin is a great solution for a number of sites,
but for a busy ISP, it is simply too resource hungry and >results in far
too many complaints from customers.
>
>Jeff

I thought you could tag the mail with SA, and then have Exim halt
delivery if you so choose (based on the existence of the header). Of
course, customers rarely complain about something they never see... :)

Second, I thought one of the main performance drags was addressed by the
daemon version of SA, which loads a single perl process, instead of
spawning new ones for each message.

All that said, I haven't had a chance to install SA yet, so I am no
expert. I am currently using a combination of RBLs and a custom system
filter. But I am salivating at the chance to try SA, as I still get
quite a bit of spam coming into my sever.

Best wishes.

Jim Roberts
Punster Productions, Inc.

On Thu, 18 Jul 2002, Jeffrey Wheat wrote:

> Spam Assassin has cost me a large number of headaches. We have
> tried to use it a number of times which only resulted in customers
> getting extremely irate about how we are altering their emails. It
> causes a busy server to drive cpu load up through the roof, resulting
> in poor performance and lost email when the server begins to reject mail
> due to system loads.

You acknowledged yourself that doing a good job of detecting spam is a
hard thing. I think it therefore follows that it will require a lot of
CPU to acheive it.

I have some spam-blocking systems in place. I (and my housemate) spent
time on them mostly for the fun of it, but they do a pretty good job of
it.

1. I keep my addressbook in LDAP (actually I use Pine, so I have scripts
to convert from Pine to LDAP daily). exim looks sender addresses up with
LDAP - if it finds a match, a header is added indicating that the address
is whitelisted. If it is it is not passed to:
2. A filter, written in Perl, which my housemate wrote. It turns out that
the stategy is kinda similar to SA - basically emails accumulate points
according to RegEx matches of the text and various other features (having
more than a certain number of addressees in the same domain, having
matching To: and From: headers etc). All these features are configuarable
on a per-user basis. Further headers are added to indicate the status of
these checks.
3. Based on the headers, the emails can be filtered away.

Of course, this is all highly inefficient. It does two LDAP lookups per
email (couldn't find any other way of doing it), and starts a Perl
interpreter for any email that the LDAP doesn't find.

John

--
It would have been jolly to talk like this, and really, it wasn't
much good having anything exciting like floods, if you couldn't share
them with anybody.

> =09Most importantly, the idea is to STOP spam, not just put a tag on it tel=
> ling my customers that the mail they are looking at is possibly spam. They =
> do not want to see that. They do not want to have to download hundreds of e=
> mails over a modem line (yes, we still have many thousands of modem users) =
> so their email client can delete it based on one rule alone. This is where =
> the problem in fact lies. Not simply making an already obvious observation =
> that an email is spam and tagging as such.
>

Its refreshing to hear this. A lot of people get it wrong by going
simply for the tagging option. Most of our customers who complain
don't want to see it at all, particularly if it involves farmyard
animals and "virgins".

The other problem with tagging it is that the recipient has to opt to
delete it or save it to a mailbox to sift through later. Both of
these can lead to a very unreliable service. It is far better to
issue a rejection message with clear reasons about why it has been
rejected, how to avoid rejection and a support address (unfiltered).
A rejection indicating that the message was rejected because it scored
8.3 with a threshold of 8.2 isn't very friendly. Ones detailing what
scored how many points aren't much better either.

> =09Tools like exiscan do exactly what a virus tool should do. It stops it. =
> It doesn't put a tag on it telling a customer that "hey you got a virus". I=
> t stops it and the customer never sees it. It doesn't kill my mail server b=
> y spawning perl processes for each mail arriving. Tools for spam should do =
> exactly that as well. Integrated into exim using local_scan, stop spam inst=
> ead of simply tagging it.
>
> =09I am sure that spam assassin is a great solution for a number of sites, =
> but for a busy ISP, it is simply too resource hungry and results in far too=
> many complaints from customers.

I may be wrong on this; it looks great technically, but appears to
have been written with little regard for customer service
considerations and on the assumption that mail will be tagged and not
rejected. I suspect that if it had been designed on the assumption
that a clear error message would be generated and the email failed
things would have been done a little differently.

It is also essential to have an opt in mechanism. Unless I've
misunderstood, SA documentation suggests that a high threshold for
opt-outs is the way to do this. This doesn't quite give an
opt-out/opt-in mechanism, and means that you have to process all
email. If you have a simple opt-in mechanism, the overheads are
significantly reduced as most of your email doesn't need to be
scanned.

Chris Bayliss

At 16:01 +0100 7/18/2002, Nigel Metheringham wrote:
>About the only ways of keeping a mailbox spam free at present are:-
>
> 1. Use a completely blocked domain - ie all MX and A records
> point to 127.0.0.1
>
> 2. Use TMDA - http://tmda.net/
>
>In both cases the solution may be worse than the problem.

From the end user (customer) viewpoint, I've been playing with MailArmor in
a heavily-spammed account for the past couple of days, replacing the
Spamfire I was playing with. (I keep the account for nostalgia
reasons...it became my address in 1993; I ditched a slightly less-old one
recently for being 110% spam.)

http://www.mailarmor.nl/

Java program for Mac OS, Mac OS X, Windows; Linux; Unix; sets itself up as
a POP3 proxy between one's client (tested with Eudora for Mac OS X) and the
POP3 or IMAP server (should anyone actually want to convert IMAP to POP).
Leaves the suspect messages on the server; looks at the headers and does
scoring based on them. Deletes problem messages from server under
user-settable rules; can send email to user listing messages blocked if
desired. User configurable rules (which I haven't messed with yet).
Blacklist/whitelist. Free for personal use with one account. Not for
"Aunt Sally" as the setup is non-trivial.

So far, so good, after changing one IMHO incorrect default setting (being
on SpamCop's blacklist gets a score high enough to trigger immediate
deletion).

---

IMHO, user control is key to doing server-level SPAM control successfully.
Or having only one sort of user; one wants to avoid getting to that point
by driving the others away.

--John

--
John Baxter jwblist@olympus.net Port Ludlow, WA, USA