Mailing List Archive: Exim 4.68 and Spam Assassin 3.2.3 mistagging email

Hi All,

I've tried posting this question to the exim-users list but received no replies at all. Sorry for the duplicate posting but I'm hoping that perhaps someone here might be able to help. Currently this issue is stopping us deploying as we don't have any ideas as to why this might have happened or how to make sure that it doesn't re-occur.

First of all, I'm sorry if this issue has already been addressed but a large amount of searching didn't find anything.

I've recently installed a new machine to act as a hub for the institution and have been hit by something that I just can't explain. I thought I'd post here to see if anyone has experienced this or has any ideas. After staring at it for nearly three days, I'm still no further on to understanding what happened.

We're using spam assassin 3.2.3 and exim 4.68 and have had a large number of messages reported as clean by spam assassin but tagged as spam by exim.
I have a solid example of a message with the additional spam headers that I can track down in the spam assassin logs to a "clean message" response.

Here's the relevant snippet of ACL from the configuration:

# Put headers in all messages (no matter if spam or not)
warn spam = nobody:true
add_header = X-Spam-Score: $spam_score ($spam_bar)

warn spam = nobody:true
add_header = X-Spam-Report: $spam_report

# Add X-Spam-Flag and a *SPAM* marker in the Subject header when message is over threshold
warn spam = nobody
add_header = X-ISS-Subject: *ISS-Detected SPAM* $h_Subject
warn spam = nobody
add_header = X-ISS-Detected-SPAM: YES

# Reject spam at high scores - value is an INTEGER!!!!
deny message = This message scored $spam_score spam points.
log_message = exceeded spam threshold with $spam_score points.
spam = nobody:true
condition = ${if >{$spam_score_int}{250}{1}{0}}

We're using the system filter to rewrite the subject line to the contents of the X-ISS-Subject header if it's set.

This is all well and good and seems to work fine when we did assorted testing, however we then started to see messages that were matching rules three and four above even when spam assassin logs them as clean. The affected messages all have the X-Spam-Score header set to
"X-Spam-Score: ()" but the report header is fine.

I then noticed that messages being rejected by rule five had the same problem, the X-Spam-Score header was effectively blank, but rule five shows the spam score in the log message.

Here's a sample of a rejected header (with addresses removed and report
trimmed):

2007-11-26 00:00:21 1IwROY-00065M-1Y H=(sloanled.com) [88.238.64.178] F=<jradzinski@bhwc.com> rejected after DATA: exceeded spam threshold with 27.6 points.
Envelope-from: <jradzinski@bhwc.com>
Envelope-to: <an.address@lancaster.ac.uk> P Received: from [88.238.64.178] (helo=sloanled.com)
by whobblebury.lancs.ac.uk with smtp (Exim 4.68)
(envelope-from <jradzinski@bhwc.com>)
id 1IwROY-00065M-1Y
for an.address@lancaster.ac.uk; Mon, 26 Nov 2007 00:00:15 +0000
* Return-Path: <Gay@bhwc.com>
P Received: from 161.58.18.5 (HELO mail-fwd.sbc-webhosting.com)
by lancaster.ac.uk with esmtp (XPYOHBGAWDO JFZGWY)
id NFeim9-s06iU2-iG
for an.address@lancaster.ac.uk; Mon, 26 Nov 2007 02:00:20 +0200 I Message-ID: <41e101c82fbf$55bf0460$c0a80102@Gay>
F From: "Gay D. Mcnally" <Gay@bhwc.com>
T To: "A Person" <an.address@lancaster.ac.uk>
Subject: Witness a miracle of pen!s enlargement with your own eyes!
Date: Mon, 26 Nov 2007 02:00:20 +0200
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_16863_4249_01C82FD0.194A4560"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1106
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
X-Spam-Score: ()
X-Spam-Report: Spam detection software, running on the system
"whobblebury.lancs.ac.uk", has processed this message.
The results are shown below.
Content analysis details: (27.6 points, 4.5 required)
pts rule name description
---- ---------------------- --------------------------------------------------
3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100%
[score: 1.0000]
X-ISS-Subject: *ISS-Detected SPAM* Witness a miracle of pen!s enlargement with your own eyes!
X-ISS-Detected-SPAM: YES

As you can see, the very first line of the log states "exceeded spam threshold with 27.6 points" so how can the X-Spam-Score be blank in rule one but not when the same variable is used in the log line of rule five?

I've checked over my ACL lines several times now and other than the redundancy of specifying rules one and two and then three and four as separate calls to the spam check, I can't see anything obviously wrong.

I'm unable to duplicate the problem on demand and I haven't been able to replicate it since we pulled the machine from service on Monday afternoon.

At this point I'm happy to hear any suggestions!

Thanks in advance, Ian.
--
Ian Norton
Postmaster & Systems Support
University of Lancaster

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

Norton, Ian wrote:
> Hi All,
>
> I've tried posting this question to the exim-users list but received no replies at all. Sorry for the duplicate posting but I'm hoping that perhaps someone here might be able to help. Currently this issue is stopping us deploying as we don't have any ideas as to why this might have happened or how to make sure that it doesn't re-occur.

I remember reading it or something like it and I thought I replied but
that doesn't seem to be the case since I can't find it in any of my sent
mail. Darn.

> We're using spam assassin 3.2.3 and exim 4.68

Almost same here.

> Here's the relevant snippet of ACL from the configuration:
>
> # Put headers in all messages (no matter if spam or not)
> warn spam = nobody:true
> add_header = X-Spam-Score: $spam_score ($spam_bar)
>
> warn spam = nobody:true
> add_header = X-Spam-Report: $spam_report

> X-Spam-Score: ()
> X-Spam-Report: Spam detection software, running on the system
> "whobblebury.lancs.ac.uk", has processed this message.
> The results are shown below.
> Content analysis details: (27.6 points, 4.5 required)

You'll have to excuse the random order of this reply. I wrote it over an
hour period while experimenting and looking at source code.

I had to go back and read your post a few times as I am getting confused
by the "clean message" response in the SA logs when what you gave is
clearly not a clean message.
SA looks like it has whole heartedly decided it's spam and so has exim,
it's just missing the variable contents on the first run which is
interesting since the message is only sent to SA once and all further
results are cached.
"matching rules three and four above even when spam assassin logs them
as clean" - Does SA really say they are clean in the SA log even though
it has a score of 27.6?

$spam_score working on a later line means it can't be a parsing error
from SA to exim or a memory overwrite issue.

So long as the user is not changed for a spam check, the cached results
are used so it's only being run once so it's not a persistent error.

If rules 1 and 2 were skipped but 3 and 4 were not it could be put down
to some sort of issue with the warn verb and a single condition that is
always going to return true.

I have a few theories as to what is happening but every time I try to
type them out I just seem to confuse myself. Basically - add_header is
getting executed before spam = nobody:true, and/or spam = nobody:true is
being evaluated to true as the only important condition and execution of
SA is being skipped for that ACL.
I didn't have much of a look at the code to see if either of these cases
were possible but it's working fine here and the only difference is the
wording of the ACL.

A possible work around would be to split the execution of SA off into
it's own one line warn statement and use the result in all subsequent
ones. ie:

warn spam = nobody

warn spam = nobody:true
add_header = X-Spam-Score: $spam_score
($spam_bar)\nX-Spam-Report: $spam_report

If all else fails, head back to the original evil warn message style.
It's so old it even has the {1}{0} on the IF statements!

# Add headers to all messages regardless of the spam threshold.
warn message = X-Spam-Score: $spam_score ($spam_bar)
condition = ${if <{$message_size}{DATA_SA_MAX}{1}{0}}
spam = spamc:true

# Add spam report to headers
warn message = X-Spam-Report: $spam_report
condition = ${if <{$message_size}{DATA_SA_MAX}{1}{0}}
spam = spamc:true
condition = ${if >{$spam_score_int}{50}{1}{0}}

# Change the subject if it's spam according to SpamAssassin
warn message = X-New-Subject: **SPAM** $spam_score $h_subject:
condition = ${if <{$message_size}{DATA_SA_MAX}{1}{0}}
spam = spamc

deny message = Spam score too high
log_message = Spam score too high ($spam_score)
condition = ${if <{$message_size}{DATA_SA_MAX}{1}{0}}
spam = spamc:true/defer_ok
condition = ${if >{$spam_score_int}{SPAMLEVEL}{1}{0}}

If you can narrow down what exact change makes the problem appear I
might have a chance of finding it in the code. I have not looked into
the configuration file parsing and ACL modifier execution code so I need
to hunt down a specific problem rather than blindly looking at the code
and hoping to spot an error in something I don't understand.
The above chunk of ACL is almost straight out of my configuration file
and it is almost identical to what you run. "message" over "add_header"
and the use "condition" are the only real differences.

Good luck and let us know if you find the root issue.

--
The Exim Manual
http://www.exim.org/docs.html
http://www.exim.org/exim-html-current/doc/html/spec_html/index.html

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##