Mailing List Archive: Number of header lines in reject log

Number of header lines in reject log

exim-users at exim

Mar 29, 2021, 4:45 AM

Post #1 of 8 (479 views)

Hi All

Our exim reject log files include headers of rejected messages. This is
useful as when we receive a report of a "false positive" rejection from
our anti-spam or anti-virus filters, we can find the logs and work out
what happened and if necessary modify the filters or whitelist something.

However exim has a limit on the number of header lines that are written
into the reject log, if the number of headers exceed this the log is
truncated. We find that some senders, like Outlook 365, include many
lines of proprietary headers. This adds to the number of lines getting
logged and the log can get truncated before lines added by our spam
assassin filter are written to the log. This makes debugging the false
positive more challenging.

Is there a setting which allows one to increase the number of header
lines logged, before the log is truncated? I cannot find any relevant
setting in the documentation.

Thanks

Ian

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: Number of header lines in reject log [ In reply to ]

exim-users at exim

Mar 29, 2021, 5:23 AM

Post #2 of 8 (479 views)

On Mon, Mar 29, 2021 at 01:45:24PM +0200, iforbes-exim--- via Exim-users wrote:
> Our exim reject log files include headers of rejected messages. This
> is useful as when we receive a report of a "false positive"
> rejection from our anti-spam or anti-virus filters, we can find the
> logs and work out what happened and if necessary modify the filters
> or whitelist something.

IMHO, you have better to use short-term archive with relayed messages
for postmortal analysis. Body content is also significant for spam
classification, header strings are not sufficient for this task.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: Number of header lines in reject log [ In reply to ]

exim-users at exim

Mar 29, 2021, 5:45 AM

Post #3 of 8 (479 views)

Am 29.03.21 um 13:45 schrieb iforbes-exim--- via Exim-users:
> However exim has a limit on the number of header lines that are
> written into the reject log, if the number of headers exceed this the
> log is truncated. We find that some senders, like Outlook 365, include
> many lines of proprietary headers. This adds to the number of lines
> getting logged and the log can get truncated before lines added by our
> spam assassin filter are written to the log.

If you refer to SpamAssassin, it logs his results to /var/log/mail ( on
my box ), so do not need the header of the actual email. There is a
message-id referenz in the log to easy correlate mails and results.

Best regards,
Marius

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: Number of header lines in reject log [ In reply to ]

exim-users at exim

Mar 29, 2021, 5:58 AM

Post #4 of 8 (479 views)

On 2021/03/29 14:23, Evgeniy Berdnikov via Exim-users wrote:
> On Mon, Mar 29, 2021 at 01:45:24PM +0200, iforbes-exim--- via Exim-users wrote:
>> Our exim reject log files include headers of rejected messages. This
>> is useful as when we receive a report of a "false positive"
>> rejection from our anti-spam or anti-virus filters, we can find the
>> logs and work out what happened and if necessary modify the filters
>> or whitelist something.
> IMHO, you have better to use short-term archive with relayed messages
> for postmortal analysis. Body content is also significant for spam
> classification, header strings are not sufficient for this task.

Interesting idea. How would we keep a copy of a message that is
subsequently rejected with a 550 error? Also we would need to ensure
that the messages are automatically deleted again within a short period,
to ensure privacy protection. A "cron" script with "find" could do that.

But that would be a separate project. For the moment I would be happy
just to increase the number of header lines that get logged. If that is
possible.

Ian

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: Number of header lines in reject log [ In reply to ]

exim-users at exim

Mar 29, 2021, 6:18 AM

Post #5 of 8 (479 views)

On Mon, Mar 29, 2021 at 02:58:57PM +0200, iforbes-exim--- via Exim-users wrote:
> Interesting idea. How would we keep a copy of a message that is subsequently
> rejected with a 550 error?

Yes, use

control = fakereject/<message string to SMTP client>

in DATA acl.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: Number of header lines in reject log [ In reply to ]

exim-users at exim

Mar 29, 2021, 7:57 AM

Post #6 of 8 (479 views)

On 2021/03/29 14:45, Cyborg via Exim-users wrote:
> If you refer to SpamAssassin, it logs his results to /var/log/mail (
> on my box ), so do not need the header of the actual email. There is a
> message-id referenz in the log to easy correlate mails and results.

Yes, we can do this. Unfortunately (our) Spam Assassin does not log
scores in the mail.log. So this becomes a 3 part look up: message id ->
spamd log -> (scores file + local scores file). I would rather log user
friendly header lines, that we can cut/paste into an explanation when
the customer cannot receive a message.

Ian

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: Number of header lines in reject log [ In reply to ]

exim-users at exim

Mar 29, 2021, 8:26 AM

Post #7 of 8 (479 views)

On 29 Mar 2021, at 12:45, iforbes-exim--- via Exim-users <exim-users@exim.org> wrote:
> However exim has a limit on the number of header lines that are written into the reject log

exim has a limit on the log *buffer* size, not the number of lines - which is why the truncation occurs at different points for different messages:

./src/macros.h: #define LOG_BUFFER_SIZE 8192

If you need more writing into the log, you’ll need to make that bigger and recompile. Bear in mind that this isn’t something I’ve tested so results might not be what you want.

Graeme
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: Number of header lines in reject log [ In reply to ]

exim-users at exim

Mar 29, 2021, 9:15 AM

Post #8 of 8 (479 views)

On Mon, Mar 29, 2021 at 04:57:19PM +0200, iforbes-exim--- via Exim-users wrote:
> I would rather log user friendly
> header lines, that we can cut/paste into an explanation when the customer
> cannot receive a message.

If you are rejecting messages on the basis of spamassassin score, then
you have extremely anti-human policy, annoying users and postmaster...

You can't ultimately deside what is spam and what is not, because any
classification algorithm have faults and false positives. The same mail
considered by one recipient as spam may be accepted as valid and impotant
by other recipient. So you have better to HELP users with spam classifier,
but not take responsibility for the result of automatic classification.
Just deliver "spam" messages to separate folder (named "Spam", etc),
or mark them (in Subject), and supply user-driven tools for automated
message processing, ideally for processing of X-Spam-* headers as well.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/