Mailing List Archive: authres missing when ran from spamass-milter

authres missing when ran from spamass-milter

May 30, 2023, 9:15 AM

Post #1 of 11 (439 views)

Hello,

I happily use spamass-milter to filter spam at SMTP time.
Prior to spamass-milter, I use pyspf-milter/opendkim/opendmarc milters to
mark if mail passes coresponding checks.

I also use authres plugin to use these results. However, it does not work
when receiving mail.

I tried debugging both spamass-milter and spamd, and I see that the headers
are indeed there:

May 30 17:57:03 fantomas spamd[1101]: authres: no Authentication-Results headers found from internal
May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; dmarc=none (p=none dis=none) header.from=xxx.sk
May 30 17:57:03 fantomas spamd[1101]: rules: [...]
May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; arc=none smtp.remote-ip=192.0.2.1
May 30 17:57:03 fantomas spamd[1101]: rules: [...]
May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; spf=pass (sender SPF authorized) smtp.mailfrom=xxx.sk (client-ip=192.0.2.1; helo=smtp8.xxx.sk; envelope-from=yyy@xxx.sk; receiver=<UNKNOWN>)

Does anyone have an idea why spamd misses these?

when I pipe message to spamd manually, those headers are there and AUTHRES matches:

X-Spam-Status: No, score=-0.9 required=5.0 tests=AUTHRES_SPF_PASS,BAYES_00,
DCC_CHECK,DMARC_MISSING,KAM_DMARC_STATUS,KAM_NUMSUBJECT,RDNS_NONE,
SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no
autolearn_force=no version=4.0.0

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watkins. -- Daffy Duck & Porky Pig

Re: authres missing when ran from spamass-milter [ In reply to ]

dbuergin at gluet

May 30, 2023, 9:48 AM

Post #2 of 11 (439 views)

Permalink

Matus UHLAR - fantomas:
> I happily use spamass-milter to filter spam at SMTP time.
> Prior to spamass-milter, I use pyspf-milter/opendkim/opendmarc milters to mark if mail passes coresponding checks.
>
> I also use authres plugin to use these results. However, it does not work when receiving mail.
>
> I tried debugging both spamass-milter and spamd, and I see that the headers are indeed there:
>
>
> May 30 17:57:03 fantomas spamd[1101]: authres: no Authentication-Results headers found from internal
> May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; dmarc=none (p=none dis=none) header.from=xxx.sk
> May 30 17:57:03 fantomas spamd[1101]: rules: [...]
> May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; arc=none smtp.remote-ip=192.0.2.1
> May 30 17:57:03 fantomas spamd[1101]: rules: [...]
> May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; spf=pass (sender SPF authorized) smtp.mailfrom=xxx.sk (client-ip=192.0.2.1; helo=smtp8.xxx.sk; envelope-from=yyy@xxx.sk; receiver=<UNKNOWN>)
>
> Does anyone have an idea why spamd misses these?
>
>
> when I pipe message to spamd manually, those headers are there and AUTHRES matches:
>
> X-Spam-Status: No, score=-0.9 required=5.0 tests=AUTHRES_SPF_PASS,BAYES_00,
>         DCC_CHECK,DMARC_MISSING,KAM_DMARC_STATUS,KAM_NUMSUBJECT,RDNS_NONE,
>         SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no
>         autolearn_force=no version=4.0.0

Did you check if the ‘Authentication-Results’ headers are above the
‘Received’ header generated by the milter? Per your own observation in
an older thread:

https://lists.apache.org/thread/q1vvoqvfv3fxjhwjzbjztq1y85hyn3mk

(To be sure I’m not currently using AuthRes, so don’t know if relevant.)

Re: authres missing when ran from spamass-milter [ In reply to ]

uhlar at fantomas

May 30, 2023, 10:08 AM

Post #3 of 11 (439 views)

Permalink

>Matus UHLAR - fantomas:
>> I happily use spamass-milter to filter spam at SMTP time.
>> Prior to spamass-milter, I use pyspf-milter/opendkim/opendmarc milters to mark if mail passes coresponding checks.
>>
>> I also use authres plugin to use these results. However, it does not work when receiving mail.
>>
>> I tried debugging both spamass-milter and spamd, and I see that the headers are indeed there:
>>
>>
>> May 30 17:57:03 fantomas spamd[1101]: authres: no Authentication-Results headers found from internal
>> May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; dmarc=none (p=none dis=none) header.from=xxx.sk
>> May 30 17:57:03 fantomas spamd[1101]: rules: [...]
>> May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; arc=none smtp.remote-ip=192.0.2.1
>> May 30 17:57:03 fantomas spamd[1101]: rules: [...]
>> May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; spf=pass (sender SPF authorized) smtp.mailfrom=xxx.sk (client-ip=192.0.2.1; helo=smtp8.xxx.sk; envelope-from=yyy@xxx.sk; receiver=<UNKNOWN>)
>>
>> Does anyone have an idea why spamd misses these?
>>
>>
>> when I pipe message to spamd manually, those headers are there and AUTHRES matches:
>>
>> X-Spam-Status: No, score=-0.9 required=5.0 tests=AUTHRES_SPF_PASS,BAYES_00,
>>         DCC_CHECK,DMARC_MISSING,KAM_DMARC_STATUS,KAM_NUMSUBJECT,RDNS_NONE,
>>         SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no
>>         autolearn_force=no version=4.0.0

On 30.05.23 18:48, David Bürgin wrote:
>Did you check if the ‘Authentication-Results’ headers are above the
>‘Received’ header generated by the milter? Per your own observation in
>an older thread:
>
>https://lists.apache.org/thread/q1vvoqvfv3fxjhwjzbjztq1y85hyn3mk

hmm, that may be that.
spamass-milter seems to put generated Received: header before Authentication-Results: added by other milters.

May 30 17:57:03 fantomas spamd[1101]: rules: ran header rule __DOS_RELAYED_EXT ======> got hit: "Received: from smtp8.xxx.sk (smtp8.xxx.sk [192.0.2.1]) by fantomas.fantomas.sk (Postfix 3.5.18/8.13.0) with SMTP id unknown Tue, 30 May 2>
May 30 17:57:03 fantomas spamd[1101]: rules: [...]
May 30 17:57:03 fantomas spamd[1101]: rules: [...] Authentication-Results: fantomas.fantomas.sk; dmarc=none (p=none dis=none) header.from=nextra.sk

that will need spamass-milter change.
thanks for noticing.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watkins. -- Daffy Duck & Porky Pig

Re: authres missing when ran from spamass-milter [ In reply to ]

dbuergin at gluet

May 31, 2023, 4:52 AM

Post #4 of 11 (432 views)

Permalink

Matus UHLAR - fantomas:
> that will need spamass-milter change.

Have you tried setting:

authres_trusted_authserv fantomas.fantomas.sk

I think this should work without changing anything in the milter …

Re: authres missing when ran from spamass-milter [ In reply to ]

uhlar at fantomas

May 31, 2023, 5:41 AM

Post #5 of 11 (432 views)

Permalink

>Matus UHLAR - fantomas:
>> that will need spamass-milter change.

On 31.05.23 13:52, David Bürgin wrote:
>Have you tried setting:
>
>authres_trusted_authserv fantomas.fantomas.sk

I did. that's why it works then checking later.

>I think this should work without changing anything in the milter …

milter adds own synthetised Received: header at the very beginning, which is
mosts possibly the correct reason.

spamass-milter should add this header behind locally added
Authentication-Results: headers, but it needs change in spamass-milter.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I drive way too fast to worry about cholesterol.

Re: authres missing when ran from spamass-milter [ In reply to ]

dbuergin at gluet

May 31, 2023, 6:23 AM

Post #6 of 11 (432 views)

Permalink

Matus UHLAR - fantomas:
> > Matus UHLAR - fantomas:
> > > that will need spamass-milter change.
>
> On 31.05.23 13:52, David Bürgin wrote:
> > Have you tried setting:
> >
> > authres_trusted_authserv fantomas.fantomas.sk
>
> I did. that's why it works then checking later.
>
> > I think this should work without changing anything in the milter …
>
> milter adds own synthetised Received: header at the very beginning, which is
> mosts possibly the correct reason.
>
> spamass-milter should add this header behind locally added
> Authentication-Results: headers, but it needs change in spamass-milter.

I understand, but I still think AuthRes can do this without a change in
the milter. Note the doc for authres_trusted_authserv:

> Use strongly recommended, possibly along with authres_networks all.

So, if you set:

authres_networks all
authres_trusted_authserv fantomas.fantomas.sk

then the relative position of ‘Received’ and ‘Authentication-Results’
headers shouldn’t matter. You just have strip out forged results in an
earlier milter. I’ll try it out some other time.

Re: authres missing when ran from spamass-milter [ In reply to ]

uhlar at fantomas

May 31, 2023, 6:27 AM

Post #7 of 11 (432 views)

Permalink

>> > Matus UHLAR - fantomas:
>> > > that will need spamass-milter change.
>>
>> On 31.05.23 13:52, David Bürgin wrote:
>> > Have you tried setting:
>> >
>> > authres_trusted_authserv fantomas.fantomas.sk
>>
>> I did. that's why it works then checking later.
>>
>> > I think this should work without changing anything in the milter …

>Matus UHLAR - fantomas:
>> milter adds own synthetised Received: header at the very beginning, which is
>> mosts possibly the correct reason.
>>
>> spamass-milter should add this header behind locally added
>> Authentication-Results: headers, but it needs change in spamass-milter.

On 31.05.23 15:23, David Bürgin wrote:
>I understand, but I still think AuthRes can do this without a change in
>the milter. Note the doc for authres_trusted_authserv:

>> Use strongly recommended, possibly along with authres_networks all.

>So, if you set:
>
>authres_networks all
>authres_trusted_authserv fantomas.fantomas.sk
>
>then the relative position of ‘Received’ and ‘Authentication-Results’
>headers shouldn’t matter. You just have strip out forged results in an
>earlier milter. I’ll try it out some other time.

I'm not going to trust remote Authentication-Results: headers. Especially
not if they set contain my local hostname.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Fighting for peace is like fucking for virginity...

Re: authres missing when ran from spamass-milter [ In reply to ]

dbfunk at engineering

May 31, 2023, 7:19 AM

Post #8 of 11 (432 views)

Permalink

On Wed, 31 May 2023, Matus UHLAR - fantomas wrote:

[snip..]
> milter adds own synthetised Received: header at the very beginning, which is
> mosts possibly the correct reason.
> spamass-milter should add this header behind locally added
> Authentication-Results: headers, but it needs change in spamass-milter.
>

tl;dr if those 'Authentication-Results: headers' are generated by the MTA itself
the milter may not ever see them.

Which agent in the whole MTA system is adding those 'Authentication-Results:
headers'?
Is it the master MTA itself (EG: postfix or sendmail) or is it some other milter
component?

A milter can only work with what it's handed by the master MTA, if the
Authentication-Results: headers aren't in its input stream then it cannot work
with them.
In the original sendmail incarnation of the milter API it was designed so that a
milter received the message input stream -before- local headers were added, thus
the need for spamassassin 'glue' milters to do that Received: header synthesis.

If those Authentication-Results: headers are being generated by another milter
then the solution is easy, just set the MTA configuration to run that milter
before the spamassassin 'glue' milter. Milter results are chained so any headers
explicitly added by one milter are passed on to succeeding milters.

If those headers are being generated by the MTA then it may not be possible for
milters to see them with out hacking the MTA itself.

--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center, 103 S Capitol St.
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{

Re: authres missing when ran from spamass-milter [ In reply to ]

uhlar at fantomas

Jun 1, 2023, 3:31 AM

Post #9 of 11 (431 views)

Permalink

>On Wed, 31 May 2023, Matus UHLAR - fantomas wrote:
>>milter adds own synthetised Received: header at the very beginning,
>>which is mosts possibly the correct reason. spamass-milter should
>>add this header behind locally added Authentication-Results:
>>headers, but it needs change in spamass-milter.

On 31.05.23 09:19, Dave Funk wrote:
>tl;dr if those 'Authentication-Results: headers' are generated by the
>MTA itself the milter may not ever see them.
>
>Which agent in the whole MTA system is adding those
>'Authentication-Results: headers'?
>Is it the master MTA itself (EG: postfix or sendmail) or is it some
>other milter component?

Headers are added by previous milter components.

>A milter can only work with what it's handed by the master MTA, if the
>Authentication-Results: headers aren't in its input stream then it
>cannot work with them.
>In the original sendmail incarnation of the milter API it was designed
>so that a milter received the message input stream -before- local
>headers were added, thus the need for spamassassin 'glue' milters to
>do that Received: header synthesis.

This is what spamass-milter does. It does see headers added by former
milters, but not yet the Received: header added by local postfix, so it must
synthetize one.

this is documented and consistent with sendmail functionality:
http://www.postfix.org/MILTER_README.html#when-inspect

>If those Authentication-Results: headers are being generated by
>another milter then the solution is easy, just set the MTA
>configuration to run that milter before the spamassassin 'glue'
>milter. Milter results are chained so any headers explicitly added by
>one milter are passed on to succeeding milters.
>
>If those headers are being generated by the MTA then it may not be
>possible for milters to see them with out hacking the MTA itself.

THe problem is that while spamass-milter generates Received: header as the
first of headers, before Authentication-Results: added by other milters.
So, while spamassassin does see those headers, it does not trust them.

One possible fix is to add Received: headers AFTER locally added
Authentication-Results, which requires parsing those headers and only
trusting those that match local hostname (and hope they don't come fake)

Another possible fix is to add local Received: header by postfix and not
spamass-milter. This requires changing both postfix and spamass-milter.

This would otoh make those headers fully trusted, but incompatible with
sendmail.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Boost your system's speed by 500% - DEL C:\WINDOWS$\*.*

Re: authres missing when ran from spamass-milter [ In reply to ]

lwilton at earthlink

Jun 1, 2023, 6:09 AM

Post #10 of 11 (431 views)

Permalink

This is not an area I know anything about, so I may be completely wrong.
That said, I seem to remember a conversation very like this some years back.
If I remember correctly, someone found some switch that could be set to get
spamass-milter to add the Received header before calling the other milters.
Even if there isn't a switch, maybe it would only take a few lines of code
change in spamass-milter to put out the Received header earlier.

Re: authres missing when ran from spamass-milter [ In reply to ]

uhlar at fantomas

Jun 1, 2023, 7:28 AM

Post #11 of 11 (431 views)

Permalink

On 01.06.23 06:09, Loren Wilton wrote:
>This is not an area I know anything about, so I may be completely wrong.
>That said, I seem to remember a conversation very like this some years back.
>If I remember correctly, someone found some switch that could be set
>to get spamass-milter to add the Received header before calling the
>other milters.

The synthetised Received: header is sent to spamd, it's not added to mail
itself (postfix adds it later).

And it's sent as first header, so spamd first sees Received: header
synthetised by spamass-milter and then headers added by other milters.

>Even if there isn't a switch, maybe it would only take a few lines of
>code change in spamass-milter to put out the Received header earlier.

spamass-milter must first send (only) trusted Authentication-Results:
and then synthetised Received: headers.

I'm afraid that wouldn't be just few lines of code.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
My mind is like a steel trap - rusty and illegal in 37 states.