Mailing List Archive

SPF plugin ignores existing Authentication-Results
I use a separate SPF component (https://crates.io/crates/spf-milter). It
conveys SPF results for both HELO and MAIL FROM identity, each in its
own Authentication-Results header:

Authentication-Results: mail.gluet.ch; spf=fail smtp.mailfrom=bounces.amazon.co.jp
Authentication-Results: mail.gluet.ch; spf=fail smtp.helo=bounces.amazon.co.jp

I remember asking here if SpamAssassin is able to use these instead of
doing its own SPF queries. Now with debug logging on, I see that
SpamAssassin isn’t actually using these results:

May 17 20:11:06 mail spf-milter[836]: bounces.amazon.co.jp (helo): fail
May 17 20:11:06 mail spf-milter[836]: gexymvnji@bounces.amazon.co.jp (mailfrom): fail
May 17 20:11:07 mail spf-milter[836]: 8599A400D309: adding Authentication-Results header
May 17 20:11:07 mail spf-milter[836]: 8599A400D309: Authentication-Results: mail.gluet.ch; spf=fail smtp.helo=bounces.amazon.co.jp
May 17 20:11:07 mail spf-milter[836]: 8599A400D309: adding Authentication-Results header
May 17 20:11:07 mail spf-milter[836]: 8599A400D309: Authentication-Results: mail.gluet.ch; spf=fail smtp.mailfrom=bounces.amazon.co.jp
May 17 20:11:07 mail spamd[11230]: spf: checking to see if the message has a Received-SPF header that we can use
May 17 20:11:07 mail spamd[11230]: spf: checking HELO (helo=bounces.amazon.co.jp, ip=160.251.60.97)
...

Is this a bug in the SPF plugin? Do I need to set something in my
config? I’m using SpamAssassin 3.4.4-1ubuntu1.1 on Ubuntu Server 20.04.
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
David Bürgin:
> I remember asking here if SpamAssassin is able to use these instead of
> doing its own SPF queries. Now with debug logging on, I see that
> SpamAssassin isn’t actually using these results:
>
> ...
>
> Is this a bug in the SPF plugin? Do I need to set something in my
> config? I’m using SpamAssassin 3.4.4-1ubuntu1.1 on Ubuntu Server 20.04.

Oh, it actually cannot work. SpamAssassin only accepts
Authentication-Results that come *before* an ‘internal’ Received header
(https://cwiki.apache.org/confluence/display/SPAMASSASSIN/TrustedRelays).
I use SpamAssassin as a milter, so the ‘internal’ Received header is
added by Postfix only *after* the milters add their headers. That means
my milters’ Authentication-Results headers will never look like internal
to SpamAssassin, and will never be used.

Bother. I think I will try to modify my SpamAssassin milter, so that it
will add a synthetic ‘internal’ Received header right after the
Authentication-Results headers … that should trick SpamAssassin into
recognising them as internal.
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
David Bürgin:
> Bother. I think I will try to modify my SpamAssassin milter, so that it
> will add a synthetic ‘internal’ Received header right after the
> Authentication-Results headers … that should trick SpamAssassin into
> recognising them as internal.

Here’s the plan to address this in SpamAssassin Milter
(https://crates.io/crates/spamassassin-milter): Add an option that takes
a regular expression which matches headers *after which* the synthetic
‘internal’ Received header should be inserted.

--trusted-relay-after <regex>

Then one would use this like this:

--trusted-relay-after '^(?i)authentication-results:[\t ]*mail\.gluet\.ch;'

The effect this would have is that a message that is sent to
SpamAssassin in this form currently:

Received: from m43-8.mailgun.net by mail.gluet.ch (Postfix) ...
Authentication-Results: mail.gluet.ch; dmarc=pass ... [? untrusted!]
Authentication-Results: mail.gluet.ch; dkim=pass ... [? untrusted!]
Authentication-Results: mail.gluet.ch; spf=pass ... [? untrusted!]
Message-ID: <123@execute-api.eu-central-1.amazonaws.com>
...

Would now be sent to SpamAssassin in this form:

Authentication-Results: mail.gluet.ch; dmarc=pass ... [? now trusted]
Authentication-Results: mail.gluet.ch; dkim=pass ... [? now trusted]
Authentication-Results: mail.gluet.ch; spf=pass ... [? now trusted]
Received: from m43-8.mailgun.net by mail.gluet.ch (Postfix) ...
Message-ID: <123@execute-api.eu-central-1.amazonaws.com>
...

I’m interested to hear what comments others might have about the problem
and this solution. (Also a bit surprised that no one seems to have
noticed this before.)

To highlight the problem once more: When used as a milter, SpamAssassin
cannot take into account Authentication-Results added by earlier
milters, because it does not trust them.
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
On Tue, 2021-05-18 at 10:00 +0200, David Bürgin wrote:
> David Bürgin:
> > Bother. I think I will try to modify my SpamAssassin milter, so that
> > it
> > will add a synthetic ‘internal’ Received header right after the
> > Authentication-Results headers … that should trick SpamAssassin into
> > recognising them as internal.
>
Have you set the 'internal_networks' configuration parameter (in
local.cf)? If not, try that first.

Martin
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
Martin Gregorie:
> Have you set the 'internal_networks' configuration parameter (in
> local.cf)? If not, try that first.

Thanks, but I don’t think this helps here. I don’t know what I could add
to internal_networks that would somehow change the behaviour. The
problem is with how milters for SpamAssassin operate.
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
>Martin Gregorie:
>>Have you set the 'internal_networks' configuration parameter (in
>>local.cf)? If not, try that first.

On 18.05.21 12:07, David Bürgin wrote:
>Thanks, but I don’t think this helps here. I don’t know what I could add
>to internal_networks that would somehow change the behaviour. The
>problem is with how milters for SpamAssassin operate.

this is more an issue of how milter itself operates.

the milter is supposed to see e-mail as it was received from (smtp) client -
even without Received: headers, just with other milters' modifications.

If SpamAssassin (SA from now) has to see Authentication-Results: headers
from other milter, the other milter must run before milter using SA
(spamass-milter, amavisd-milter, etc)

SA milter has to synthetize the Received: header it passsed with mail to
SpamAssassin. If it prepends Received: header (as expected), the
Authentication-Results: header(s) added by the former milter appear after
Received: and SA doesn't trust it.

...nobody sees the synthetized Received: header later, they see Received:
added by MTA, before Authentication-Results added by mentioned milter.


Possible workarounds require trusting the Authentication-Results: header
either via SA milter (which would add synthetized Received: header after
it), or via SpamAssassin itself (trust headers added by "host" immediately
after last trusted/internal "Received" header.)

I prefer the second setup, as it's possible to re-check SA score for such
e-mail later.

I have tried receiving mail with fake Authentication-Results: header and it
got deleted by opendkim-milter, to opendkim-milter may be trusted for this
setup.

SA would need an option which hosts to trust Authentication-Results: from.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
BSE = Mad Cow Desease ... BSA = Mad Software Producents Desease
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
Matus UHLAR - fantomas:
> this is more an issue of how milter itself operates.
>
> the milter is supposed to see e-mail as it was received from (smtp) client -
> even without Received: headers, just with other milters' modifications.
>
> If SpamAssassin (SA from now) has to see Authentication-Results: headers
> from other milter, the other milter must run before milter using SA
> (spamass-milter, amavisd-milter, etc)
>
> SA milter has to synthetize the Received: header it passsed with mail to
> SpamAssassin.  If it prepends Received: header (as expected), the
> Authentication-Results: header(s) added by the former milter appear after
> Received: and SA doesn't trust it.
>
> ...nobody sees the synthetized Received: header later, they see Received:
> added by MTA, before Authentication-Results added by mentioned milter.

Thank you, this is a good summary of the problem.

> Possible workarounds require trusting the Authentication-Results: header
> either via SA milter (which would add synthetized Received: header after
> it), or via SpamAssassin itself (trust headers added by "host" immediately
> after last trusted/internal "Received" header.)
>
> I prefer the second setup, as it's possible to re-check SA score for such
> e-mail later.
>
> I have tried receiving mail with fake Authentication-Results: header and it
> got deleted by opendkim-milter, to opendkim-milter may be trusted for this
> setup.
>
> SA would need an option which hosts to trust Authentication-Results: from.

To the SpamAssassin developers: Would you consider adding a
configuration option to trust *all* Authentication-Results headers with
a certain authserv-id
(https://www.rfc-editor.org/rfc/rfc8601#section-2.5)? That would be very
helpful for setups that (securely) manage Authentication-Results headers
themselves.
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
>Matus UHLAR - fantomas:
>>Possible workarounds require trusting the Authentication-Results: header
>>either via SA milter (which would add synthetized Received: header after
>>it), or via SpamAssassin itself (trust headers added by "host" immediately
>>after last trusted/internal "Received" header.)
>>
>>I prefer the second setup, as it's possible to re-check SA score for such
>>e-mail later.
>>
>>I have tried receiving mail with fake Authentication-Results: header and it
>>got deleted by opendkim-milter, to opendkim-milter may be trusted for this
>>setup.
>>
>>SA would need an option which hosts to trust Authentication-Results: from.

On 18.05.21 14:12, David B?rgin wrote:
>To the SpamAssassin developers: Would you consider adding a
>configuration option to trust *all* Authentication-Results headers with
>a certain authserv-id
>(https://www.rfc-editor.org/rfc/rfc8601#section-2.5)? That would be very
>helpful for setups that (securely) manage Authentication-Results headers
>themselves.

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6918

I'll add the info there.


--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Emacs is a complicated operating system without good text editor.
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
David Bürgin:
> David Bürgin:
> > Bother. I think I will try to modify my SpamAssassin milter, so that it
> > will add a synthetic ‘internal’ Received header right after the
> > Authentication-Results headers … that should trick SpamAssassin into
> > recognising them as internal.
>
> Here’s the plan to address this in SpamAssassin Milter
> (https://crates.io/crates/spamassassin-milter): Add an option that takes
> a regular expression which matches headers *after which* the synthetic
> ‘internal’ Received header should be inserted.

For those following along, I have added a new experimental option to my
SpamAssassin milter implementation. Here’s the documentation:

--synth-relay POS
Synthesize the internal relay header after position POS. POS may
be either an integer, in which case the relay header will be in?
serted after skipping POS header lines, or a regular expression,
in which case the relay header will be inserted after skipping all
header lines matching POS. This option is useful in situations
where the default position of the synthesized internal relay
header (the MTA’s Received header, https://cwiki.apache.org/con?
fluence/display/SPAMASSASSIN/TrustedRelays) at the very beginning
of the message header is not appropriate. For example, when other
milters insert Authentication-Results headers for SpamAssassin to
consume, SpamAssassin will not trust these headers unless they ap?
pear before the internal relay header; in this case setting POS to
“^(?i)authentication-results:” achieves proper placement of the
relay header after such header lines.

Now, using the following setting I should get SpamAssassin to reuse
Authentication-Results provided by SPF Milter:

--synth-relay '^(?i)authentication-results: *mail\.gluet\.ch;'

And here we go:

May 21 19:08:21 mail spf-milter[42418]: mxout1-he-de.apache.org (helo): pass
May 21 19:08:21 mail spf-milter[42418]: users-return-124577-dbuergin=gluet.ch@spamassassin.apache.org (mailfrom): pass
May 21 19:08:21 mail spf-milter[42418]: E87B4400D309: adding Authentication-Results header
May 21 19:08:21 mail spf-milter[42418]: E87B4400D309: Authentication-Results: mail.gluet.ch; spf=pass smtp.helo=mxout1-he-de.apache.org
May 21 19:08:21 mail spf-milter[42418]: E87B4400D309: adding Authentication-Results header
May 21 19:08:21 mail spf-milter[42418]: E87B4400D309: Authentication-Results: mail.gluet.ch; spf=pass smtp.mailfrom=spamassassin.apache.org
May 21 19:08:22 mail spamassassin-milter[55644]: E87B4400D309: inserting synthetic relay header at index 4
May 21 19:08:22 mail spamd[56256]: spf: checking to see if the message has a Received-SPF header that we can use
May 21 19:08:22 mail spamd[56256]: spf: found an Authentication-Results header added by an internal host: Authentication-Results: mail.gluet.ch; spf=pass smtp.helo=mxout1-he-de.apache.org
May 21 19:08:22 mail spamd[56256]: spf: re-using helo result from Authentication-Results header: pass
May 21 19:08:22 mail spamd[56256]: spf: found an Authentication-Results header added by an internal host: Authentication-Results: mail.gluet.ch; spf=pass smtp.mailfrom=spamassassin.apache.org
May 21 19:08:22 mail spamd[56256]: spf: re-using mfrom result from Authentication-Results header: pass

Success! For now I’m keeping this on a separate branch. Don’t know if it
feels too hacky to keep, feedback welcome. I would prefer if this could
be solved in SpamAssassin (bug 6918).

Ciao,
David
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
>Matus UHLAR - fantomas:
>>this is more an issue of how milter itself operates.
>>
>>the milter is supposed to see e-mail as it was received from (smtp) client -
>>even without Received: headers, just with other milters' modifications.
>>
>>If SpamAssassin (SA from now) has to see Authentication-Results: headers
>>from other milter, the other milter must run before milter using SA
>>(spamass-milter, amavisd-milter, etc)
>>
>>SA milter has to synthetize the Received: header it passsed with mail to
>>SpamAssassin.? If it prepends Received: header (as expected), the
>>Authentication-Results: header(s) added by the former milter appear after
>>Received: and SA doesn't trust it.
>>
>>...nobody sees the synthetized Received: header later, they see Received:
>>added by MTA, before Authentication-Results added by mentioned milter.

On 18.05.21 14:12, David B?rgin wrote:
>Thank you, this is a good summary of the problem.

I want to add to this old thread, that last few changes fixed this problem.

When milter adds header, it can add it at position 0 which means before
MTA-added Received: header.

pull requests were submitted to put Authentication-Results in case of
OpenDKIM - https://github.com/trusteddomainproject/OpenDKIM/pull/126
OpenDMARC - https://github.com/trusteddomainproject/OpenDMARC/pull/171
and openARC - https://github.com/trusteddomainproject/OpenARC/pull/141

so hopefully, next releases will put headers at proper place.

I have patched OpenDKIM and OpenDMARC and can verify, that not only other
milters, but also later programs see Authentication-Results: at proper
place, so sendmail can trust them.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Saving Private Ryan...
Private Ryan exists. Overwrite? (Y/N)
Re: SPF plugin ignores existing Authentication-Results [ In reply to ]
Matus UHLAR - fantomas:
>> Matus UHLAR - fantomas:
>>> this is more an issue of how milter itself operates.
>>>
>>> the milter is supposed to see e-mail as it was received from (smtp) client -
>>> even without Received: headers, just with other milters' modifications.
>>>
>>> If SpamAssassin (SA from now) has to see Authentication-Results: headers
>>> from other milter, the other milter must run before milter using SA
>>> (spamass-milter, amavisd-milter, etc)
>>>
>>> SA milter has to synthetize the Received: header it passsed with mail to
>>> SpamAssassin.  If it prepends Received: header (as expected), the
>>> Authentication-Results: header(s) added by the former milter appear after
>>> Received: and SA doesn't trust it.
>>>
>>> ...nobody sees the synthetized Received: header later, they see Received:
>>> added by MTA, before Authentication-Results added by mentioned milter.
>
> On 18.05.21 14:12, David Bürgin wrote:
>> Thank you, this is a good summary of the problem.
>
> I want to add to this old thread, that last few changes fixed this problem.
>
> When milter adds header, it can add it at position 0 which means before
> MTA-added Received: header.
>
> pull requests were submitted to put Authentication-Results in case of
> OpenDKIM - https://github.com/trusteddomainproject/OpenDKIM/pull/126
> OpenDMARC - https://github.com/trusteddomainproject/OpenDMARC/pull/171
> and openARC - https://github.com/trusteddomainproject/OpenARC/pull/141
>
> so hopefully, next releases will put headers at proper place.
>
> I have patched OpenDKIM and OpenDMARC and can verify, that not only other
> milters, but also later programs see Authentication-Results: at proper
> place, so sendmail can trust them.

Just to be clear, these changes don’t help in the SpamAssassin milter
case. A workaround like ‘--synth-relay’ proposed in this thread is still
necessary when SpamAssassin is integrated as a milter.