Mailing List Archive

> I've got a long standing server, where I run FreeBSD (13.1) + sendmail
> (8.17.1) + MIMEDefang (2.84) + SpamAssassin (3.4.6).
> (I know there are more recent versions, but that's what ports currently
> provide).
> This has been working perfectly for years.

yes time changes, currently gmail is sometimes blocking emails complain about spf+dkim, while these messages are not even configured for spf/dkim.

> Since the beginning of this year, however, incoming (SMTP authenticated)
> mail from clients outside the LAN is marked as spam.
> E.g.
> > X-Spam-Score: 10.756 (**********)
> BAYES_00,KAM_DMARC_REJECT,KAM_DMARC_STATUS,KAM_LOTSOFHASH,KHOP_HELO_FCRDNS,LOT
> S_OF_MONEY,PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAM
> IC,SPF_FAIL,TO_EQ_FM_DOM_SPF_FAIL

Don't you have more details? Looks to me you are on dns blacklists, your spf is not good etc.

> Right now I instructed MIMEDefang to avoid passing authenticated mails
> to SpamAssassin, but this is not what I ideally want. (If a client gets
> compromised...).

maybe just stat it only (with prometheus)?
https://www.mail-archive.com/users@spamassassin.apache.org/msg109914.html

> My real wish would be to always run messages through SpamAssassin, but
> avoid RBL/SPF/DMARC/dynamic IPs/etc... checks for those that come from
> an authenticated client, as these rules make no sense in that case.

I prefer to have spf, dns rbl connect done in the milter, that is more efficient. As a last I parse message data to resource intensive tasks like spamassassin and clamav.

> What's the best practice to achieve this result?
>

Separate in and out going servers and different configurations for their spamassassin. It is almost impossible to have in/out going combined.

>> Since the beginning of this year, however, incoming (SMTP authenticated)
>> mail from clients outside the LAN is marked as spam.
>> E.g.
>> > X-Spam-Score: 10.756 (**********)
>> BAYES_00,KAM_DMARC_REJECT,KAM_DMARC_STATUS,KAM_LOTSOFHASH,KHOP_HELO_FCRDNS,LOT
>> S_OF_MONEY,PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAM
>> IC,SPF_FAIL,TO_EQ_FM_DOM_SPF_FAIL

On 23.01.23 16:05, Marc wrote:
>Don't you have more details? Looks to me you are on dns blacklists, your spf is not good etc.

You have misunderstood the problem. Authenticated clients are those who
submit mail wia OP's server, so the SPF/DKIM/DMARC can't match as they match
when they go out of the OP's server.

Also, it's common for authenticated clients to send mail from dynamic IP
addresses, they don't leave the OP's server using dynamic IP anymore.

>> Right now I instructed MIMEDefang to avoid passing authenticated mails
>> to SpamAssassin, but this is not what I ideally want. (If a client gets
>> compromised...).

I fully understand this. except checking DNSBLs for dynamic IP and
SPF/DKIM/DMARC checks, all other checks like BAYES, RAZOR/PYZOR/DCC are
useful there.

>> My real wish would be to always run messages through SpamAssassin, but
>> avoid RBL/SPF/DMARC/dynamic IPs/etc... checks for those that come from
>> an authenticated client, as these rules make no sense in that case.
>
>> What's the best practice to achieve this result?
>
>Separate in and out going servers and different configurations for their
> spamassassin. It is almost impossible to have in/out going combined.

That's correct but often also expensive, impossible and ineffective

(e.g. when you want to match incoming mail onto outgoing to check whether
it's real reply to existing problem)


Unfortunately SpamAssassin does not have set of rules to ignore for
outgoing mail, nor special scores for those.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Depression is merely anger without enthusiasm.

On 2023-01-23 at 10:51:14 UTC-0500 (Mon, 23 Jan 2023 16:51:14 +0100)
Andrea Venturoli <ml@netfence.it>
is rumored to have said:

> Hello.
>
> I've got a long standing server, where I run FreeBSD (13.1) + sendmail
> (8.17.1) + MIMEDefang (2.84) + SpamAssassin (3.4.6).
> (I know there are more recent versions, but that's what ports
> currently provide).

SA4 has been in ports for a while. MD3.x should be but is not. This is
unlikely to be relevant to your problem.

> This has been working perfectly for years.
>
> Since the beginning of this year, however, incoming (SMTP
> authenticated) mail from clients outside the LAN is marked as spam.

Very odd. Since you're still on SA3.4.6, the only piece that should have
changed about SA is the rules and the data in external resources like
DNSBLs. That should not have been able to affect how SA detects
authenticated clients.

> E.g.
>> X-Spam-Score: 10.756 (**********)
>> BAYES_00,KAM_DMARC_REJECT,KAM_DMARC_STATUS,KAM_LOTSOFHASH,KHOP_HELO_FCRDNS,LOTS_OF_MONEY,PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAMIC,SPF_FAIL,TO_EQ_FM_DOM_SPF_FAIL

Some external data sources there: sender domain DMARC/SPF records,
SpamHaus, client rDNS. I think the KAM_DMARC_* rules may be new as well.

It is also possible that there were changes in your system that could
trigger this, but I would expect that you'd have mentioned it if you had
made any obvious ones: hostname, local.cf, mimedefang-filter. It would
also be notable if your users have started connecting from a new range
of addresses.


> Right now I instructed MIMEDefang to avoid passing authenticated mails
> to SpamAssassin, but this is not what I ideally want. (If a client
> gets compromised...).

Correct. SA should be able to detect trustworthy authentication
indications in the trusted Received headers which prevent it from
applying *most* of those rules.

> My real wish would be to always run messages through SpamAssassin, but
> avoid RBL/SPF/DMARC/dynamic IPs/etc... checks for those that come from
> an authenticated client, as these rules make no sense in that case.
>
> What's the best practice to achieve this result?

Configure your internal_networks, msa_networks, and trusted_networks
properly and make sure that your mimedefang-filter calls
synthesize_received_header() before spam_assassin_check(). With those
parameters set correctly and the local Received header included, SA
should be able to detect authenticated clients of trusted machines and
skip those rules.

--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

>
> >> Since the beginning of this year, however, incoming (SMTP authenticated)
> >> mail from clients outside the LAN is marked as spam.
> >> E.g.
> >> > X-Spam-Score: 10.756 (**********)
> >>
> BAYES_00,KAM_DMARC_REJECT,KAM_DMARC_STATUS,KAM_LOTSOFHASH,KHOP_HELO_FCRDNS,LOT
> >>
> S_OF_MONEY,PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAM
> >> IC,SPF_FAIL,TO_EQ_FM_DOM_SPF_FAIL
>
> On 23.01.23 16:05, Marc wrote:
> >Don't you have more details? Looks to me you are on dns blacklists, your spf
> is not good etc.
>
> You have misunderstood the problem. Authenticated clients are those who
> submit mail wia OP's server, so the SPF/DKIM/DMARC can't match as they match
> when they go out of the OP's server.
>
> Also, it's common for authenticated clients to send mail from dynamic IP
> addresses, they don't leave the OP's server using dynamic IP anymore.

yes I got this, but it looks like the stage where the message is being parsed to spamassassin, spamassassin uses the client ip. This is also the problem with the rbl, the client ip is being parsed.

I think this was just always working like this, until more and more ip's are listed on dns blacklist and now all of a sudden he passed the threshold.

As you wrote, you can't have such checks on the email, only content can be checked by spamassassin in this setup.

On 1/23/23 17:53, Bill Cole wrote:
> On 2023-01-23 at 10:51:14 UTC-0500 (Mon, 23 Jan 2023 16:51:14 +0100)
> Andrea Venturoli <ml@netfence.it>
> is rumored to have said:
>
>> Hello.
>>
>> I've got a long standing server, where I run FreeBSD (13.1) + sendmail (8.17.1) + MIMEDefang (2.84) + SpamAssassin (3.4.6).
>> (I know there are more recent versions, but that's what ports currently provide).
>
> SA4 has been in ports for a while. MD3.x should be but is not. This is unlikely to be relevant to your problem.
>
>> This has been working perfectly for years.
>>
>> Since the beginning of this year, however, incoming (SMTP authenticated) mail from clients outside the LAN is marked as spam.
>
> Very odd. Since you're still on SA3.4.6, the only piece that should have changed about SA is the rules and the data in external resources like DNSBLs. That should not have been able to affect how SA detects authenticated clients.
>
>> E.g.
>>> X-Spam-Score: 10.756 (**********) BAYES_00,KAM_DMARC_REJECT,KAM_DMARC_STATUS,KAM_LOTSOFHASH,KHOP_HELO_FCRDNS,LOTS_OF_MONEY,PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAMIC,SPF_FAIL,TO_EQ_FM_DOM_SPF_FAIL
>
> Some external data sources there: sender domain DMARC/SPF records, SpamHaus, client rDNS. I think the KAM_DMARC_* rules may be new as well.
>
> It is also possible that there were changes in your system that could trigger this, but I would expect that you'd have mentioned it if you had made any obvious ones: hostname, local.cf, mimedefang-filter. It would also be notable if your users have started connecting from a new range of addresses.
>
>
>> Right now I instructed MIMEDefang to avoid passing authenticated mails to SpamAssassin, but this is not what I ideally want. (If a client gets compromised...).
>
> Correct. SA should be able to detect trustworthy authentication indications in the trusted Received headers which prevent it from applying *most* of those rules.
>
>> My real wish would be to always run messages through SpamAssassin, but avoid RBL/SPF/DMARC/dynamic IPs/etc... checks for those that come from an authenticated client, as these rules make no sense in that case.
>>
>> What's the best practice to achieve this result?
>
> Configure your internal_networks, msa_networks, and trusted_networks properly and make sure that your mimedefang-filter calls synthesize_received_header() before spam_assassin_check(). With those parameters set correctly and the local Received header included, SA should be able to detect authenticated clients of trusted machines and skip those rules.
>
in MIMEDefang 2.84 synthesize_received_header() doesn't add a correct header if the email is authenticated,
this has been fixed in MIMEDefang 2.85 with this commit:
https://github.com/The-McGrail-Foundation/MIMEDefang/commit/34ffd6fa31c4d9e79494fae427ec3b9da6a1c8b1

The problem could have been spotted only recently because more domains started to use DMARC.
Giovanni

On 2023-01-23 at 11:37:04 UTC-0500 (Mon, 23 Jan 2023 17:37:04 +0100)
Matus UHLAR - fantomas <uhlar@fantomas.sk>
is rumored to have said:

> Unfortunately SpamAssassin does not have set of rules to ignore for
> outgoing mail, nor special scores for those.

This isn't quite true.

The *_networks parameters determine how Received headers are interpreted
and which client IPs are tested against which DNSBLs. We also have the
ALL_TRUSTED rule which should match on authenticated initial message
submission to the local host or to an authorized MSA, giving your own
users (or anyone who has compromised their credentials...) a special
benefit of the doubt.




--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

On 1/23/23 16:58, Reindl Harald wrote:

> split inbound and outbound mail on different servers and run a dedicated
> SA instance for submission port - clients don't have a business
> connecting to port 25 at all

Thanks for answering.
Having two mail servers is something we have considered: while possible,
and maybe beneficial for other reasons too, we'd like to avoid the
hassle, if we can achieve the goal in other ways.

bye & Thanks
av.

P.S.
Clients don't connect to 25; they use 465.
However sendmail currently always pass the messages to
MIMEDefang/SpamAssassin, independent of the port that is used (25, 587,
465).

On 1/23/23 17:53, Bill Cole wrote:

Hello.



> SA4 has been in ports for a while. MD3.x should be but is not. This is
> unlikely to be relevant to your problem.

Yes, I know, but on HEAD.
I'm using quarterly port branch (currently 2023Q1), otherwise, with so
frequent changes, maintenance would be a nightmare.


bye & Thanks
av.

P.S.
To you, Bill and Giovanni: I read all your suggestions.
I'm not replying right now, because I want to investigate them before.
Thanks, in the meantime.

>> >> Since the beginning of this year, however, incoming (SMTP authenticated)
>> >> mail from clients outside the LAN is marked as spam.
>> >> E.g.
>> >> > X-Spam-Score: 10.756 (**********)
>> >>
>> BAYES_00,KAM_DMARC_REJECT,KAM_DMARC_STATUS,KAM_LOTSOFHASH,KHOP_HELO_FCRDNS,LOT
>> >>
>> S_OF_MONEY,PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAM
>> >> IC,SPF_FAIL,TO_EQ_FM_DOM_SPF_FAIL
>>
>> On 23.01.23 16:05, Marc wrote:
>> >Don't you have more details? Looks to me you are on dns blacklists, your spf
>> is not good etc.
>>
>> You have misunderstood the problem. Authenticated clients are those who
>> submit mail wia OP's server, so the SPF/DKIM/DMARC can't match as they match
>> when they go out of the OP's server.
>>
>> Also, it's common for authenticated clients to send mail from dynamic IP
>> addresses, they don't leave the OP's server using dynamic IP anymore.

On 23.01.23 17:04, Marc wrote:
>yes I got this, but it looks like the stage where the message is being
> parsed to spamassassin, spamassassin uses the client ip. This is also the
> problem with the rbl, the client ip is being parsed.

setting up trustpath

https://cwiki.apache.org/confluence/display/SPAMASSASSIN/TrustPath

and mailserver tagging authenticated mail properly help much

>I think this was just always working like this, until more and more ip's
> are listed on dns blacklist and now all of a sudden he passed the
> threshold.
>
>As you wrote, you can't have such checks on the email, only content can be
> checked by spamassassin in this setup.

perhaps the mail wasn't authenticated or the client wasn't in trustpath?
Can you post the Received: headers?


--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
It's now safe to throw off your computer.

On 1/24/23 19:01, Matus UHLAR - fantomas wrote:

> Can you post the Received: headers?

I'm trying...
I've prepared a long and detailed message, but it doesn't seem to come
through...
Curious if this simpler message will...

bye & Thanks
av.

>On 1/24/23 19:01, Matus UHLAR - fantomas wrote:
>
>>Can you post the Received: headers?

On 25.01.23 11:30, Andrea Venturoli wrote:
>I'm trying...
>I've prepared a long and detailed message, but it doesn't seem to come
>through...
>Curious if this simpler message will...

just the headers should be enough.
You can also post headers on site like pastebin.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
It's now safe to throw off your computer.

On 1/25/23 12:37, Matus UHLAR - fantomas wrote:

> just the headers should be enough.
> You can also post headers on site like pastebin.

Trying again, with fewer details...

Looking at a quarantined message, the only received header is (anonymized):
> Received: from [192.168.xxx.xxx] (xxx-xxx-xxx-xxx.dyn.eolo.it [xxx.xxx.xxx.xxx])
> (authenticated bits=0)
> by xxxxxx.xxxxxxxxxxxxxxxxxxxxx.xx (8.17.1/8.17.1) with ESMTPSA id 30G71OZ7043441
> (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO)
> for <xxxxxx.xxxx@xxxxxxxxxxxxxxxxxxxxx.xx>; Mon, 16 Jan 2023 08:01:24 +0100 (CET)
> (envelope-from xxxxxxxx.xxxxxxx@xxxxxxxxxxxxxxxxxxxxx.xx)

Running this message through "spamassassin -D -t", I get:
> dbg: received-header: parsed as [ ip=xxx.xxx.xxx.xxx rdns=xxx-xxx-xxx-xxx.dyn.eolo.it helo=!192.168.xxx.xxx! by=xxxxxx.xxxxxxxxxxxxxxxxxxxxx.xx ident= envfrom=xxxxxxxx.xxxxxxx@xxxxxxxxxxxxxxxxxxxxx.it intl=0 >
> dbg: received-header: authentication method ESMTPSA
> dbg: received-header: relay xxx.xxx.xxx.xxx trusted? yes internal? yes msa? no

So, I'm tempted to conclude that I don't need to mess with
internal_networks, msa_networks, and trusted_networks, or call
synthesize_received_header in MIMEDefang.

Also, strangely, running through the command line, this give a score
close to 0 now.



> We also have the ALL_TRUSTED rule which

Alas, for some reason, this does not seem to trigger :(



bye & Thanks
av.

>On 1/25/23 12:37, Matus UHLAR - fantomas wrote:
>>just the headers should be enough.
>>You can also post headers on site like pastebin.

On 25.01.23 15:34, Andrea Venturoli wrote:
>Trying again, with fewer details...
>
>Looking at a quarantined message, the only received header is (anonymized):

>>Received: from [192.168.xxx.xxx] (xxx-xxx-xxx-xxx.dyn.eolo.it [xxx.xxx.xxx.xxx])
>> (authenticated bits=0)
>> by xxxxxx.xxxxxxxxxxxxxxxxxxxxx.xx (8.17.1/8.17.1) with ESMTPSA id 30G71OZ7043441
>> (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO)
>> for <xxxxxx.xxxx@xxxxxxxxxxxxxxxxxxxxx.xx>; Mon, 16 Jan 2023 08:01:24 +0100 (CET)
>> (envelope-from xxxxxxxx.xxxxxxx@xxxxxxxxxxxxxxxxxxxxx.xx)
>
>Running this message through "spamassassin -D -t", I get:

>>dbg: received-header: parsed as [ ip=xxx.xxx.xxx.xxx rdns=xxx-xxx-xxx-xxx.dyn.eolo.it helo=!192.168.xxx.xxx! by=xxxxxx.xxxxxxxxxxxxxxxxxxxxx.xx ident= envfrom=xxxxxxxx.xxxxxxx@xxxxxxxxxxxxxxxxxxxxx.it intl=0 >
>>dbg: received-header: authentication method ESMTPSA
>>dbg: received-header: relay xxx.xxx.xxx.xxx trusted? yes internal? yes msa? no

>So, I'm tempted to conclude that I don't need to mess with
>internal_networks, msa_networks, and trusted_networks,

Not here, because the "with ESMTPSA" means that mail was received encrypted
("S"ecure) and "A"utenticated. Configuring trusted_networks for mail
submission is for clients submitting mail without authentication (which was
very common >10 years ago and still persists somewhere).

> or call synthesize_received_header in MIMEDefang.

With milter, you need to synthetize Received: header, because milter does
see the mail as it came to your MTA, without the locally added Received:
header.

>Also, strangely, running through the command line, this give a score
>close to 0 now.

I guess it's just because of this Received: header that wasn't seen when
mimedefang processed the mail.

>> We also have the ALL_TRUSTED rule which
>
>Alas, for some reason, this does not seem to trigger :(

Perhaps there are other Received: headers in the e-mail?

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
(R)etry, (A)bort, (C)ancer

On 1/26/23 08:23, Matus UHLAR - fantomas wrote:

>> So, I'm tempted to conclude that I don't need to mess with
>> internal_networks, msa_networks, and trusted_networks,
>
> Not here

Ok.



> clients submitting mail without
> authentication (which was very common >10 years ago and still persists
> somewhere).

Dreadful :)



>> or call synthesize_received_header in MIMEDefang.
>
> With milter, you need to synthetize Received: header, because milter
> does see the mail as it came to your MTA, without the locally added
> Received: header.

So, this is possibly the problem. I'll investigate.
(I'll also need to upgrade/patch MIMEDefang before I can use this.
Thanks Giovanni for pointig this out! I guess this will save me a lot of
would be wasted time).



> I guess it's just because of this Received: header that wasn't seen when
> mimedefang processed the mail.

Hmm, then how could spamassassin possibly apply
PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAMIC,...
rules? Where does it get the source IP from?
I only see it there and in an X-Authentication-Warning header (but I
guess MIMEDefang would also not see this one).



> Perhaps there are other Received: headers in the e-mail?

Absolutely not.
There's only the one I posted.


bye & Thanks
av.

On 1/26/23 08:51, Andrea Venturoli wrote:
> On 1/26/23 08:23, Matus UHLAR - fantomas wrote:
>
>>> So, I'm tempted to conclude that I don't need to mess with internal_networks, msa_networks, and trusted_networks,
>>
>> Not here
>
> Ok.
>
>
>
>> clients submitting mail without authentication (which was very common >10 years ago and still persists somewhere).
>
> Dreadful :)
>
>
>
>>> or call synthesize_received_header in MIMEDefang.
>>
>> With milter, you need to synthetize Received: header, because milter does see the mail as it came to your MTA, without the locally added Received: header.
>
> So, this is possibly the problem. I'll investigate.
> (I'll also need to upgrade/patch MIMEDefang before I can use this. Thanks Giovanni for pointig this out! I guess this will save me a lot of would be wasted time).
>
>
>
>> I guess it's just because of this Received: header that wasn't seen when mimedefang processed the mail.
>
> Hmm, then how could spamassassin possibly apply PDS_RDNS_DYNAMIC_FP,RCVD_IN_PBL,RCVD_IN_ZEN_LASTEXTERNAL,RDNS_DYNAMIC,... rules? Where does it get the source IP from?
> I only see it there and in an X-Authentication-Warning header (but I guess MIMEDefang would also not see this one).
>
MIMEDefang 2.84 will syntetize an header like:
by $hostname (envelope-sender $Sender) (MIMEDefang) with ESMTP id $MessageID"
even for authenticated emails while MIMEDefang 2.85+ will inject ESMTPA header for authenticated emails.
This will change which SpamAssassin rules are triggered.

Giovanni

>
>
>> Perhaps there are other Received: headers in the e-mail?
>
> Absolutely not.
> There's only the one I posted.
>
>
>  bye & Thanks
>     av.

On 1/26/23 09:02, giovanni@paclan.it wrote:

> MIMEDefang 2.84 will syntetize an header like:
> by $hostname (envelope-sender $Sender) (MIMEDefang) with ESMTP id
> $MessageID"
> even for authenticated emails while MIMEDefang 2.85+ will inject ESMTPA
> header for authenticated emails.
> This will change which SpamAssassin rules are triggered.

Hello.

I can confirm updating MIMEDefang to 2.86 solved this.
Thanks to everyone.

bye
av.