Mailing List Archive

KAM_SENDGRID and SPF_HELO_NONE
Hi,

I have an email that matched KAM_SENDGRID because it also matched
SPF_HELO_NONE, despite it apparently being a legitimate sendgrid
email. This is from SA trunk.

0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
-0.0 SPF_PASS SPF: sender matches SPF record
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature
from author's
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not
necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
1.5 KAM_SENDGRID Sendgrid being exploited by scammers

Received-SPF: Pass (mailfrom) identity=mailfrom;
client-ip=167.89.39.250; helo=o1678939x250.outbound-mail.sendgrid.net;
envelope-from=bounces+3940809-b10a-43194=hotel.example.com@em8909.cookspest.com;
receiver=<UNKNOWN>

X-Envelope-From:
<bounces+3940809-b10a-43194=hotel.example.com@em8909.cookspest.com>

I'm noticing what I think are a lot of false positives for this rule.
Is there something more we should be doing to reduce the false
positives here, or is it really warranted?

The mail server does appear to have an SPF record:

# dig +short txt em8909.cookspest.com
u3940809.wl060.sendgrid.net.
"v=spf1 ip4:167.89.39.18 ip4:167.89.39.188 ip4:167.89.39.217
ip4:167.89.39.227 ip4:167.89.39.248 ip4:167.89.39.250 ip4:167.89
.39.45 ip4:167.89.39.75 ip4:167.89.39.79 ip4:208.117.61.64 -all"

Or perhaps it's because it's announcing itself as
o1678939x250.outbound-mail.sendgrid.net, which does not have an SPF
record?

Is it even possible for a sendgrid client to control their SPF record,
let alone SPF HELO?

Perhaps it's because Return-Path is null?
Return-Path: <>
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
On Thu, 2021-05-20 at 16:12 -0400, Alex wrote:
>
> X-Envelope-From:
>         <bounces+3940809-b10a-43194=hotel.example.com@em8909.cookspest.com>
>
>
> Perhaps it's because Return-Path is null?
> Return-Path: <>

Return-Path is supposed to be where your MTA stores the envelope sender. That
it doesn't match is probably a problem.


And yes, SPF falls back to testing the HELO host if the envelope sender is
empty (which should only occur in bounces or auto-responses).
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
And that rule is probably designed to hit legitimate sendgrid emails.

They have become a hacker and spammer haven over the last year and a half
approximately.

On Thu, May 20, 2021, 16:49 Alan Hodgson <ahodgson@lists.simkin.ca> wrote:

> On Thu, 2021-05-20 at 16:12 -0400, Alex wrote:
>
>
> X-Envelope-From:
> <bounces+3940809-b10a-43194=hotel.example.com@em8909.cookspest.com
> >
>
>
> Perhaps it's because Return-Path is null?
> Return-Path: <>
>
>
> Return-Path is supposed to be where your MTA stores the envelope sender.
> That it doesn't match is probably a problem.
>
>
> And yes, SPF falls back to testing the HELO host if the envelope sender is
> empty (which should only occur in bounces or auto-responses).
>
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
On 2021-05-20 at 16:12:40 UTC-0400 (Thu, 20 May 2021 16:12:40 -0400)
Alex <mysqlstudent@gmail.com>
is rumored to have said:

> Hi,
>
> I have an email that matched KAM_SENDGRID because it also matched
> SPF_HELO_NONE, despite it apparently being a legitimate sendgrid
> email. This is from SA trunk.

KAM_SENDGRID is NOT from "SA trunk" it is from the independent rules
channel that Kevin offers, which is NOT part of the ASF SpamAssassin
project.

And FWIW: no matter what version of SA you are running, if you use the
project's default rules channel you get the "trunk" rules. There is only
one current version of the default rules, and it only exists in the
source tree on the 'trunk' branch.

> 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
> -0.0 SPF_PASS SPF: sender matches SPF record
> -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature
> from author's
> 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not
> necessarily valid
> -0.1 DKIM_VALID Message has at least one valid DKIM or DK
> signature
> 1.5 KAM_SENDGRID Sendgrid being exploited by scammers
>
> Received-SPF: Pass (mailfrom) identity=mailfrom;
> client-ip=167.89.39.250; helo=o1678939x250.outbound-mail.sendgrid.net;
> envelope-from=bounces+3940809-b10a-43194=hotel.example.com@em8909.cookspest.com;
> receiver=<UNKNOWN>
>
> X-Envelope-From:
> <bounces+3940809-b10a-43194=hotel.example.com@em8909.cookspest.com>
>
> I'm noticing what I think are a lot of false positives for this rule.

In what way is this a false positive? Looks like a correct positive to
me.

> Is there something more we should be doing to reduce the false
> positives here, or is it really warranted?

Not a false positive. Note that the score of KAM_SENDGRID is much less
than any reasonable spam threshold.

If you disagree with the scoring or purpose of that rule, you are free
to reduce the score locally or discuss it with KAM. He's a very
reasonable guy who listens to reason. His rule QA is not the same as
that applied to the default rule channel. The default rule QA process is
slow, imperfect, and transparent IF you care to examine the code. KAM's
QA is a 100% black box but he makes changes fast when needed.

> The mail server does appear to have an SPF record:

Not relevant.

> # dig +short txt em8909.cookspest.com
> u3940809.wl060.sendgrid.net.
> "v=spf1 ip4:167.89.39.18 ip4:167.89.39.188 ip4:167.89.39.217
> ip4:167.89.39.227 ip4:167.89.39.248 ip4:167.89.39.250 ip4:167.89
> .39.45 ip4:167.89.39.75 ip4:167.89.39.79 ip4:208.117.61.64 -all"
>
> Or perhaps it's because it's announcing itself as
> o1678939x250.outbound-mail.sendgrid.net, which does not have an SPF
> record?

Correct. That is what "SPF_HELO_NONE" means, as it documented in the
rules file 25_spf.cf:

describe SPF_HELO_NONE SPF: HELO does not publish an SPF Record

> Is it even possible for a sendgrid client to control their SPF record,
> let alone SPF HELO?

Probably not, but that's not relevant. It seems to me that the
SPF_HELO_NONE involvement in KAM_SENDGRID is heuristic, not
prescriptive.

> Perhaps it's because Return-Path is null?
> Return-Path: <>

That's a different problem, apparently with your MTA->SA glue. The fact
that something added a non-null "X-Envelope-From:" header and something
(else?) added a null "Return-Path:" header indicates fundamental
breakage. Whether SA is seeing that or if it is a delivery artifact is
unclear.

--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
Hi,

> > I have an email that matched KAM_SENDGRID because it also matched
> > SPF_HELO_NONE, despite it apparently being a legitimate sendgrid
> > email. This is from SA trunk.

I only meant it as a reference for the version of SA (and SPF.pm)
that's being used, in case it was necessary.

> > X-Envelope-From:
> > <bounces+3940809-b10a-43194=hotel.example.com@em8909.cookspest.com>
> >
> > I'm noticing what I think are a lot of false positives for this rule.
>
> In what way is this a false positive? Looks like a correct positive to
> me.

Because it was a legitimate email with an invoice from a pest control
company to their customer.

> If you disagree with the scoring or purpose of that rule, you are free
> to reduce the score locally or discuss it with KAM. He's a very

Nope, just trying to understand.

> KAM's
> QA is a 100% black box but he makes changes fast when needed.

Yes, and just wanted to be sure that wasn't necessary here.

> > Perhaps it's because Return-Path is null?
> > Return-Path: <>
>
> That's a different problem, apparently with your MTA->SA glue. The fact
> that something added a non-null "X-Envelope-From:" header and something
> (else?) added a null "Return-Path:" header indicates fundamental
> breakage. Whether SA is seeing that or if it is a delivery artifact is
> unclear.

Perhaps this is a problem with my amavis configuration? It appears all
quarantined messages have a null Return-Path header.
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
On 2021-05-20 22:12, Alex wrote:

> Is it even possible for a sendgrid client to control their SPF record,
> let alone SPF HELO?

no, all next hop will change envelope sender

and sendgrid breaks dkim

> Perhaps it's because Return-Path is null?
> Return-Path: <>

return path <> would not give spf fail

all you can do is not use sendgrid
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
----- Message from Alan Hodgson <ahodgson@lists.simkin.ca> ---------
Date: Thu, 20 May 2021 13:48:48 -0700
From: Alan Hodgson <ahodgson@lists.simkin.ca>
Subject: Re: KAM_SENDGRID and SPF_HELO_NONE
To: users@spamassassin.apache.org

> And yes, SPF falls back to testing the HELO host if the envelope sender is
> empty (which should only occur in bounces or auto-responses).

Whilst the thread has passed on beyond this, this incorrect statement
needs to be corrected.

The SPF RFC states (rfc7208 2.3):

It is RECOMMENDED that SPF verifiers not only check the "MAIL FROM"
identity but also separately check the "HELO" identity by applying
the check_host() function (Section 4) to the "HELO" identity as the
<sender>. Checking "HELO" promotes consistency of results and can
reduce DNS resource usage. If a conclusive determination about the
message can be made based on a check of "HELO", then the use of DNS
resources to process the typically more complex "MAIL FROM" can be
avoided. Additionally, since SPF records published for "HELO"
identities refer to a single host, when available, they are a very
reliable source of host authorization status. Checking "HELO" before
"MAIL FROM" is the RECOMMENDED sequence if both are checked.

...and at 2.4:

SPF verifiers MUST check the "MAIL FROM" identity if a "HELO" check
either has not been performed or has not reached a definitive policy
result by applying the check_host() function to the "MAIL FROM"
identity as the <sender>.

A HELO SPF check is most certainly not a "fall-back".

Whether the SPF checking tool used follows the RFC is another matter
entirely :-)

Simon.

--
Simon Wilson
M: 0400 12 11 16
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
On 2021-05-20 at 18:24:51 UTC-0400 (Thu, 20 May 2021 18:24:51 -0400)
Alex <mysqlstudent@gmail.com>
is rumored to have said:

>>> I'm noticing what I think are a lot of false positives for this
>>> rule.
>>
>> In what way is this a false positive? Looks like a correct positive
>> to
>> me.
>
> Because it was a legitimate email with an invoice from a pest control
> company to their customer.

And it would have been marked as not spam, according to the hits you
showed, with the standard threshold value. Were there more hits?

Hitting one rule with a significant positive score is not a false
positive per se.

>> If you disagree with the scoring or purpose of that rule, you are
>> free
>> to reduce the score locally or discuss it with KAM. He's a very
>
> Nope, just trying to understand.

I think KAM addressed it: Sendgrid has become an objective empirical
indicator of spam. Not a perfect indicator, but the whole design theory
of SA is that we can put together a lot of imperfect indicators to make
the spam/ham judgment.

>> KAM's
>> QA is a 100% black box but he makes changes fast when needed.
>
> Yes, and just wanted to be sure that wasn't necessary here.

I don't see any reason that it would be.

>>> Perhaps it's because Return-Path is null?
>>> Return-Path: <>
>>
>> That's a different problem, apparently with your MTA->SA glue. The
>> fact
>> that something added a non-null "X-Envelope-From:" header and
>> something
>> (else?) added a null "Return-Path:" header indicates fundamental
>> breakage. Whether SA is seeing that or if it is a delivery artifact
>> is
>> unclear.
>
> Perhaps this is a problem with my amavis configuration? It appears all
> quarantined messages have a null Return-Path header.

If Amavis is doing the quarantining, it is doing so weirdly but I'd
guess that's probably a harmless delivery artifact, in the sense that it
is delivering mail to the quarantine with the null Return-Path after SA
has seen it. If it is quarantining messages that score less than 2 as
suspect spam, that's an unusual configuration choice designed to cause
unnecessary false positives.



--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
>> > Perhaps it's because Return-Path is null?
>> > Return-Path: <>
>>
>> That's a different problem, apparently with your MTA->SA glue. The fact
>> that something added a non-null "X-Envelope-From:" header and something
>> (else?) added a null "Return-Path:" header indicates fundamental
>> breakage. Whether SA is seeing that or if it is a delivery artifact is
>> unclear.

On 20.05.21 18:24, Alex wrote:
>Perhaps this is a problem with my amavis configuration? It appears all
>quarantined messages have a null Return-Path header.

I have checked 2 of my installations and both have Return-Path equivalent to
X-Envelope-From: and recipients in X-Envelope-To:

I assume amavis only uses X-Envelope-* when picking mail from quarantine and
that Return-Path is not important.

Why it's empty, no idea.


--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Remember half the people you know are below average.
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
Kevin A. McGrail wrote:
> And that rule is probably designed to hit legitimate sendgrid emails.
>
> They have become a hacker and spammer haven over the last year and a
> half approximately.
>
Damned straight.  I'd say more like 2.5 years, maybe 1.5 pre-pandemic years.

SendGrid -> novel (at thie time) Positive Delivery company.
SendGrid -> API opens up for quazi-spam/newsletter delivery..
SendGrid -> adds support for smaller ISPs and their infected customers.

For my part, I made some changes to my rules in CHAOS to differentiate
between the occurrence of a SendGrid header versus encapsulated SendGrid
headers like you'll get when larger mail systems populate the References
header for forwarding. Respectively, the rules set are JR_SGRID_DIRECT
and JR_SGRID_FWD. At least that seems to be a little more effective for
Comcast and BellSouth mail systems.

You just haven't lived until you've seen endless mailserver rejects
issued to SendGrid and SendGrid Partners  who are sending you Aaron
Smith Sextortions or Emotet variants.   If I'm a hostile, nation-state
actor,  I probably already have an account with SendGrid.

Nobody should be using SendGrid; NEVER, EVER.  One thing is certain, if
this matter is NOT addressed by the mail admins on this list, it WILL BE
addressed by the US Department of Commerce.

What started out as an interesting project has become a National
Security risk.


-- Jared Hall
Re: KAM_SENDGRID and SPF_HELO_NONE [ In reply to ]
Interesting for sure. For me I saw the issue start to really get noticed
last February.

I think there might be correlation with a hack on their platform too.

I reached out to Twilio leadership with nothing but crickets too.

Here is a great cyber security reporter and an article from August 2020:
https://krebsonsecurity.com/2020/08/sendgrid-under-siege-from-hacked-accounts/

What's amazing to me is how much they've done to fix the problem oh wait
they've done nothing...

-KAM


On Fri, May 21, 2021, 08:28 Jared Hall <jared@jaredsec.com> wrote:

> Kevin A. McGrail wrote:
> > And that rule is probably designed to hit legitimate sendgrid emails.
> >
> > They have become a hacker and spammer haven over the last year and a
> > half approximately.
> >
> Damned straight. I'd say more like 2.5 years, maybe 1.5 pre-pandemic
> years.
>
> SendGrid -> novel (at thie time) Positive Delivery company.
> SendGrid -> API opens up for quazi-spam/newsletter delivery..
> SendGrid -> adds support for smaller ISPs and their infected customers.
>
> For my part, I made some changes to my rules in CHAOS to differentiate
> between the occurrence of a SendGrid header versus encapsulated SendGrid
> headers like you'll get when larger mail systems populate the References
> header for forwarding. Respectively, the rules set are JR_SGRID_DIRECT
> and JR_SGRID_FWD. At least that seems to be a little more effective for
> Comcast and BellSouth mail systems.
>
> You just haven't lived until you've seen endless mailserver rejects
> issued to SendGrid and SendGrid Partners who are sending you Aaron
> Smith Sextortions or Emotet variants. If I'm a hostile, nation-state
> actor, I probably already have an account with SendGrid.
>
> Nobody should be using SendGrid; NEVER, EVER. One thing is certain, if
> this matter is NOT addressed by the mail admins on this list, it WILL BE
> addressed by the US Department of Commerce.
>
> What started out as an interesting project has become a National
> Security risk.
>
>
> -- Jared Hall
>
>
>
>
>
>
>