Mailing List Archive

On Tuesday 13 March 2007 08:49, John A. Martin wrote:
> >>>>> "Scott" == Scott Kitterman

> Scott> That was in the original page posted by Stuart Gathman.
> Scott> He, AFAIK, doesn't read spf-help, so I'd recommend you ask
> Scott> him on spf-discuss (it's a better topic for discuss
> Scott> anyway).
>
> Thank you, Scott. I am merely asking for information.
>
No problem. As a general rule, I think spf-help is a good list for 'How do
I ..." questions and spf-discuss is a good list for what/why questions.
Generally speaking the subscribers to spf-help know much less about e-mail
and SPF than those to spf-discuss and so it's more likely a free ranging
discussion about forwarding or other issues is going to cause confusion.

I'm curious too now. I hadn't noticed that before and I certainly didn't know
what it meant either.

Scott K

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Tue, 13 Mar 2007, John A. Martin wrote:

> Scott> On Monday 12 March 2007 18:52, John A. Martin wrote:
> >> Would someone please explain precisely what is meant at
> >> <http://www.openspf.org/Best_Practices/Forwarding> by the
> >> following and perhaps provide an illustrative example?
> >>
> >> If your implementation allows it, also check SPF for a
> >> "pretend" MAIL FROM that your forwarder could use. This
> >> verifies that the forwarded mail really came from your trusted
> >> forwarder.

Suppose MyForwarder is your alias forwarder. They do not use SRS.
However, they *do* control their own myforwarder.com domain (where your
forwarded address resides) with an SPF record. So, when you get mail, the MAIL
FROM will be

MAIL FROM: <joe@randomdomain.com>

*But*, before check SPF for randomdomain.com, you check SPF as if the
MAIL FROM was:

MAIL FROM: <postmaster@myforwarder.com>

instead.

If that gets a pass, then you know the mail was forwarded (and SPF checking
on the actual MAIL FROM is useless).

If myforwarder.com doesn't actually have an SPF record, then some
SPF libraries (e.g. pyspf) will allow you to supply a substitute that
you figure out and maintain yourself.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Tue, 13 Mar 2007, John A. Martin wrote:

> Scott> On Monday 12 March 2007 18:52, John A. Martin wrote:
> >> Would someone please explain precisely what is meant at
> >> <http://www.openspf.org/Best_Practices/Forwarding> by the
> >> following and perhaps provide an illustrative example?
> >>
> >> If your implementation allows it, also check SPF for a
> >> "pretend" MAIL FROM that your forwarder could use. This
> >> verifies that the forwarded mail really came from your trusted
> >> forwarder.
> >>

The explanation in pymilter says:

# Connections that get an SPF pass for a pretend MAIL FROM of
# postmaster@sometrustedforwarder.com skip SPF checks for the real MAIL FROM.
# This is for non-SRS forwarders. It is a simple implementation that
# is inefficient for more than a few entries.
trusted_forwarder = careerbuilder.com

And that is a real life example used by the HR person at a client.
Careerbuilder forwards email from job applicants without changing the
mail from.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Stuart D. Gathman wrote:

[before you]
> check SPF for randomdomain.com, you check SPF as if the MAIL FROM was:

> MAIL FROM: <postmaster@myforwarder.com>

> instead.

> If that gets a pass, then you know the mail was forwarded (and SPF
> checking on the actual MAIL FROM is useless).

I'd call that "white listing based on a HELO PASS" instead of some
"pretended MAIL FROM". The latter is fine to explain the concept
of SPF checks in the case of an empty MAIL FROM. But in your case
the MAIL FROM isn't empty.

> If myforwarder.com doesn't actually have an SPF record, then some
> SPF libraries (e.g. pyspf) will allow you to supply a substitute
> that you figure out and maintain yourself.

Is that something like a "wannabe-white list" for forwarders, adding
a "best guess" policy "v=spf1 a ?all" (or similar) for their HELO ?

Frank


-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Tue, 13 Mar 2007, Frank Ellermann wrote:

> > check SPF for randomdomain.com, you check SPF as if the MAIL FROM was:
>
> > MAIL FROM: <postmaster@myforwarder.com>
>
> > instead.
>
> > If that gets a pass, then you know the mail was forwarded (and SPF
> > checking on the actual MAIL FROM is useless).
>
> I'd call that "white listing based on a HELO PASS" instead of some
> "pretended MAIL FROM". The latter is fine to explain the concept
> of SPF checks in the case of an empty MAIL FROM. But in your case
> the MAIL FROM isn't empty.

Except it is *not* HELO, but the original rcpt to domain.

A -> user@forwarder.com -> final@receiver.com
Checks for PASS on forwarder.com
(HELO is something else like mx19.forwarder.com,
with no easy way to list them all)

> > If myforwarder.com doesn't actually have an SPF record, then some
> > SPF libraries (e.g. pyspf) will allow you to supply a substitute
> > that you figure out and maintain yourself.

> Is that something like a "wannabe-white list" for forwarders, adding
> a "best guess" policy "v=spf1 a ?all" (or similar) for their HELO ?

It is a local private repository of SPF records for senders that are
too lazy or backwards to do their own. I put my list in DNS where it
is accessible to all my clients. The SPF records are contructed by
observing where they send legit mail from and guessing. It is not
authoritative, and is labor intensive, but it gets the mail through for
lazy/backward senders that get forged a lot. (I personally would
not mind just rejecting their misconfigured mail, but my clients think
differently.)

For example, if _spf.example.com is a private substitute SPF repository,
then when SPF result for somesender.com is None, pyspf looks for an SPF
record at somesender.com._spf.example.com and uses that instead of
the default "best guess" policy if found. It is basically customized best
guess policies.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Stuart D. Gathman wrote:

>> I'd call that "white listing based on a HELO PASS"
[...]
> Except it is *not* HELO, but the original rcpt to domain.

> A -> user@forwarder.com -> final@receiver.com
> Checks for PASS on forwarder.com (HELO is something else
> like mx19.forwarder.com, with no easy way to list them all)

Oops, yes, I missed that important point. That requires a
list of "known forwarders for Y (forwarding mails to X)" (?)
You're probably not trying to find "for X" in a timestamp
line.

Each Y can have various X1, X2, X3, etc. where all mails to
X1, X2, X3, etc. are forwarded to Y. You're not interested
in local parts, so you'd use the (shorter) list of domains
D(X1), D(X2), D(X3), etc. without duplicates. Is that your
approach ?

> I put my list in DNS where it is accessible to all my clients.

How does that work for different users Y1 != Y2 with different
lists of forwarders ? Do you join "forwarding domains" over
all users ? That would have "interesting" security issues if
one bad apple in say Y1's basket could spam all other users.

I'd never guessed that a "pretended MAIL FROM" stands for
something like this idea to mitigate forwarding issues -
it's apparently in the direction of a "forward master plan".

Frank


-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Tue, 13 Mar 2007, Frank Ellermann wrote:

> > I put my list in DNS where it is accessible to all my clients.
>
> How does that work for different users Y1 != Y2 with different
> lists of forwarders ? Do you join "forwarding domains" over
> all users ? That would have "interesting" security issues if
> one bad apple in say Y1's basket could spam all other users.

It doesn't matter. The DNS repo is simply my approximation of SPF records that
senders *should* have published if they had a clue. Nothing to do
with forwarding, except for supplying an SPF record when the forwarder
won't.

> I'd never guessed that a "pretended MAIL FROM" stands for
> something like this idea to mitigate forwarding issues -
> it's apparently in the direction of a "forward master plan".

It solves non-SRS forwarding issues. Reputation is currently accrued against
the (pretend) forwarder domain.

I am working on a revision that would accrue reputation to the actual MAIL
FROM, but with the forwarder domain as the qualifier. Furthermore, if the
forwarder does do SPF (including SRS forwarders), then the reputation should be
charged to the domain qualified by SPF result contained in the Received-SPF
header. (For trusted forwarders only, of course.)

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Stuart D. Gathman wrote on Tuesday, March 13, 2007 11:53 AM -0600:

> A -> user@forwarder.com -> final@receiver.com
> Checks for PASS on forwarder.com
> (HELO is something else like mx19.forwarder.com,
> with no easy way to list them all)

The reason you have to apply the list of your users' trusted forwarders
is because the zone cut method for finding the parent domain of an HELO
FQDN is deprecated. Every outbound relay in a domain needs its own SPF
record to get a HELO pass, which is fine for domains that want this, but
a PITA when you have to supply your own local records for someone else's
domain.

This suggests another related guessing method: apply the list of
synthetic local SPF records during the HELO test when the HELO name does
not pass, and treat a pass as a HELO pass. Alternatively, treat a PASS
as it's own entity in the reputation system: guessed HELO. Yet another
alternative is to implement a zone cut algorithm for the HELO name, but
restrict it to the local DNS resolver to avoid abusing DNS. Since this
extension is clearly _not_ SPF (the sender did not publish this record),
the recipient is arguably not bound by the requirement to perform
checkhost() on MAIL FROM in the RFC in the case where the guessed HELO
pass result gives a strong enough reputation score to whitelist (or
blacklist).

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Wednesday 14 March 2007 04:51, Seth Goodman wrote:

> The reason you have to apply the list of your users' trusted forwarders
> is because the zone cut method for finding the parent domain of an HELO
> FQDN is deprecated.

It's actually stronger than that IMO. No such thing exists in SPF. If you
are doing zone cuts to find parent domains, you are not doing SPF.

Scott K

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Scott Kitterman wrote on Wednesday, March 14, 2007 7:02 AM -0600:

> On Wednesday 14 March 2007 04:51, Seth Goodman wrote:
>
> > The reason you have to apply the list of your users' trusted
> > forwarders is because the zone cut method for finding the parent
> > domain of an HELO FQDN is deprecated.
>
> It's actually stronger than that IMO. No such thing exists in SPF.
> If you are doing zone cuts to find parent domains, you are not
> doing SPF.

That's correct. Zone cuts are no longer part of SPF, though they
were in earlier drafts. The objection to this was the load on DNS.
If you restrict the search to a local resolver, or any other local
store, that's no longer a problem, but I didn't mean to imply that
this was SPF.

Actually, any kind of guess is contrary to SPF, since the recipient
provides the record, not the sender. Doing zone cuts, whether to
find an SPF record published by the sender for a parent domain, or
a local record provided by the recipient, are not SPF because the
recipient is guessing the sender's intent. We're just talking
about different variations of a best guess that might handle
forwards in the absence of SRS, and in the absence of SPF records
for the forwarder's HELO FQDN's.

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Wed, 14 Mar 2007, Seth Goodman wrote:

> Actually, any kind of guess is contrary to SPF, since the recipient
> provides the record, not the sender. Doing zone cuts, whether to
> find an SPF record published by the sender for a parent domain, or
> a local record provided by the recipient, are not SPF because the
> recipient is guessing the sender's intent. We're just talking
> about different variations of a best guess that might handle
> forwards in the absence of SRS, and in the absence of SPF records
> for the forwarder's HELO FQDN's.

Just to reiterate, the "pretend" MAIL FROM is *not* a HELO FQDN. It
is the actual forwarder domain, the original "rcpt to".

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Stuart D. Gathman wrote on Wednesday, March 14, 2007 9:18 AM -0600:

> Just to reiterate, the "pretend" MAIL FROM is *not* a HELO FQDN. It
> is the actual forwarder domain, the original "rcpt to".

Agreed, though you don't know the original RCPT TO, you guess what you
suppose it was for a forward. Your description sounds like prior to
testing SPF on MAIL FROM, it runs checkhost() on the connect IP against
a list of your users' non-SRS forwarders for whom you maintain local SPF
records. A pass is treated like a MAIL FROM pass for the forwarding
domain. This either provides direct indication of a forwarder you want
to whitelist, or is simply a way to accumulate and apply reputation data
for non-SRS forwarders. The end result is the same: you accept the
message as whitelisted (or deny as blacklisted, if you permit this) and
forgo testing the real MAIL FROM address.

The alternative I suggested was to test for a locally maintained domain
list at the end of the HELO test, *iff* there is no HELO pass. This is
only a slight variation to your method. Since it tests the real HELO
name first, it automatically degrades into normal SPF when the forwarder
publishes SPF records for their HELO names. Since you whitelist based
partly on locally maintained SPF records for specific domains, I think
you accomplish the same thing by doing this as part of the HELO test,
and that gives you a result earlier in the SMTP conversation. Frankly,
testing against any guessed identity that the sender didn't provide,
whether MAIL FROM or HELO, is equivalent to adding the IP's directly to
a local whitelist. Local SPF records are more maintainable than an IP
list, though, and easily transitions to using the forwarders' own SPF
records when they eventually publish them (thinking positively).

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Seth Goodman wrote:

> The alternative I suggested was to test for a locally maintained domain
> list at the end of the HELO test, *iff* there is no HELO pass. This is
> only a slight variation to your method.

Sounds like a good idea, like the very old op=trusted (formerly op=meng)
plan, but with the trust on the right (= receiver) side of the equation
(and therefore without modifier).

It won't help immediately if they add new HELO identities with or without
SPF policy for new mailouts, the receiver still has to maintain a set of
trusted MTAs. At least I now understand an older thread about "sets of
MTAs" started by David some weeks ago. That's a point where we have to
admit that SPF HELO checks are fine, but CSV would be better.

Frank


-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Frank Ellermann wrote on Wednesday, March 14, 2007 4:18 PM -0600:

> Seth Goodman wrote:
>
> > The alternative I suggested was to test for a locally maintained
> > domain list at the end of the HELO test, *iff* there is no HELO
> > pass. This is only a slight variation to your method.

> It won't help immediately if they add new HELO identities with or
> without SPF policy for new mailouts, the receiver still has to
> maintain a set of trusted MTAs. At least I now understand an older
> thread about "sets of MTAs" started by David some weeks ago.
> That's a point where we have to admit that SPF HELO checks are
> fine, but CSV would be better.

This is true, however, the HELO FQDN generally contains the parent
domain name, and that just might provide a way to emulate CSV and
automatically track non-SRS forwarders who publish SPF records. If
the reputation system were to look for entries, and create them if
they don't already exist, for each parent domain of the HELO FQDN,
the reputation system behaves as if it were CSV-aware and the
domains all published CSV records. This provides a number of
possible places to whitelist a forwarder by domain, and it would
automatically whitelist newly appearing MTA's that share a naming
structure. Even if they are not whitelisted, MTA's automatically
share reputation among other MTA's with related HELO names.

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

As a further observation, you don't strictly need SPF records to do
something like this. What you want is a confirmed HELO identity, and
that confirmation can be SPF, CSV, reverse DNS, forward DNS or
anything else you are willing to accept.

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Seth Goodman wrote:
> As a further observation, you don't strictly need SPF records to do
> something like this. What you want is a confirmed HELO identity, and
> that confirmation can be SPF, CSV, reverse DNS, forward DNS or
> anything else you are willing to accept.

Except that all of these have different notions of "confirmed". If you
don't care, fine. But otherwise, be aware that there is a difference.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFF+VqxwL7PKlBZWjsRAjVtAKDfiC6EFoT40/MBSHD8JgMh+DGIkwCgnQuy
1hNgOwvGjDbxpckRUngRW2Y=
=RbME
-----END PGP SIGNATURE-----

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Julian Mehnle wrote on Thursday, March 15, 2007 9:40 AM -0600:

> Seth Goodman wrote:
> > As a further observation, you don't strictly need SPF records to do
> > something like this. What you want is a confirmed HELO identity,
> > and that confirmation can be SPF, CSV, reverse DNS, forward DNS or
> > anything else you are willing to accept.
>
> Except that all of these have different notions of "confirmed". If
> you don't care, fine. But otherwise, be aware that there is a
> difference.

That's right, and I'm sure most everyone reading this cares about these
differences. Stuart previously presented a very rational hierarchy of
confirmation methods. The basic idea is to attempt confirmation with
the methods that you prefer first, and fall back to less stringent ones
later.

What methods are on the list, and in what order, is an individual
choice. My expectation is that SPF would be at the top of the list, as
it confirms that the domain owner designates a machine as permitted to
send mail. As Frank points out, CSV does this best, but it is
virtually dead. Other DNS methods can confirm the HELO identity, but
give a recipient no information about whether a machine is a legitimate
mail host.

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Thu, 15 Mar 2007, Seth Goodman wrote:

> This is true, however, the HELO FQDN generally contains the parent
> domain name, and that just might provide a way to emulate CSV and
> automatically track non-SRS forwarders who publish SPF records. If

If the forwarder publishes an SPF record for their domain (not the HELO
FQDN), then there is no problem. It doesn't matter if they publish
an SPF record for HELO names, because a proper HELO FQDN matches
the connect IP anyway. The pymilter best_guess algorithm already
matches correct HELO FQDNs against the domain:

if res != 'pass' and hres == 'pass' and spf.domainmatch([q.h],q.o):
res = 'pass' # get a guessed pass for valid matching HELO

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Stuart D. Gathman wrote on Thursday, March 15, 2007 11:55 AM -0600:

> On Thu, 15 Mar 2007, Seth Goodman wrote:
>
> > This is true, however, the HELO FQDN generally contains the parent
> > domain name, and that just might provide a way to emulate CSV and
> > automatically track non-SRS forwarders who publish SPF records. If
>
> If the forwarder publishes an SPF record for their domain (not the
> HELO FQDN), then there is no problem. It doesn't matter if they
> publish an SPF record for HELO names, because a proper HELO FQDN
> matches the connect IP anyway. The pymilter best_guess algorithm
> already matches correct HELO FQDNs against the domain:
>
> if res != 'pass' and hres == 'pass' and
> spf.domainmatch([q.h],q.o): res = 'pass' # get a guessed pass for
> # valid matching HELO

What I suggested operates the same as this when the forwarder publishes
SPF for their mail domain, or you construct local SPF records for
trusted forwarders who don't publish SPF, *and* you're willing to check
*all* incoming messages against the whole list of trusted forwarders
(or until you get a match). Maintaining local SPF records for your
forwarders is a significant burden and testing all messages against a
list of known forwarders could consume a good deal of resources. The
suggestion was an attempt to mitigate these two issues.

Regarding the need to identify all your users' forwarders, and then
construct SPF records for forwarders that don't publish SPF,
automatically creating reputation database entries for parent domains
of confirmed HELO FQDN's may accomplish the goal most of the time. It
may not yield the actual mail domain of the forwarder, but this doesn't
matter because the return-path doesn't include that domain anyway.

The system would create reputation database entries for parent domains
of confirmed HELO FQDN's that handed you mail. Those entries represent
the mail flow of all the forwarder's MTA's that have related HELO
FQDN's. If the mail flow from a forwarder is very spammy and your
reputation system begins to refuse their mail, the task now is to find
the relevant parent domain that is already in your database and
whitelist it. As long as the forwarder's MTA's have related HELO
FQDN's, you shouldn't have to create or maintain local SPF records for
them, and you shouldn't have to preemptively test all known
forwarding domains for every incoming message. This is somewhat more
correct (in a strict sense) than using a "pretend" MAIL FROM domain,
since the reputation applied in this case is from a HELO identity,
and it is explicitly tracked that way.

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Thu, 15 Mar 2007, Seth Goodman wrote:

> (or until you get a match). Maintaining local SPF records for your
> forwarders is a significant burden and testing all messages against a
> list of known forwarders could consume a good deal of resources. The
> suggestion was an attempt to mitigate these two issues.

Sequentially comparing connections to a list of forwarder SPF records
is a naive implementation, and only workable for a short list.
The smart implementation is to compile the SPF records into IP sets,
like libspf2 does (and which pyspf will do when I get a round tuit).

> Regarding the need to identify all your users' forwarders, and then
> construct SPF records for forwarders that don't publish SPF,
> automatically creating reputation database entries for parent domains
> of confirmed HELO FQDN's may accomplish the goal most of the time. It
> may not yield the actual mail domain of the forwarder, but this doesn't
> matter because the return-path doesn't include that domain anyway.

You have to know when not to reject on SPF fail. Hence the non-SRS
forwarder list.

> The system would create reputation database entries for parent domains
> of confirmed HELO FQDN's that handed you mail. Those entries represent

Except I'd be rejecting most of them on SPF fail.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Stuart D. Gathman wrote on Thursday, March 15, 2007 4:38 PM -0600:

> On Thu, 15 Mar 2007, Seth Goodman wrote:
>
> > Regarding the need to identify all your users' forwarders, and then
> > construct SPF records for forwarders that don't publish SPF,
> > automatically creating reputation database entries for parent
> > domains of confirmed HELO FQDN's may accomplish the goal most of
> > the time. It may not yield the actual mail domain of the
> > forwarder, but this doesn't matter because the return-path doesn't
> > include that domain anyway.
>
> You have to know when not to reject on SPF fail. Hence the non-SRS
> forwarder list.
>
> > The system would create reputation database entries for parent
> > domains of confirmed HELO FQDN's that handed you mail. Those
> > entries represent
>
> Except I'd be rejecting most of them on SPF fail.

Starting off with the idea that you will reject on SPF fail unless the
sender is whitelisted is one way to solve the problem. This certainly
works, but it's not terribly practical for large systems. As you have
shown, it requires the recipient to track all their users' forwarding
arrangements and create SPF records for forwarders that don't publish
themselves. This remains a substantial burden for large systems.
Compiling the locally maintained SPF records into IP whitelists reduces
the run-time penalty, but not the burden of tracking your forwarders'
MTA's. This suggestion is an attempt to get around that requirement for
SPF in the presence of alias forwarding.

One approach is to move beyond the idea that you reject on SPF fail for
MAIL FROM unless whitelisted. With a real reputation system, you can
track reputation for each confirmed identity and whether to accept mail
based on any applicable reputations, plus other metrics. You can use
the "highest" confirmed identity, or combine results from all confirmed
identities for a given sender. The reputation system you described
tracks reputation for a variety of identities from MAIL FROM all the way
down to IP. Because of the prevalence of alias forwarding, SPF fail on
MAIL FROM can be viewed as the absence of a confirmed MAIL FROM identity
to assess for reputation. Other confirmed identities for a sender may
still have sufficient positive reputation to accept mail from them. In
the case of legitimate alias forwards, you can presumably confirm the
HELO identity and track reputation for each HELO FQDN from whom you have
accepted any mail. When there is no MAIL FROM domain to query for
reputation (non-SRS alias forwarding), it sounds possible to accumulate
enough reputation on the parent domains of your forwarders' HELO FQDN's
that they would effectively whitelist themselves in groups without
operator intervention.

In some of the cases where forwarding domains don't whitelist themselves
through this mechanism, for example because they forward lots of spam
but your users expect you to accept and filter it, you can then
whitelist the domain manually. The difference is that the reputation
database would already have an entry for an appropriate HELO FQDN parent
domain, so you don't have to create or maintain SPF records for these
forwarders. New MTA's for each forwarder are immediately associated
with the reputation, or whitelisting status, of the HELO FQDN parent
domain. You may have to create an SPF record in the odd case where a
forwarder's outbound relays have no common parent domain in their HELO
FQDN's, but that should be a much smaller group of domains. This
basically tricks the reputation system into behaving like there are CSV
records and tracking group reputation for related MTA's.

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Thu, 15 Mar 2007, Seth Goodman wrote:

> > Except I'd be rejecting most of them on SPF fail.
>
> Starting off with the idea that you will reject on SPF fail unless the
> sender is whitelisted is one way to solve the problem. This certainly
> works, but it's not terribly practical for large systems. As you have

Such a policy would be *per recipient*, as I outlined in best practices.
If a mail user didn't setup forwarders, you don't reject on SPF fail
for that user.

> One approach is to move beyond the idea that you reject on SPF fail for

While large systems can fall back to IP blacklists, that is not practical
for a small system, and rejection on FAIL is essential. A large system
would of course not reject on fail for every user, as I have said
many times. In best practices, we recommend that large system
delay checking SPF until RCPT TO so that such policies can be
per user.

> down to IP. Because of the prevalence of alias forwarding, SPF fail on
> MAIL FROM can be viewed as the absence of a confirmed MAIL FROM identity
> to assess for reputation. Other confirmed identities for a sender may
> still have sufficient positive reputation to accept mail from them. In

The goal here is to reject *more* mail. There is too much as it is.
You suggestion might be appropriate for a large system for specific
users that haven't configured forwarders. Admittedly, probably most
of them - but at least power users get some real SPF fail rejection.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

Stuart D. Gathman wrote on Thursday, March 15, 2007 8:54 PM -0600:

> On Thu, 15 Mar 2007, Seth Goodman wrote:
>
> > One approach is to move beyond the idea that you reject on SPF fail
>
> While large systems can fall back to IP blacklists, that is not
> practical for a small system, and rejection on FAIL is essential.

I'm not suggesting reverting to IP blacklists. I'm exploring the
possibility of automating forwarder whitelisting, because as you
correctly point out:

> A large system would of course not reject on fail for every user, as
> I have said many times. In best practices, we recommend that large
> system delay checking SPF until RCPT TO so that such policies can be
> per user.

Since large systems are unlikely to whitelist forwarders per user,
and global forwarder whitelisting means too much additional spam,
the recommended best practice is not realistic for large systems.


> > Because of the prevalence of alias forwarding, SPF fail on MAIL
> > FROM can be viewed as the absence of a confirmed MAIL FROM
> > identity to assess for reputation. Other confirmed identities
> > for a sender may still have sufficient positive reputation to
> > accept mail from them.
>
> The goal here is to reject *more* mail. There is too much as it is.
> You suggestion might be appropriate for a large system for specific
> users that haven't configured forwarders. Admittedly, probably most
> of them - but at least power users get some real SPF fail rejection.

We obviously agree on the goal. Rejecting on SPF fail, except for
whitelisted forwarders on a per user basis, is certainly one way to
accomplish this. It's similar to other methods that are overly
aggressive and use whitelists to mitigate the damage: very effective
but not practical for large systems. It's impractical because
tracking user forwarding arrangements for large numbers of users is
burdensome and nobody wants to alienate users.

SPF could be much more than a technique for small systems with a few
power users. The argument that alias forwarding is broken and we'd be
better off without it has technical merit, but tying SPF to this turns
it into another superior technical concept that is never widely used.
Simply put, alias forwarding in its present form was an enormous
mistake, yet we are probably stuck with it as long we have SMTP.

Other fixes to the forwarding problem, like SRS at forwarders and SES
at senders, do solve their specific problems but don't give recipients
assurance that they won't reject their users forwards. A recipient-
side solution has a better chance of changing this situation. I
don't know if automated whitelisting of forwarders is even feasible,
but that's my motivation for exploring it.

--
Seth Goodman

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735

On Fri, 16 Mar 2007, Seth Goodman wrote:

> Since large systems are unlikely to whitelist forwarders per user,
> and global forwarder whitelisting means too much additional spam,
> the recommended best practice is not realistic for large systems.

Large systems that I am familiar with (e.g. spamsoap.com) *already*
whitelist senders per user. The only change required is to
flag some senders as forwarders and apply the "pretend" sender rule.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
Sender Policy Framework: http://www.openspf.org/
Archives at http://archives.listbox.com/spf-discuss/current/
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=735