Mailing List Archive

99.95% of all SPF records use no macros
Hi, <http://spf-all.com/stats.html> is quite fascinating,
it says that over 99.95% of about 2,500,000 evaluated SPF
records use no macros at all. Of course this can't tell
us how much FAIL-protected mails desperately need macros.

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: 99.95% of all SPF records use no macros [ In reply to ]
On Wed, 23 Jul 2008, Frank Ellermann wrote:

> Hi, <http://spf-all.com/stats.html> is quite fascinating,
> it says that over 99.95% of about 2,500,000 evaluated SPF
> records use no macros at all. Of course this can't tell
> us how much FAIL-protected mails desperately need macros.

Most things macros can do, a machine generated zonefile (static
or dynamic) can do as well or better. Macros mainly give the policy
access to localpart and helo (both of which are more likely to tickle
SPF implementation bugs due to their unconstrained evilness).

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: 99.95% of all SPF records use no macros [ In reply to ]
Stuart D. Gathman wrote:

> Most things macros can do, a machine generated zonefile (static
> or dynamic) can do as well or better. Macros mainly give the policy
> access to localpart and helo (both of which are more likely to tickle
> SPF implementation bugs due to their unconstrained evilness).

How about deprecating localpart macros when we ever get around
to post a 4408bis fixing the errata ? Most text in the SPF-EAI
draft is about localpat weirdness - and that without copying a
single word of Doug's macro rants... :-)

BTW, the next three EAI RFCs (defining the real thing, UTF8SMTP,
among others) were recently approved. More important, 2821bis
was also approved.

Related, did you see my proposal for three additional macro test
cases for the test suite on the devel list ?

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Re: 99.95% of all SPF records use no macros [ In reply to ]
Frank Ellermann wrote:
> Stuart D. Gathman wrote:
>
>> Most things macros can do, a machine generated zonefile (static
>> or dynamic) can do as well or better. Macros mainly give the policy
>> access to localpart and helo (both of which are more likely to tickle
>> SPF implementation bugs due to their unconstrained evilness).
>
> How about deprecating localpart macros when we ever get around
> to post a 4408bis fixing the errata ? Most text in the SPF-EAI
> draft is about localpat weirdness - and that without copying a
> single word of Doug's macro rants... :-)

As a user of localpart macros, I would be unhappy about this. Why do I
use localpart? I have a fail rule that (essentially) says that invalid
users in my domain may not send mail (from anywhere). This cuts down on
invalid mail and allows me to be laxer about allowing forwarders that
don't do SRS.

Philip



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: 99.95% of all SPF records use no macros [ In reply to ]
Philip Gladstone wrote:

[How about deprecating localpart macros]
> As a user of localpart macros, I would be unhappy about this.

IMO "deprecate" would mean that new SPF applications are free
to honour or ignore SPF policies using this feature. They
could abort SPF checks with result NONE when they see a local-
part macro outside of explanations, for example.

They can of course already do this today as part of a "I don't
like localpart macros" receiver policy. But RFC 4408 does not
say how to implement "unsupported feature". It is certainly no
PERMERROR, "no match and move on" also won't cut it, so I think
it is a special case of NONE.

> Why do I use localpart? I have a fail rule that (essentially)
> says that invalid users in my domain may not send mail (from
> anywhere). This cuts down on invalid mail and allows me to be
> laxer about allowing forwarders that don't do SRS.

That sounds as if you have "?exists:%[.l}.aux.%[.d} -all" to get
NEUTRAL instead of FAIL for existing local parts. Or the other
way around: "-exists:%[l].aux.%{d} ?all" to get FAIL for any
non-existing local part. BTW, that would be very near to SES.

I'd prefer either PASS or FAIL as receiver, in simple policies.
SPF always had it clear that forwarding to 3rd parties without
indicating the responsibility violates the original spirit of
this feature in RFC 821.

Frank
--
Unrelated, I fear this list does something with your mail that
isn't compatible with your signature.



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Re: 99.95% of all SPF records use no macros [ In reply to ]
On Thursday 24 July 2008 03:40, Frank Ellermann wrote:
> Philip Gladstone wrote:
>
> [How about deprecating localpart macros]
>
> > As a user of localpart macros, I would be unhappy about this.
>
> IMO "deprecate" would mean that new SPF applications are free
> to honour or ignore SPF policies using this feature. They
> could abort SPF checks with result NONE when they see a local-
> part macro outside of explanations, for example.

I would be against this. I'm not aware of any library that currently has
trouble with macros. Their use may be limited, but they are important to
support certain use cases. I'd be a lot more excited about deprecating ip4
and ip6 and making a apply to IP addresses too. There is really no advantage
to having the split out and it leads to a lot of mistakes in records.

Scott K


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: 99.95% of all SPF records use no macros [ In reply to ]
Scott Kitterman wrote:

[not only new]
>> SPF applications are free to honour or ignore SPF policies
>> using this feature. They could abort SPF checks with result
>> NONE when they see a localpart macro outside of explanations,
>> for example.

> I would be against this. I'm not aware of any library that
> currently has trouble with macros.

Doug's "SPF scripts take DNS down" nightmares were certainly
exaggerated. But he had a point wrt abuse obscured by local-
part macros. Most SPF macros are harmless, but %{l} is not.

I fear that no SPF implementations gets some weirder macro
cases right. For "\"take\\that\""@example.com RFC 2821bis
(until yesterday RFC 2821) said that the "semantic content"
is "take\that" (including the quotes).

IIRC Julian or you said that you remove outer quotes, after
that step you are at \"take\\that\". Do you also trim the
odd backslashes in a <target-name> macro expansion of %{l} ?

You don't support embedded dots as in "take.that", i.e. you
treat that as two labels take.that instead of only one label
take\.that.

Consequently "do..ts" won't work, now covered by an erratum.
My confidence that similar strange local parts (without the
known dots cases) work consistently is limited. DNS can do,
in theory, but can existing SPF implementations ?

What about "space me"@example.com with two spaces ? That's
where even RFC 2821bis and I-D 2822upd gave up, it is legal
(syntactically), but might not work as expected. At least
RFC 2821bis killed control characters including TAB for this
madness. I-D 2822upd didn't, nobody had the energy to fix
this discrepancy.

With your logic (as implemented, this is unspecified) you'd
strip the quotes and arrive at space me (with two embedded
spaces). No problem for DNS, but does this really work, are
there test cases for it ?

Of course not only %{l} can have this effect, strange HELOs
- actually syntactically invalid HELOs for RFC 2821bis - can
also result in tricky %{h} cases. Ditto %{s}, that is worse
than %{l} if policies use it in a <domain-spec> expanded into
<target-name>.

If these three macros (or rather six, h+l+s+H+L+S) would be
limited to explanations they'd be fine. But this is not the
case.

> Their use may be limited, but they are important to support
> certain use cases.

For Doug's DDoS theories. In practice see subject, less than
0.5 permille use *any* macros, if this statistics is correct.

I've serious doubts wrt the interoperability for quoted pairs
and quoted strings - confirmed by 2821bis and its predecessor.

> I'd be a lot more excited about deprecating ip4 and ip6 and
> making a apply to IP addresses too.

That would get you in a syntactical mess. a: supports macros,
it comes with a <domain-spec>. But ip4/ip6 don't. So you'd
get a first special case "if it contains macros it is no IP".

After that IPv6 and IPv4 can or must have a dot. Currently
the syntax insists on at least one dot to figure out where
the <toplabel> is. So you'd need a clear rule what is and
what is not a <toplabel>. The IETF has so far no consensus
about this. IOW I thought John got it right in RFC 3696, but
John says he got this wrong, and lots of other folks starting
with ICANN have lots of other ideas not necessarily related
to technical questions.

[.If you're interested in the bloody details, RFC 4408 allows
0xFF as <toplabel>. Some applications including Firefox
treat that as 255 in an IPv4 if they can. So John is right,
his RFC 3696 and our RFC 4408 were wrong. Shit happens.]

What is worse, ICANN intends to introduce hundreds, thousands,
or more TLDs. Possibly some of these TLDs will be hosts, but
SPF insists on at least one dot like the old RFC 2821. The
new RFC 2821bis still says (in prose) that "no dot" might not
work as expected, but in theory it allows this. But RFC 4408
doesn't. IMO no practical problem yet, and maybe irrelevant
forever, but not a situation where we should try to make this
worse wrt future developments.

IMNSHO, if the publisher of a sender policy does not know the
difference between "name" (a) and "address" (ip4 or ip6), then
he has no business to publish a policy, because he is clueless
and should find someone who knows the basics. Even our wizard
gets this right, and that is a script.

SPF has the potential to delete legit mails, it is a dangerous
tool. Folks confusing "a" and "ip4" should buy a product or
get professional help. You've written a validator. Folks not
willing or unable to check their policy before going live with
an SPF FAIL are a public danger.

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Re: 99.95% of all SPF records use no macros [ In reply to ]
On Thursday 24 July 2008 16:35, Frank Ellermann wrote:
> Scott Kitterman wrote:
>
> [not only new]
>
> >> SPF applications are free to honour or ignore SPF policies
> >> using this feature. They could abort SPF checks with result
> >> NONE when they see a localpart macro outside of explanations,
> >> for example.
> >
> > I would be against this. I'm not aware of any library that
> > currently has trouble with macros.
>
> Doug's "SPF scripts take DNS down" nightmares were certainly
> exaggerated. But he had a point wrt abuse obscured by local-
> part macros. Most SPF macros are harmless, but %{l} is not.
>
> I fear that no SPF implementations gets some weirder macro
> cases right. For "\"take\\that\""@example.com RFC 2821bis
> (until yesterday RFC 2821) said that the "semantic content"
> is "take\that" (including the quotes).

Yes, but that doesn't change RFC 4408.

> IIRC Julian or you said that you remove outer quotes, after
> that step you are at \"take\\that\". Do you also trim the
> odd backslashes in a <target-name> macro expansion of %{l} ?
>
> You don't support embedded dots as in "take.that", i.e. you
> treat that as two labels take.that instead of only one label
> take\.that.
>
> Consequently "do..ts" won't work, now covered by an erratum.
> My confidence that similar strange local parts (without the
> known dots cases) work consistently is limited. DNS can do,
> in theory, but can existing SPF implementations ?
>
> What about "space me"@example.com with two spaces ? That's
> where even RFC 2821bis and I-D 2822upd gave up, it is legal
> (syntactically), but might not work as expected. At least
> RFC 2821bis killed control characters including TAB for this
> madness. I-D 2822upd didn't, nobody had the energy to fix
> this discrepancy.
>
> With your logic (as implemented, this is unspecified) you'd
> strip the quotes and arrive at space me (with two embedded
> spaces). No problem for DNS, but does this really work, are
> there test cases for it ?
>
> Of course not only %{l} can have this effect, strange HELOs
> - actually syntactically invalid HELOs for RFC 2821bis - can
> also result in tricky %{h} cases. Ditto %{s}, that is worse
> than %{l} if policies use it in a <domain-spec> expanded into
> <target-name>.
>
> If these three macros (or rather six, h+l+s+H+L+S) would be
> limited to explanations they'd be fine. But this is not the
> case.

Philip Gladstone's SPF record drove a lot of bug fixes into pyspf when I first
pubilshed my validator in July 2005. If we're missing stuff in the test
suite, let's add it, but AFAIK all the major libraries that are maintained,
get the test cases we have correct.

4408 says what it says and we need to make sure we've done the best we can
with it.

Scott K



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: 99.95% of all SPF records use no macros [ In reply to ]
Scott Kitterman wrote:

>> For "\"take\\that\""@example.com RFC 2821bis (until yesterday
>> RFC 2821) said that the "semantic content" is "take\that"
>> (including the quotes).

> Yes, but that doesn't change RFC 4408.

AFAIK RFC 4408 doesn't say anything about it, it has a normative
reference to RFC 2821. IOW if you don't arrive at the DNS label
"take\that" (11 characters) it is wrong. Quoting RFC 2821bis:

| Note that the backslash, "\", is a quote character, which is used to
| indicate that the next character is to be used literally (instead of
| its normal interpretation). For example, "Joe\,Smith" indicates a
| single nine character user field with the comma being the fourth
| character of the field.

We all had more pressing needs than to care about quoted string
oddities when Wayne's drafts were on their way to RFC 4408. And
in theory 2281(bis) sayd or says it all. But in practice I fear
that the "embedded dot" erratum is not the only oddity we missed.

> If we're missing stuff in the test suite, let's add it, but
> AFAIK all the major libraries that are maintained, get the test
> cases we have correct.

Unless I missed it we have yet no test cases for the errata, and
the only two backslashes in the test suite check that <toplabel>
cannot contain backslash, so that's not about quoted pairs in a
<Quoted-string>. Actually we have no <Quoted-string> local part
at all in the test suite.

> 4408 says what it says and we need to make sure we've done the
> best we can with it.

I've submitted three simple "macro mania" test cases some weeks
ago, when a developer wanted to get away with a broken DNS API:
<http://article.gmane.org/gmane.mail.spam.spf.devel/1923>

For <Quoted-string> and backslashes I'm not sure in what form
YAML would accept this, maybe hex. notation could work. I've
only looked briefly in the YAML spec. - I'm sure that this is
possible, but immediately forgot how... :-)

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: 99.95% of all SPF records use no macros [ In reply to ]
Scott Kitterman wrote:

> I've submitted three simple "macro mania" test cases some weeks
> ago, when a developer wanted to get away with a broken DNS API:
> <http://article.gmane.org/gmane.mail.spam.spf.devel/1923>

> For <Quoted-string> and backslashes I'm not sure in what form
> YAML would accept this, maybe hex. notation could work. I've
> only looked briefly in the YAML spec. - I'm sure that this is
> possible, but immediately forgot how... :-)

The problem was what form the mail from would be presented to the API.
In case of a policy/milter daemon, perhaps the MTA has already dequoted
the localpart. However, since sendmail presents the entire MFROM as a
single string, I think we must assume that localpart is still quoted for
at least some use cases - or we couldn't know which '@' is the real one.
So adding some quoted test cases would be good.



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Re: 99.95% of all SPF records use no macros [ In reply to ]
Frank Ellermann wrote:
> Scott Kitterman wrote:
>
> [not only new]
>>> SPF applications are free to honour or ignore SPF policies
>>> using this feature. They could abort SPF checks with result
>>> NONE when they see a localpart macro outside of explanations,
>>> for example.
>
>> I would be against this. I'm not aware of any library that
>> currently has trouble with macros.

+1, I don't think it's smart to abjure released features.

> Doug's "SPF scripts take DNS down" nightmares were certainly
> exaggerated. But he had a point wrt abuse obscured by local-
> part macros. Most SPF macros are harmless, but %{l} is not.

Being local, it shouldn't harm DNSs. Correct?

> I fear that no SPF implementations gets some weirder macro
> cases right. For "\"take\\that\""@example.com [...]
> [...] nobody had the energy to fix this discrepancy.

Then, we shouldn't be more royalist than the king. I'd classify those
as problematic local parts rather than reproach %{l}...

> If these three macros (or rather six, h+l+s+H+L+S) would be
> limited to explanations they'd be fine. But this is not the
> case.

Do you mean they should not affect the result?

>> Their use may be limited, but they are important to support
>> certain use cases.
>
> For Doug's DDoS theories. In practice see subject, less than
> 0.5 permille use *any* macros, if this statistics is correct.

I don't think that's correct. One should count the number of times a
record has been used. However, there's no practical way to trace those
calls: SPF lacks support for statistics, as well as for debugging.
Should such issues be addressed in a future release?

Ale


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: 99.95% of all SPF records use no macros [ In reply to ]
Alessandro Vesely wrote:

>>> I would be against this. I'm not aware of any library
>>> that currently has trouble with macros.

> +1, I don't think it's smart to abjure released features.

As operator I'd use a feature to disable localpart macros
if available, and as implementor I'd consider to add this
feature. I'm biassed after about a dozen flamewars with
Doug Otis on various lists with topic "SPF scripts".

It's also no fun to work on the SPF-EAI draft when half
of it is about localpart issues, they get worse for EAI.
Anything else in SPF works like a charme - also for EAI.

Okay, impementations need the IDNAbis U-label to A-label
(punycode) magic "somewhere", and it would be a pain to
implement this from scratch. But in practice this boils
down to "find libidn", or use ICU, or whatever supports
IDNA(bis) on your platform.

Joke, one thing SPF should not say is that "EAI is only
an obscure IETF experiment" ;-) DKIM cannot support EAI
so far, as usual SPF is the only thing that really works.
At least in theory, excluding localpart macro oddities.

> Being local, it shouldn't harm DNSs. Correct?

NAK. It's hard to extract what Doug really means buried
under tons of *FUD* and a peculiar terminology. Julian
tried to translate this into plain English, check out:
http://www.openspf.org/draft-otis-spf-dos-exploit_Analysis

Doug's concern wrt SPF localpart macros is that once a
receiver has a policy with this feature in the DNS cache,
that the cached record can cause various DNS queries to
a victim triggered by varying local parts in an attack.

In an attack the SPF policy doesn't need to make sense.
It can be v=spf1 a:%{l}.a0.victim.example etc. up to a9,
ten queries per unique local part. The attacker owns a
botnet and sends 100,000 mails with unique local parts,
resulting in a million (bogus) queries to the victim.

Modify "100,000" until this starts to cause real harm.
Make sure to use long local parts for longer queries,
or use longer labels than "a0" to "a9". Doug's idea is
to cause as much DNS traffic as possible to the victim.

From various receivers checking SPF for the DDoS effect,
blocking a DoS attack amplified by only one receiver is
too simple.

The localpart macro is required. The queries need to be
different for each mail, otherwise the NXDOMAIN answers
could be cached for some time, resulting in only ten DNS
queries per receiver instead of a million (for 100,000).

Doug's proposal is far more complex involving MX queries
to the attacker, but that is only because he wanted more
*FUD* about an SPF amplification factor greater than 10.
But the factor 10 is no nonsense, it is pure RFC 4408.

>> [...] nobody had the energy to fix this discrepancy.
> Then, we shouldn't be more royalist than the king.

Blasphemy, there are no kings in the IETF... ;-) SPF is
anyway not interested in 2822upd syntax. Otherwise we'd
intervene to get "white space in quoted string" questions
answered. No HTABs as far as RFC 2821bis is concerned,
good enough also for SPF.

> I'd classify those as problematic local parts rather
> than reproach %{l}...

Your classification is certainly confirmed by 2821(bis),
but there's no clear "don't use problematic local parts
if you use %{l}" rule in RFC 4408.

Besides spammers would ignore such a rule if this offers
them a way to bypass SPF policies relying on %{l} macros.

>> If these three macros (or rather six, h+l+s+H+L+S)
>> would be limited to explanations they'd be fine. But
>> this is not the case.

> Do you mean they should not affect the result?

In this statement I meant that these macros are harmless
if used in SPF FAIL explanations behind exp= modifiers.

Elsewhere, "get rid of them" boils down to "abort your
SPF check with a special NONE result whenever a policy
does something you don't like".

E.g. if a sender uses the exists: mechanism to enforce
checks of specific DNSBLs the receivers are free to say:

"Gee, thanks, but *I* decide which DNSBLs I use. That
your SPF policy is syntactically valid does not mean
that I'm *forced* to support your DNSBL choices."

BTW, it also doesn't mean that the receiver is *allowed*
to use a specific DNSBL used in an exists:. Many DNSBL
operators have their own policies wrt DNS query traffic.

>> In practice see subject, less than 0.5 permille use
>> *any* macros, if this statistics is correct.

> I don't think that's correct. One should count the number
> of times a record has been used. However, there's no
> practical way to trace those calls: SPF lacks support for
> statistics, as well as for debugging.

You can certainly log and analyze DNS queries to your own
name servers, and one of the proposed uses of SPF macros
is to get precisely this logging affect. For an example
see the policy of schlitt.net, also explained in RFC 4408.

OTOH why should recivers interested in PASS or FAIL spend
parts of their time and bandwidth to support your logging
efforts ? That has the decent charme of tracking cookies
and "Web bugs"... :-(

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: SPF-EAI (was: Re: 99.95% of all SPF records use no macros) [ In reply to ]
Frank Ellermann wrote:
> It's also no fun to work on the SPF-EAI draft when half
> of it is about localpart issues, they get worse for EAI.
> Anything else in SPF works like a charme - also for EAI.

The localpart may require different encodings if it is used for DNS
lookups, as a CGI parameter in a URL, or for internally looking up the
user db. The third case is not interesting for SPF, but it may be so
for an MTA if it does other funny things with a user's ID, e.g. digest
auth. Unfortunately, the alphabet we use doesn't allow a third case
beyond %{l} and %{L}. Does that call for additional macro transformers?

> Okay, impementations need the IDNAbis U-label to A-label
> (punycode) magic "somewhere", and it would be a pain to
> implement this from scratch. But in practice this boils
> down to "find libidn", or use ICU, or whatever supports
> IDNA(bis) on your platform.

When spf-eai says "The local part MUST NOT be transformed" it implies
that %{l} cannot be used for DNS lookups if internationalized local
parts are present. I would tend to agree, as I think people sharing
the same domain name should agree on a common policy. However, that
might be an issue for larger organizations whose users disagree about
what SPF policy they want.

To simplify, consider the case of 50% users prising "v=spf1 +all",
while the other 50% insist on "v=spf1 +mx -all". What should the
postmaster do?

> [...]
> In an attack the SPF policy doesn't need to make sense.
> It can be v=spf1 a:%{l}.a0.victim.example etc. up to a9,
> ten queries per unique local part. The attacker owns a
> botnet and sends 100,000 mails with unique local parts,
> resulting in a million (bogus) queries to the victim.

OTOH the attackers will have a hard time explaining to a court why
exactly they did publish that policy.

What is unfair in that scenario is that the risk is not burdening the
server that publishes the risky policy.

>> I'd classify those as problematic local parts rather
>> than reproach %{l}...
>
> Your classification is certainly confirmed by 2821(bis),
> but there's no clear "don't use problematic local parts
> if you use %{l}" rule in RFC 4408.

The term "problematic" doesn't appear in spf-eai either.

> Besides spammers would ignore such a rule if this offers
> them a way to bypass SPF policies relying on %{l} macros.
>
>>> If these three macros (or rather six, h+l+s+H+L+S)
>>> would be limited to explanations they'd be fine. But
>>> this is not the case.
>
>> Do you mean they should not affect the result?
>
> In this statement I meant that these macros are harmless
> if used in SPF FAIL explanations behind exp= modifiers.
>
> Elsewhere, "get rid of them" boils down to "abort your
> SPF check with a special NONE result whenever a policy
> does something you don't like".
>
> E.g. if a sender uses the exists: mechanism to enforce
> checks of specific DNSBLs the receivers are free to say:
>
> "Gee, thanks, but *I* decide which DNSBLs I use. That
> your SPF policy is syntactically valid does not mean
> that I'm *forced* to support your DNSBL choices."
>
> BTW, it also doesn't mean that the receiver is *allowed*
> to use a specific DNSBL used in an exists:. Many DNSBL
> operators have their own policies wrt DNS query traffic.

Fair enough! An implementation should accept a list of DNSBLs that it
is allowed to query. As a special case, it can safely be allowed to
query the domain where the spf record itself is coming from, i.e.
given MFROM is someone@example.com it is possible for postmasters
there to publish, say, "include:%{l}.users.example.com". They do it at
their own risk. (Part of this consideration is independent of whether
the local part is involved in the query.)

Conversely, it seems quite unlikely that an external domain knows so
much about local parts at example.com that querying it makes sense.
The only case I can think of is organizations supplying such
particular service. Relevant organizations can then be included in the
fixed list of domains that can always be queried. (Or should queries
involving the local part require an additional authorization bit?)

That implies amending RFC4408.

>>> In practice see subject, less than 0.5 permille use
>>> *any* macros, if this statistics is correct.
>
>> I don't think that's correct. One should count the number
>> of times a record has been used. However, there's no
>> practical way to trace those calls: SPF lacks support for
>> statistics, as well as for debugging.
>
> You can certainly log and analyze DNS queries to your own
> name servers, and one of the proposed uses of SPF macros
> is to get precisely this logging affect. For an example
> see the policy of schlitt.net, also explained in RFC 4408.

I can log a query, but I won't know if it passed or failed.

> OTOH why should receivers interested in PASS or FAIL spend
> parts of their time and bandwidth to support your logging
> efforts ?

For the same reason they bother sending DSNs: cooperation.

> That has the decent charme of tracking cookies
> and "Web bugs"... :-(

Huh... what was that?



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: SPF-EAI (was: Re: 99.95% of all SPF records use no macros) [ In reply to ]
Alessandro Vesely wrote:

> the alphabet we use doesn't allow a third case beyond %{l}
> and %{L}. Does that call for additional macro transformers?

I think %{L} is good enough for the purpose of URIs in an
explanation. But now that you mentioned it, the I-D needs
a reference to RFC 3987 for this purpose. Percent-encoded
UTF-8, no rocket science.

Unless folks try to put %{L} into a host name for an http-
URL in an explanation. But that never worked as expected
depending on the local part, that is no new SPF-EAI issue.

> When spf-eai says "The local part MUST NOT be transformed"
> it implies that %{l} cannot be used for DNS lookups if
> internationalized local parts are present.

No, it implies "use the local part octets as is", because
that's also what happens outside of EAI in SPF.

For x+y@example and %{l}.test you get x+y.test. That is
no LDH host name as used in many protocols, but it is a
valid DNS name (FQDN).

For ÄÖÜ@example you get ÄÖÜ.test using the UTF-8 octets.

You don't get the IDNAbis punycode xn--whatever.test for
ÄÖÜ. EAI local parts are not limited to what's allowed
in IDNAbis, e.g., Ä+Ü is a valid EAI local part, but not
a valid IDNAbis U-label.

> I can log a query, but I won't know if it passed or failed.

Add the sending IP to your SPF exists: logging magic, then
you'd know what the result should be. With exist you can
dictate the result:

-exists:***.fail.your.domain.example
+exists:***.pass.your.domain.example

If *** gives you all info offered by macros, e.g. a unique
BATV local part, sending IP, etc., you can answer "make it
so" either for the fail- or the pass-exists. We are about
to reinvent SES here... :-)

>> That has the decent charme of tracking cookies and "Web
>> bugs"... :-(

> Huh... what was that?

Logging. A receiver wastes his resources to do stuff for
the sender unrelated to what the receiver really wants -
same idea as Web bugs (transparent 1*1 images) or third
party tracking cookies.

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Re: SPF-EAI [ In reply to ]
Frank Ellermann wrote:
> Alessandro Vesely wrote:
>> the alphabet we use doesn't allow a third case beyond %{l}
>> and %{L}. Does that call for additional macro transformers?
>
> I think %{L} is good enough for the purpose of URIs in an
> explanation.

Yes, it is. Since there is a script that receives such parameter, it
can always work out whatever transformation it needs to do its job.
> Unless folks try to put %{L} into a host name [...]

>> When spf-eai says "The local part MUST NOT be transformed"
>> it implies that %{l} cannot be used for DNS lookups if
>> internationalized local parts are present.
>
> No, it implies "use the local part octets as is", because
> that's also what happens outside of EAI in SPF.

Thus, if a site suitably restricts the local parts that users can
choose, then it will be able to use %{l} for actual DNS queries.
Otherwise local parts may not be used in, say, exist mechanisms -with
or without EAI. Therefore, it makes little sense to widen the set of
DNS-querable local parts by including U2A transformations. Is that
correct?

I wonder if, that being the case, the deeper analysis in your I-D
shouldn't be directed toward library implementors rather than policy
publishers.

>> I can log a query, but I won't know if it passed or failed.
>
> Add the sending IP to your SPF exists: logging magic, then
> you'd know what the result should be.

Hm... except for failures, policy restrictions, or possible
misinterpretations in the remote spf implementation.



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: SPF-EAI [ In reply to ]
Alessandro Vesely wrote:

>> I think %{L} is good enough for the purpose of URIs in an
>> explanation.

> Yes, it is. Since there is a script that receives such
> parameter, it can always work out whatever transformation
> it needs to do its job.

ACK, I add a section about explanations to the draft, with
a summary of what we discussed here some weeks ago wrt i18n
in SPF *outside* of EAI. Plus a RFC 3987 reference for the
small step to i18n explanations in conjunction with EAI.

>>> When spf-eai says "The local part MUST NOT be transformed"
>>> it implies that %{l} cannot be used for DNS lookups if
>>> internationalized local parts are present.

>> No, it implies "use the local part octets as is", because
>> that's also what happens outside of EAI in SPF.

> if a site suitably restricts the local parts that users can
> choose, then it will be able to use %{l} for actual DNS
> queries.

Sure, however the necessary restrictions are not obvious:

1a: %{l} can expand into a <dot-atom-text> (2822upd term),
that's one or more dot-separated <Atom>s. As long as
each <Atom> has length 1..63 SPF has no problem with it.

1u: For EAI it can also expand into dot-separated <uAtom>s,
with the same length restriction for SPF. However now
we are talking about octets: A single UTF-8 character
consists of 1..4 octets.

2a: Simplified DNS treats ASCII letters as case insensitive,
therefore you can have only one policy with an <Atom>
xyz. If the variants xyZ, xYz, Xyz, XYz, XyZ, xYZ, XYZ
are treated as different users (mailboxes) DNS and SPF
cannot handle this.

2u: DNS does not support case-insensitive <UTF8-non-ascii>:
äöü, äöÜ, ..., ÄÖÜ are eight different UTF-8 <uAtom>s
as far as DNS + SPF are concerned. There are actually
*far* more more variants of the äöü-strings if you can't
guarantee NFC (normalization form C). Sites supporting
EAI hopefully limit this äöü-zoo to one äöü-mailbox.

A piece of software doing this will affect UTF8SMTP and
final delivery, but it won't do anything for DNS + SPF
queries. DNS and UTF8SMTP can be different departments,
nobody is going to "canonicalize" incoming SPF queries.

3a: There are various issues if %{l} is a <Quoted-string>.
It begins with removing quotes and the backslashes of
quoted pairs, leading to an "embedded dot" erratum and
a bunch of missing test cases.

3u: EAI has no effect on 3a, neither better nor worse. BTW,
nobody supported to get rid of the moronic concept of a
quoted pair for <UTF8-non-ascii>. You still have about
seven weeks to appeal it as hopeless nonsense. Because
it doesn't affect SPF-EAI I won't appeal it, but the EAI
folks are aware what I think about this "feature"... :-(

4: I think 1a+1b+2a+2b+3a+3b also affect %{L} if used in a
<domain-spec>. Ditto %{s} and %{S} containing the local
part. Maybe that's another missing test case: For %{s}
in a <domain-spec> SPF implementations get an "@" within
a label of <target-name>, and have to treat this "as is".

> Otherwise local parts may not be used in, say, exist
> mechanisms -with or without EAI.

Note that it is no MUST NOT, publishers can try it, but they
are in trouble if they do. And spammers could intentionally
try say "do..ts" as local part, if they find receivers where
this helps them to avoid a FAIL.

> it makes little sense to widen the set of DNS-querable
> local parts by including U2A transformations. Is that
> correct?

U2A in IDNAbis means "U-label to A-label", and an A-label is
in essence "xn--" followed by a punycoded normalized U-label.
An A-label has the form of a traditional LDH-label.

We are already outside of LDH-labels in 1a + 2a + 3a, without
EAI. Therefore talking about "U2A" for 1u + 2u + 3u makes no
sense, it misses the point. And if that was your question we
agree, "U2A" is undefined for arbitrary <uAtom> labels.

In theory punycode doesn't have this limitation. It can take
any Unicode input and transform it into ASCII, but that's not
how punycode is used in IDNA(bis).

> the deeper analysis in your I-D shouldn't be directed toward
> library implementors rather than policy publishers.

The draft could be better organized, but I'm to lazy to change
it. A radical summary for implementors would be "get U2A magic
for the RHS, but you MUST NOT not touch the LHS, have fun".

For publishers the summary is "stay away from local part and
sender macros in <domain-spec>, or read and understand the
fine print".

For UTF8SMTP MSA operators the summary is "maybe transform the
RHS to A-labels, that should work 'as is' with SPF everywhere".

For Sender ID fans the summary is always the same "DON'T CHECK
PRA, IT DOES NOT WORK AS EXPECTED, ESPECIALLY NOT WITH v=spf1".

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Re: SPF-EAI [ In reply to ]
Frank Ellermann wrote:
> [...]
> [See http://www.openspf.org/Community/TempLoaclpartMacroWarn ]

Thank you, that looks clear enough. I copied it on the wiki page
above, in case someone finds a suitable place for that text.



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: SPF-EAI [ In reply to ]
Alessandro Vesely wrote on the DISCUSS list:

> [See http://www.openspf.org/Community/TempLoaclpartMacroWarn ]

For your Website write access request I can't tell who can
handle that - my password died together with my old box a
year ago. Julian can, but he wasn't around for some weeks,
maybe Alex, Scott, or Stuart can handle it.

"Thingy" (Julian's wiki software) does not cover its own
configuration files, I think - could be wrong, maybe this
depends on the installation - and needs a shell account for
such tasks.

Until we figure out how this can be arranged I could copy
your table to the i18n article, or to a subpage of this
article. Another idea is to *include* it there as is - I
forgot how that's done with "thingy", but it was straight
forward, and I'd know where to find a recipe.

Frank
--
I fear OE + GMaNe don't support Followup-To: Webmasters ;-)



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Re: SPF-EAI [ In reply to ]
On Wednesday 06 August 2008 10:41, Frank Ellermann wrote:
> Alessandro Vesely wrote on the DISCUSS list:
> > [See http://www.openspf.org/Community/TempLoaclpartMacroWarn ]
>
> For your Website write access request I can't tell who can
> handle that - my password died together with my old box a
> year ago. Julian can, but he wasn't around for some weeks,
> maybe Alex, Scott, or Stuart can handle it.
>
> "Thingy" (Julian's wiki software) does not cover its own
> configuration files, I think - could be wrong, maybe this
> depends on the installation - and needs a shell account for
> such tasks.
>
> Until we figure out how this can be arranged I could copy
> your table to the i18n article, or to a subpage of this
> article. Another idea is to *include* it there as is - I
> forgot how that's done with "thingy", but it was straight
> forward, and I'd know where to find a recipe.
>

Julian has been active on #spf IRC. I gave him a ping and he said he'd look
into it tonight.

Scott K


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: SPF-EAI [ In reply to ]
Scott Kitterman wrote:

> Julian has been active on #spf IRC. I gave him
> a ping and he said he'd look into it tonight.

Thanks Scott, the OpenSPF site needs editors with
fresh ideas.

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com