Mailing List Archive

Re: Upcoming new test-suite release
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> + e14.example.com:
> + - SPF: v=spf1 a:example..com
>
> There was already a test for this: invalid-domain-empty-label.

Oh, I didn't see that one. I didn't look for something like that in the
"Record evaluation" scenario. Maybe I should have.

> It currently allows for either ignoring the empty label, or permerror.
> If there is an official errata requiring nomatch instead of permerror,
> then simply change the result set of the existing test.

AFAICT we still have to decide on that one:

http://www.openspf.org/auth/RFC_4408/Errata#permerror-invalid-domains

What are the arguments for allowing a PermError result in this case?

> Or were you concerned about 2 adjacent dots vs 3?

No.

> + e5a.example.com:
> + - SPF: v=spf1 a:museum
>
> This seems to be not redundant. However, it seems unintuitive to me
> that example..com must be ignored, but museum gets a permerr.

Well, the first part ("example..com must be ignored") is still to be
decided, but the spec is unambiguous with regard to the second part.

> How about this case:
>
> Result: pass ?
>
> e5b.example.com:
> - SPF: v=spf1 a:museum.

What would be the point? "a:museum." is still a syntax error because the
trailing dot doesn't count. The grammar doesn't allow "museum." for a
<domain-spec>, just as it doesn't allow plain "museum".

> + e11.example.com:
> + - SPF: v=spf1 exists:%{i}.%{l2r-}.user.%{d2}
> + 1.2.3.4.gladstone.philip.user.example.com:
> + - A: 127.0.0.2
>
> Good, but the actual failing example in the field used %{l1r-}.
> Shouldn't we use that?

No, because %{l2r-} is more general. An implementation could by accident
_always_ use only one "part" in the "reverse" (r) or "non-dot-separator"
(-) case, even if a higher part limit (e.g. 2) was specified. (Yes, that
would be a really weird bug, but it doesn't hurt to test for it, does
it?) Plus, I added an excess "-test" part in the mailfrom's localpart so
as to test whether the part limit is actually honored.

Maybe these two aspects are already covered by other test cases in the
suite (are they??), but I wasn't certain of that and thought it wouldn't
hurt to have them covered again.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWFydwL7PKlBZWjsRAmAPAKDY9fEMp/OxbALluZyx3FbI2ty74gCfWfzn
90ch9427Kxcb+CDVpJSJ25I=
=q+4W
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73306579-9946d4
Powered by Listbox: http://www.listbox.com
Re: Re: Upcoming new test-suite release [ In reply to ]
On Thu, 6 Dec 2007, Julian Mehnle wrote:

> > e5b.example.com:
> > - SPF: v=spf1 a:museum.
>
> What would be the point? "a:museum." is still a syntax error because the
> trailing dot doesn't count. The grammar doesn't allow "museum." for a
> <domain-spec>, just as it doesn't allow plain "museum".

Since this isn't obvious, there should be a test case for both (with
and without trailing dot).

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73346692-7006dd
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Thu, 6 Dec 2007, Julian Mehnle wrote:
> > > e5b.example.com:
> > > - SPF: v=spf1 a:museum.
> >
> > What would be the point? "a:museum." is still a syntax error because
> > the trailing dot doesn't count. The grammar doesn't allow "museum."
> > for a <domain-spec>, just as it doesn't allow plain "museum".
>
> Since this isn't obvious, there should be a test case for both (with
> and without trailing dot).

But what are the chances of an implementation throwing PermError on
"a:museum" but not on "a:museum."?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWHEGwL7PKlBZWjsRAgM6AKDjAJtOfT12dEGtx1AUgKtBB0IWSgCg3tCD
igT8geUkz8rMqSTIK/DV7Tk=
=2znC
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73353009-530eb1
Powered by Listbox: http://www.listbox.com
Re: [spf-devel] Re: Upcoming new test-suite release [ In reply to ]
On Thu, 6 Dec 2007, Julian Mehnle wrote:

> > Since this isn't obvious, there should be a test case for both (with
> > and without trailing dot).
>
> But what are the chances of an implementation throwing PermError on
> "a:museum" but not on "a:museum."?

This is a test suite. Such considerations do not apply. Ideally, all
bizarre wrong behaviour should be flagged. In practice, we can't hit it all.
But deliberately throwing out a case we actually thought of, just
because we feel no one would make that mistake, is a silly mistake. The very
act of making *that* silly mistake will cause someone else to make the other
mistake, via a quantum mechanical application of Murphy's Law.

I'll go ahead and add the trailing dot case, if you are *sure* that both
should be permerror. The inconsistency is very ugly, but I suppose it
can't be helped. At least these are corner cases.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73357947-8f90b5
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Thu, 6 Dec 2007, Julian Mehnle wrote:
> > But what are the chances of an implementation throwing PermError on
> > "a:museum" but not on "a:museum."?
>
> This is a test suite. Such considerations do not apply.

Oh, I think they do. Maintainability and leanness do matter. They're
just not top priorities.

> Ideally, all bizarre wrong behaviour should be flagged. In practice, we
> can't hit it all. But deliberately throwing out a case we actually
> thought of, just because we feel no one would make that mistake, is a
> silly mistake. The very act of making *that* silly mistake will cause
> someone else to make the other mistake, via a quantum mechanical
> application of Murphy's Law.

I don't follow entirely ;-) ...

> I'll go ahead and add the trailing dot case, if you are *sure* that
> both should be permerror. The inconsistency is very ugly, but I
> suppose it can't be helped. At least these are corner cases.

... but I don't have a problem with adding it if others think it is
useful. Please do go ahead. I am 100% certain that both "museum" and
"museum." are NOT valid <domain-spec>s per RFC 4408.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWH0YwL7PKlBZWjsRAgx7AKCPXz4uZ7W4YxpopesKOc4fKmGNlgCg8orw
da3+otCH6ypDcEN5xj3QsSw=
=C6ip
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73380748-d271e7
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> + e14.example.com:
> + - SPF: v=spf1 a:example..com
>
> There was already a test for this: invalid-domain-empty-label.

The reason I missed the "invalid-domain-empty-label" test is that its
description isn't very expressive, and I hadn't read the comment.

What about removing my redundant test in favor of an improved description
for the existing "invalid-domain-empty-label" test? (See attached diff.)

The "invalid-domain-long" test's description has the same problem.
However, I did not want to edit that one before having discussed the
following: This test tests the "overlong label" case. But why is it
using macro expansion to that end? It could just as well say
"a:<64chars>.bar". That would allow to simplify the description even
further.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWZrEwL7PKlBZWjsRAtseAKD2sLWNVl8mqlEzGXAXAP1uRpeosQCeI3l3
uRF7Ej4giyV4MNZm6rW3yx0=
=5LZs
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73772822-ed5bfe
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Fri, 7 Dec 2007, Julian Mehnle wrote:
> > What about removing my redundant test in favor of an improved
> > description for the existing "invalid-domain-empty-label" test? (See
> > attached diff.)
>
> The "description" field is the "short" description. It is supposed to
> be a one line (or maybe several when quoting ABNF from the RFC)
> summary. The "comment" field is the multiline explanation.

I never understood it that way. Where do you get that from?
(<http://www.openspf.org/Test_Suite/Schema> doesn't say it must be a
one-liner. It merely says it should not be longer than one or two
sentences, which my attempt isn't.)

The problem we're getting into here is that we're testing things that
aren't clearly defined in the RFC, so we have to explain precisely what
the test is about. The description should be able to stand on its own
for someone who knows the RFC well, which isn't easy if the spec itself
is fuzzy on the test subject.

> You are welcome to make the description + comment better, but please
> keep the one-line summary + explanatory paragraph format.

OK, I'll give it a shot.

> > The "invalid-domain-long" test's description has the same problem.
> > However, I did not want to edit that one before having discussed the
> > following: This test tests the "overlong label" case. But why is it
> > using macro expansion to that end? It could just as well say
> > "a:<64chars>.bar". That would allow to simplify the description even
> > further.
>
> See toolonglabel. You need to test both before and after macro
> expansion. The "spec" field is very informative for why tests are
> there. The toolonglabel test is for 4.3/1. The invalid-domain-long
> test is for 8.1/2 and 5/10.

"toolonglabel" tests 4.3/1, whereas "invalid-domain-long" only tests 4.3/1
by analogy at best. 8.1/2 doesn't apply because "invalid-domain-long"
isn't about a regular syntax error (not even if we change the target-name
to "<64chars>.bar", which is a valid <domain-spec>). 5/10 doesn't apply
either, at least not 5/10/2 (the "TempError" case) -- perhaps 5/10/3
(the "domain does not exist" case), though.

I agree that in the absence of an explicit statement about malformed
labels, 4.3/1 is a valid analogy. However, can we please drop the
reference to 8.1/2, and change the one to 5/10 into 5/10/3?

Then, given that "<64chars>.bar" is a valid <domain-spec>, what's the
point in having the indirection via some macro, if what we want to test
is just the handling of malformed (but syntax-valid) domain names?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWfkpwL7PKlBZWjsRAgOGAKCK2iB6TFme+qfrLnLNwqC5Bv3iWACcDQSZ
RNXkuRIctb+rTNmgESebT40=
=G+59
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73901182-6b70c3
Powered by Listbox: http://www.listbox.com
Re: Re: Upcoming new test-suite release [ In reply to ]
On Sat, 8 Dec 2007, Julian Mehnle wrote:

> I never understood it that way. Where do you get that from?
> (<http://www.openspf.org/Test_Suite/Schema> doesn't say it must be a
> one-liner. It merely says it should not be longer than one or two
> sentences, which my attempt isn't.)

Literally, you are correct. In George MacDonalds "The Lost Princess",
there is a single sentence in the first paragraph that goes on for
the rest of the page. That would pass literal muster also.
If it will help to say "lines" instead of "sentences", I'll change
the informal Schema.

> Then, given that "<64chars>.bar" is a valid <domain-spec>, what's the
> point in having the indirection via some macro, if what we want to test
> is just the handling of malformed (but syntax-valid) domain names?

Because the behaviour may be different? It could easily be a different
code path in an implementation. You are very good with improving the
explanations, but you really don't seem to get the value of testing
for all the stupid things we can think of. It is a big help to someone
creating an implementation.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73906186-3b4b6d
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> Julian Mehnle wrote:
> > Then, given that "<64chars>.bar" is a valid <domain-spec>, what's the
> > point in having the indirection via some macro, if what we want to
> > test is just the handling of malformed (but syntax-valid) domain
> > names?
>
> Because the behaviour may be different? It could easily be a different
> code path in an implementation.

OK, but where is the "a:<64chars>.bar" case being tested then?
No, "toolonglabel" is not it.

The "a:<64chars>.bar" case seems way more fundamental than "a:%{macro-
that-expands-to-64+chars}.bar", yet you seem to think that testing only
the latter is sufficient.

I have thought long about how a PermError (let alone TempError) could be
justified for those cases, but I couldn't find a reasonable rationale, so
I took the liberty of narrowing it down to "no-match". Is there any
dissent? If not, then do we really need an erratum (currently noted as
<http://www.openspf.org/RFC_4408/Errata#permerror-invalid-domains>)
codifying the "no-match" behavior, or can we just drop the draft erratum
entry?

What about the attached diff? It hopefully resolves all open concerns.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWq4FwL7PKlBZWjsRAjiCAKDiuSD110wTsCaXe4JTmQ6vIPPZjwCdGhUG
cKDZJUJgE06LPFmBaiiv3jQ=
=tmb6
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=73936133-5235a4
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Julian Mehnle wrote:
> > The "a:<64chars>.bar" case seems way more fundamental than
> > "a:%{macro-that-expands-to-64+chars}.bar", yet you seem to think that
> > testing only the latter is sufficient.
>
> IMO test both, it's in the spirit of Stuart's proposal to cover all
> plausible code paths in a buggy implementation.

Fine, that's what my last proposed patch does.

> > I have thought long about how a PermError (let alone TempError) could
> > be justified for those cases, but I couldn't find a reasonable
> > rationale, so I took the liberty of narrowing it down to "no-match".
>
> Does that agree with your proposal of a new erratum wrt "Permerror after
> macro expansion" ?

Well, it doesn't agree, but I think 2.5.7/1/3 (the third sentence, which I
quoted in the message you are referring below) is actually a mistake.
After all, the entirety of RFC 4408 doesn't spell out the concept of
doing any syntax checking _after_ macro expansion, except for this one
sentence. Even if we think it would have been a good idea, it simply
isn't in the spec.

> > do we really need an erratum (currently noted as
> > <http://www.openspf.org/RFC_4408/Errata#permerror-invalid-domains>)
> > codifying the "no-match" behavior, or can we just drop the draft
> > erratum entry?
>
> That wannabe-erratum doesn't propose a fix, I guess what I meant is
> "whatever you do, this is no TempError", or "better let's say what it
> is" (no match or PermError).

Good. Can you think of a possible wording (and place in the RFC to put
it)?

> But you found something about a PermError in the spec.,
> http://permalink.gmane.org/gmane.mail.spam.spf.devel/1804
> about http://www.openspf.org/RFC_4408#op-result-permerror

See above.

> Have all syntatically valid identities expected formats ?
> "good..dots"@example is syntactically valid, but simple
> good..dots.example queries won't work for %{l}.example, while
> good\.\.dots.example should work.

I think that's exactly the case that 2.5.7/1/3 is referring to (even
though SPF doesn't know any concept of \-escaping, so your second example
would consist of four (4) labels, not two).

> OTOH bad..dots.example is a syntactically invalid HELO, does 2.5.7 mean
> that a:%{h} should throw a PermError ?

Yes, I think that's the intent. However, as I said, this is inconsistent
with the overall tune of the spec. I think it should be dropped for
consistency's sake. We can always revive the idea for SPFv3.

> We could say 2.5.7 got it wrong, and it should be "no match", not
> PermError. Or we could remove the wannabe-erratum keeping 2.5.7 and its
> obscure PermError as is.

I'd strongly prefer the former and would refrain from doing the latter.
What do others think?

> In both cases I think that interpreting dots within a quoted string
> (LHS) as "embedded dots" makes sense, it is almost the same as
> "back\\slash"@example or other odd creatures you might find in a local
> part for %{l}.

It might make sense, but the concept would be completely novel to RFC
4408, so I don't think we can do that.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHW+DBwL7PKlBZWjsRApBvAKCxx7RC0VfVLSCRpnVZuGKQ71jnPQCgm2F5
PBj/Re7WCtkEjNuKupiUtI4=
=Ztq4
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=74030782-467c1c
Powered by Listbox: http://www.listbox.com
Re: Re: Upcoming new test-suite release [ In reply to ]
Julian Mehnle wrote:
<snip>
>
>> OTOH bad..dots.example is a syntactically invalid HELO, does 2.5.7 mean
>> that a:%{h} should throw a PermError ?
>>
>
> Yes, I think that's the intent. However, as I said, this is inconsistent
> with the overall tune of the spec. I think it should be dropped for
> consistency's sake. We can always revive the idea for SPFv3.
>
>
>> We could say 2.5.7 got it wrong, and it should be "no match", not
>> PermError. Or we could remove the wannabe-erratum keeping 2.5.7 and its
>> obscure PermError as is.
>>
>
> I'd strongly prefer the former and would refrain from doing the latter.
> What do others think?
>
I agree


>
>> In both cases I think that interpreting dots within a quoted string
>> (LHS) as "embedded dots" makes sense, it is almost the same as
>> "back\\slash"@example or other odd creatures you might find in a local
>> part for %{l}.
>>
>
> It might make sense, but the concept would be completely novel to RFC
> 4408, so I don't think we can do that.
>
Exactly

Terry Fielder
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFHW+DBwL7PKlBZWjsRApBvAKCxx7RC0VfVLSCRpnVZuGKQ71jnPQCgm2F5
> PBj/Re7WCtkEjNuKupiUtI4=
> =Ztq4
> -----END PGP SIGNATURE-----
>
> -------------------------------------------
> Sender Policy Framework: http://www.openspf.org
> Archives: http://v2.listbox.com/member/archive/735/=now
> RSS Feed: http://v2.listbox.com/member/archive/rss/735/
> Modify Your Subscription: http://v2.listbox.com/member/?&
> Powered by Listbox: http://www.listbox.com
>
>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=74034019-9d3419
Powered by Listbox: http://www.listbox.com
Re: Re: Upcoming new test-suite release [ In reply to ]
On Sat, 8 Dec 2007, Julian Mehnle wrote:

> OK, but where is the "a:<64chars>.bar" case being tested then?
> No, "toolonglabel" is not it.

Yes, that tests long label in mailfrom. You are right, we need a
too long label test for A: (and maybe MX and include) as well.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=74102155-9e149d
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Sat, 8 Dec 2007, Julian Mehnle wrote:
> > OK, but where is the "a:<64chars>.bar" case being tested then?
> > No, "toolonglabel" is not it.
>
> Yes, that tests long label in mailfrom. You are right, we need a
> too long label test for A: (and maybe MX and include) as well.

OK, but I don't think we need separate tests for "mx:" and "include:".
It can be reasonably expected that an SPF implementation uses the same
<domain-spec> validation code for all of them.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXHoxwL7PKlBZWjsRArIEAKCmGz1aslt3IvAYpzCxE1L73cWWCgCaAtvb
bnT/XUpxsZeLdUbS5z1gQEI=
=GJsC
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=74119232-b6de96
Powered by Listbox: http://www.listbox.com
RE: Re: Upcoming new test-suite release [ In reply to ]
Does the test-suite test for extra TXT records? My domain has an extra TXT
record from a test I did years ago. I left it in, but it has caused some
SPF checkers to report invalid spf record errors. I reported such errors at
least twice in the past 2 years. But it still comes up as an issue from
time to time.

To see my 2 records:
dig watkins-home.com txt

watkins-home.com. 360000 IN TXT "line1" "line2" "line3"
"line4" "line5" "Line n"
watkins-home.com. 360000 IN TXT "v=spf1 mx "
"ip4:63.240.76.0/17 " "ip4:204.127.192.0/18 " "ip4:206.18.177.0/24 "
"ip4:216.148.227.0/24 "
"+exists:_h.%{h}._l.%{l}._o.%{o}._i.%{i}._spf.%{d}.spf-tracker.watkins-home.
com " "?all"

Thanks,
Guy

} -----Original Message-----
} From: Julian Mehnle [mailto:julian@mehnle.net]
} Sent: Sunday, December 09, 2007 6:29 PM
} To: SPF Discussion; SPF Development
} Subject: [spf-discuss] Re: Upcoming new test-suite release
}
} -----BEGIN PGP SIGNED MESSAGE-----
} Hash: SHA1
}
} Stuart D. Gathman wrote:
} > On Sat, 8 Dec 2007, Julian Mehnle wrote:
} > > OK, but where is the "a:<64chars>.bar" case being tested then?
} > > No, "toolonglabel" is not it.
} >
} > Yes, that tests long label in mailfrom. You are right, we need a
} > too long label test for A: (and maybe MX and include) as well.
}
} OK, but I don't think we need separate tests for "mx:" and "include:".
} It can be reasonably expected that an SPF implementation uses the same
} <domain-spec> validation code for all of them.
}
} -----BEGIN PGP SIGNATURE-----
} Version: GnuPG v1.4.6 (GNU/Linux)
}
} iD8DBQFHXHoxwL7PKlBZWjsRArIEAKCmGz1aslt3IvAYpzCxE1L73cWWCgCaAtvb
} bnT/XUpxsZeLdUbS5z1gQEI=
} =GJsC
} -----END PGP SIGNATURE-----
}
} -------------------------------------------
} Sender Policy Framework: http://www.openspf.org
} Archives: http://v2.listbox.com/member/archive/735/=now
} RSS Feed: http://v2.listbox.com/member/archive/rss/735/
} Modify Your Subscription:
} http://v2.listbox.com/member/?&
} Powered by Listbox: http://www.listbox.com

-------------------------------------------
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=74359632-ddde00
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guy wrote:
> Does the test-suite test for extra TXT records? My domain has an extra
> TXT record from a test I did years ago. I left it in, but it has
> caused some SPF checkers to report invalid spf record errors. I
> reported such errors at least twice in the past 2 years. But it still
> comes up as an issue from time to time.
>
> To see my 2 records:
> dig watkins-home.com txt
>
> watkins-home.com. 360000 IN TXT "line1" "line2" "line3" "line4" "line5" "Line n"
> watkins-home.com. 360000 IN TXT "v=spf1 mx " "ip4:63.240.76.0/17 " "ip4:204.127.192.0/18 " "ip4:206.18.177.0/24 " "ip4:216.148.227.0/24 " "+exists:_h.%{h}._l.%{l}._o.%{o}._i.%{i}._spf.%{d}.spf-tracker.watkins-home. com " "?all"

I am interpreting your question as follows:

Does the test suite test whether an SPF implementation correctly selects a
singular "v=spf1 ..." record in the presence of other, non-"v=spf1" TXT
records?

Answer:

Yes, it does, via the "nospace2" test case, assuming the test suite is
being interpreted correctly. (The test suite is merely a specification of
scenarios and expected results. For it to "work" correctly, it depends on
a test suite "driver" feeding the data to an SPF implementation and
checking the results returned.)

A correct test suite driver will find the SPF-type records in the
"nospace2" test's zonedata and synthesize identical TXT-type records from
them (see <http://www.openspf.org/Test_Suite/Schema>). The "v=spf10"
record (SPF-type or TXT-type) must then be treated as any random non-
"v=spf1" record (just like your "line1" ... "line n" record) by the
implementation and ignored in favor of the other record, which starts
with "v=spf1".

An implementation that croaks on a non-"v=spf1" record even though another
record exists that starts with "v=spf1", will fail the test suite,
provided the test suite driver works correctly.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXdQ9wL7PKlBZWjsRAhILAJ98rIq9GB2yTgZpcA1KJ3GUH2RrrwCgwCKO
zXc1J1oUOcmVqo/es/yu+ug=
=mXwV
-----END PGP SIGNATURE-----

-------------------------------------------
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=74435380-e1789f
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Julian Mehnle wrote:
> [ "a:<64chars>.bar" / "a:%{macro-that-expands-to-64+chars}.bar" ]
>
> I have thought long about how a PermError (let alone TempError) could
> be justified for those cases, but I couldn't find a reasonable
> rationale, so I took the liberty of narrowing it down to "no-match".
> Is there any dissent?
> [...]
> What about the attached diff? It hopefully resolves all open concerns.

No opposition to that patch was raised, so I now committed exactly that,
as r95 of the test suite:

http://www.openspf.org/source/project/test-suite/rfc4408-tests.yml?view=log
http://www.openspf.org/source/project/test-suite/rfc4408-tests.yml?r1=94&r2=95

Can we now make a new test suite release?

> If not, then do we really need an erratum (currently noted as
> <http://www.openspf.org/RFC_4408/Errata#permerror-invalid-domains>)
> codifying the "no-match" behavior, or can we just drop the draft
> erratum entry?

Any comments on that?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFH6ss/wL7PKlBZWjsRAkQqAKCt0wzlX9w9h9h3qVbw7hxLmFejswCg2gns
oGmCozTB7DJTcmQ/jZ90QTo=
=Uprt
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
Stuart D. Gathman wrote:

> I was for TempError myself. But no match is ok.

Okay, let's say TempError is out, depending on how we
read RFC 4408 that alone is not necessarily an erratum.

> There should probably be an errata to modify 2.5.7 if
> we go with no match.

ACK, "no match" is not what 2.5.7 proposes, if we want
"no match" it needs an erratum.

[2.5.7]
> This does not say specifically that 'example..com'
> should be a PermError, just that some unspecified
> condition might result in one.

The leading / trailing / adjacent dot cases as well as
macro-expanded label too long, or anything else that
makes it impossible to ask DNS for a second opinion,
are in an "unexpected format" (actually unsupported,
as far as DNS is concerned).

IMO it's not "some unspecified condition", it is an
"unexpected condition" from SPF's POV. The DNS STDs
specify that labels can't have zero length (excluding
the root), arguably that covers all dot-issues. The
DNS STDs also tell us various length limits, and IIRC
RFC 4408 cites them elsewhere.

ITYM "some unexpected conditions not specified in RFC
4408, but elsewhere" (RFC 1034/1035/1123, but I didn't
check the details).

> Since 'example..com' cannot actually be sent to a DNS
> server, it cannot result in an RCODE other than 0 or
> 3, and cannot result in a timeout, and hence cannot
> be a TempError.

+1, TempError is no sound option.

> Besides, in the non-macro case, the error is decidedly
> permanent, no temporary.

+1

> A PermError is wrong because in the macro case, the
> error is *not* permanent, but may depend on the sender.

Here we have a disconnect, the same MAIL FROM and HELO
combo from the same IP with the same policy will always,
permanently, reproducible fail. Of course depending on
the alleged sender, that's just the S in SPF.

If it is a legit sender the policy needs to be fixed to
work also for the given MAIL FROM and HELO. If it's a
weird policy where the problem depends on a legit IP it
also needs to be fixed.

If it is spam we are not sure what is the best course,
the next matching mechanism after "no match" can be -all,
+all or anything, nothing an RFC 4408 erratum could say
is guaranteed to do the right thing after "no match".

But for legit mail it is a broken policy, and I'd have
no problem with saying PermError.

> the spec is far from clear on the point, despite what
> Julian says.

If we'd say PermError the wannabe-erratum degenerates
into a clarification, to be inserted at the end of 2.5.7.

The more important question is what you want in the test
suite and by extension in conforming SPF implementations.

Not necessarily what implementations happen to do today
with this obscure corner case, but what they *should* do:

Is it really "okay" (= no match) if a legit mail results
in an invalid <target-name> after macro-expansion, or is
this an error of the domain owner, his policy not working
for legit local parts used in his domain (or similar) ?

Do I miss an important case "permanently" triggering this
condition (any legit case, not abuse), where a "no match"
handling could be intended or at least better ?

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Stuart D. Gathman wrote:
> > I was for TempError myself. But no match is ok.
>
> Okay, let's say TempError is out, depending on how we read RFC 4408 that
> alone is not necessarily an erratum.

Right. The issues of how RFC 4408 as-is is to be interpreted and whether
RFC 4408 should be amended via an erratum are generally separate.

> > There should probably be an errata to modify 2.5.7 if we go with no
> > match.
>
> ACK, "no match" is not what 2.5.7 proposes, if we want "no match" it
> needs an erratum.

<RFC authoring mode>
Again, I'm pretty sure that 2.5.7/1/3 was a mistake. Nowhere else
does RFC 4408 suggest an "after-macro-expansion syntax checking"
concept.
</RFC authoring mode>

<RFC interpretation mode>
The question however is: Given the present wording of RFC 4408,
should we accept implementations generating a PermError on "broken"
(i.e., valid syntax per RFC 4408 but invalid per RFCs 1034/1035/1123)
<target-name>s, as wrong as it may occur to us based on the RFC 4408
authoring background knowledge we posses?
</RFC interpretation mode>

<RFC authoring mode>
If we decide to tolerate PermError on broken <target-name>s, we cannot
simply remove 2.5.7/1/3 ...

| Be aware that if the domain owner uses macros (Section 8), it is
| possible that this result is due to the checked identities having an
| unexpected format.

... because we will then want implementors to be aware of the issue as
well! So we will either have to leave it as it is or amend it to say
that PermError is not an intended result for those cases, but that
there may be implementations working like that due to a legitimate
misinterpretation of earlier versions of the spec (including the
unamended RFC 4408).

If, however, we decide NOT to tolerate PermError, the best amendment
is probably to simply delete 2.5.7/1/3.
</RFC authoring mode>

I think we should make decisions in this order. The RFC 4408 test suite
depends solely on how we interpret RFC 4408 as it is now. Any RFC 4408
errata should then amend the RFC to clarify what it already says (albeit
contradictorily or otherwise ambiguously). In any case, RFC 4408 errata
are not supposed to modify the (originally intended) semantics of RFC
4408.

> > A PermError is wrong because in the macro case, the error is *not*
> > permanent, but may depend on the sender.
>
> Here we have a disconnect, the same MAIL FROM and HELO combo from the
> same IP with the same policy will always, permanently, reproducible
> fail.

Let's face it, PermError is far from clearly defined per RFC 4408. From
an entirely utilitarian point of view, all that can be said is that it
covers all error conditions that are not already covered by TempError,
whose semantics are a lot better defined in RFC 4408.

The real question then is: Does a "broken" <target-name> (either verba-
tim, e.g., "<more-than-63-chars>.com", or via macro expansion) need to be
_flagged_as_an_error_condition_ in the first place?

Had you asked me that question two years ago, I'd have said: definitely
yes! You may recall me having fought long and hard for making
SPF(NXDOMAIN) to be considered a case of PermError. It was all in vain;
the rough consensus was that checking for whether the authority domain
(whose policy is to be evaluated) actually exists was not SPF's problem
and that None should be returned. Likewise, consider 5/10/3 in section
5, "Mechanism Definitions":

| Several mechanisms rely on information fetched from DNS. [...] If the
| server returns "domain does not exist" (RCODE 3), then evaluation of the
| mechanism continues as if the server returned no error (RCODE 0) and
| zero answer records.

Now I think we ought to stick to that school of thought and treat "broken"
<target-name>s as no-match, too.

After all, given a policy using %{h}, why should "HELO whacky..spammer"
result in "PermError" when "HELO whacky.spammer" results in "Fail" or
"SoftFail"? (If anyone argues that "whacky..spammer" isn't even a valid
HELO per RFC 2821, consider %{l} and "MAIL FROM:<whacky..spammer@...>".)

As for the "mech:<more-than-63-chars>.com" case, I could be convinced that
this is a separate issue from the macro expansion one, however since it
clearly is NOT a syntax error per RFC 4408, it would still be hard to
justify a PermError result.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFH7SEXwL7PKlBZWjsRAmLiAJ0bri0uFkfAHMJYkiwo4djI5McfPQCeLqSM
/gC6CxQ1qZgeB1fNXVm4D7Y=
=YsYr
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/735/=now
RSS Feed: http://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yet another attempt:

http://www.openspf.org/source/project/test-suite/rfc4408-tests.yml?view=diff&r1=98&r2=89
http://www.openspf.org/source/project/test-suite/rfc4408-tests.CHANGES?rev=98&view=auto

This should incorporate the resolutions of all the errata that were under
discussion since the 2007.05 test-suite release.

Any objections to making that a 2008.08 release?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkimIHoACgkQwL7PKlBZWju59ACgjyx423Mk6wH0C6RKn78IXuls
gIUAnRqLk+8JBMk8/QxN9FsTzKop7FAc
=fBIR
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com
Re: Upcoming new test-suite release [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Julian Mehnle wrote:
> > Any objections to making that a 2008.08 release?
>
> Looking only in the change log I see no problem.

Good, thanks for lending your eyeballs. Release impending.

> Copied from my "last erratum" reply, because the end was wrong; please
> add the "macro mania" tests:

I added them, but rolled them up into a single test case ("macro-mania-
in-domain"). No need to be extra redundant.

Next stop: 2008.08 release announcement.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkioGeMACgkQwL7PKlBZWjtexgCg45tTST8dgzsdtFV4IJ56ZHxm
xeQAnR1TkiIRpooqhlzOZMSqfFQPi62F
=qhZP
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com