Mailing List Archive

On Thu, 6 Dec 2007, Julian Mehnle wrote:

> > e5b.example.com:
> > - SPF: v=spf1 a:museum.
>
> What would be the point? "a:museum." is still a syntax error because the
> trailing dot doesn't count. The grammar doesn't allow "museum." for a
> <domain-spec>, just as it doesn't allow plain "museum".

Since this isn't obvious, there should be a test case for both (with
and without trailing dot).

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73346155-418336
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Thu, 6 Dec 2007, Julian Mehnle wrote:
> > > e5b.example.com:
> > > - SPF: v=spf1 a:museum.
> >
> > What would be the point? "a:museum." is still a syntax error because
> > the trailing dot doesn't count. The grammar doesn't allow "museum."
> > for a <domain-spec>, just as it doesn't allow plain "museum".
>
> Since this isn't obvious, there should be a test case for both (with
> and without trailing dot).

But what are the chances of an implementation throwing PermError on
"a:museum" but not on "a:museum."?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWHEGwL7PKlBZWjsRAgM6AKDjAJtOfT12dEGtx1AUgKtBB0IWSgCg3tCD
igT8geUkz8rMqSTIK/DV7Tk=
=2znC
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73352186-7170f5
Powered by Listbox: http://www.listbox.com

On Thu, 6 Dec 2007, Julian Mehnle wrote:

> > Since this isn't obvious, there should be a test case for both (with
> > and without trailing dot).
>
> But what are the chances of an implementation throwing PermError on
> "a:museum" but not on "a:museum."?

This is a test suite. Such considerations do not apply. Ideally, all
bizarre wrong behaviour should be flagged. In practice, we can't hit it all.
But deliberately throwing out a case we actually thought of, just
because we feel no one would make that mistake, is a silly mistake. The very
act of making *that* silly mistake will cause someone else to make the other
mistake, via a quantum mechanical application of Murphy's Law.

I'll go ahead and add the trailing dot case, if you are *sure* that both
should be permerror. The inconsistency is very ugly, but I suppose it
can't be helped. At least these are corner cases.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73357762-f00a92
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Thu, 6 Dec 2007, Julian Mehnle wrote:
> > But what are the chances of an implementation throwing PermError on
> > "a:museum" but not on "a:museum."?
>
> This is a test suite. Such considerations do not apply.

Oh, I think they do. Maintainability and leanness do matter. They're
just not top priorities.

> Ideally, all bizarre wrong behaviour should be flagged. In practice, we
> can't hit it all. But deliberately throwing out a case we actually
> thought of, just because we feel no one would make that mistake, is a
> silly mistake. The very act of making *that* silly mistake will cause
> someone else to make the other mistake, via a quantum mechanical
> application of Murphy's Law.

I don't follow entirely ;-) ...

> I'll go ahead and add the trailing dot case, if you are *sure* that
> both should be permerror. The inconsistency is very ugly, but I
> suppose it can't be helped. At least these are corner cases.

... but I don't have a problem with adding it if others think it is
useful. Please do go ahead. I am 100% certain that both "museum" and
"museum." are NOT valid <domain-spec>s per RFC 4408.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWH0YwL7PKlBZWjsRAgx7AKCPXz4uZ7W4YxpopesKOc4fKmGNlgCg8orw
da3+otCH6ypDcEN5xj3QsSw=
=C6ip
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73380426-921a7c
Powered by Listbox: http://www.listbox.com

Julian Mehnle wrote:

> what are the chances of an implementation throwing PermError on
> "a:museum" but not on "a:museum."?

John tried also this solution for some 2821bis drafts. It didn't
fly in the end, just checking both PERMERRORs, as Stuart proposed
it, catches this plausible "one-dot-anywhere" approach.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73758371-5494ac
Powered by Listbox: http://www.listbox.com

Stuart D. Gathman wrote:
> + e14.example.com:
> + - SPF: v=spf1 a:example..com
>
> There was already a test for this: invalid-domain-empty-label.

The reason I missed the "invalid-domain-empty-label" test is that its
description isn't very expressive, and I hadn't read the comment.

What about removing my redundant test in favor of an improved description
for the existing "invalid-domain-empty-label" test? (See attached diff.)

The "invalid-domain-long" test's description has the same problem.
However, I did not want to edit that one before having discussed the
following: This test tests the "overlong label" case. But why is it
using macro expansion to that end? It could just as well say
"a:<64chars>.bar". That would allow to simplify the description even
further.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73768073-756678
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> + e14.example.com:
> + - SPF: v=spf1 a:example..com
>
> There was already a test for this: invalid-domain-empty-label.

The reason I missed the "invalid-domain-empty-label" test is that its
description isn't very expressive, and I hadn't read the comment.

What about removing my redundant test in favor of an improved description
for the existing "invalid-domain-empty-label" test? (See attached diff.)

The "invalid-domain-long" test's description has the same problem.
However, I did not want to edit that one before having discussed the
following: This test tests the "overlong label" case. But why is it
using macro expansion to that end? It could just as well say
"a:<64chars>.bar". That would allow to simplify the description even
further.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWZrEwL7PKlBZWjsRAtseAKD2sLWNVl8mqlEzGXAXAP1uRpeosQCeI3l3
uRF7Ej4giyV4MNZm6rW3yx0=
=5LZs
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73777393-4241c9
Powered by Listbox: http://www.listbox.com

On Fri, 7 Dec 2007, Julian Mehnle wrote:

> What about removing my redundant test in favor of an improved description
> for the existing "invalid-domain-empty-label" test? (See attached diff.)

The "description" field is the "short" description. It is supposed to be a one
line (or maybe several when quoting ABNF from the RFC) summary. The
"comment" field is the multiline explanation. You are welcome
to make the description + comment better, but please keep the
one-line summary + explanatory paragraph format.

> The "invalid-domain-long" test's description has the same problem.
> However, I did not want to edit that one before having discussed the
> following: This test tests the "overlong label" case. But why is it
> using macro expansion to that end? It could just as well say
> "a:<64chars>.bar". That would allow to simplify the description even
> further.

See toolonglabel. You need to test both before and after macro expansion.
The "spec" field is very informative for why tests are there. The
toolonglabel test is for 4.3/1. The invalid-domain-long test is for
8.1/2 and 5/10.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73869703-ee7ee3
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Fri, 7 Dec 2007, Julian Mehnle wrote:
> > What about removing my redundant test in favor of an improved
> > description for the existing "invalid-domain-empty-label" test? (See
> > attached diff.)
>
> The "description" field is the "short" description. It is supposed to
> be a one line (or maybe several when quoting ABNF from the RFC)
> summary. The "comment" field is the multiline explanation.

I never understood it that way. Where do you get that from?
(<http://www.openspf.org/Test_Suite/Schema> doesn't say it must be a
one-liner. It merely says it should not be longer than one or two
sentences, which my attempt isn't.)

The problem we're getting into here is that we're testing things that
aren't clearly defined in the RFC, so we have to explain precisely what
the test is about. The description should be able to stand on its own
for someone who knows the RFC well, which isn't easy if the spec itself
is fuzzy on the test subject.

> You are welcome to make the description + comment better, but please
> keep the one-line summary + explanatory paragraph format.

OK, I'll give it a shot.

> > The "invalid-domain-long" test's description has the same problem.
> > However, I did not want to edit that one before having discussed the
> > following: This test tests the "overlong label" case. But why is it
> > using macro expansion to that end? It could just as well say
> > "a:<64chars>.bar". That would allow to simplify the description even
> > further.
>
> See toolonglabel. You need to test both before and after macro
> expansion. The "spec" field is very informative for why tests are
> there. The toolonglabel test is for 4.3/1. The invalid-domain-long
> test is for 8.1/2 and 5/10.

"toolonglabel" tests 4.3/1, whereas "invalid-domain-long" only tests 4.3/1
by analogy at best. 8.1/2 doesn't apply because "invalid-domain-long"
isn't about a regular syntax error (not even if we change the target-name
to "<64chars>.bar", which is a valid <domain-spec>). 5/10 doesn't apply
either, at least not 5/10/2 (the "TempError" case) -- perhaps 5/10/3
(the "domain does not exist" case), though.

I agree that in the absence of an explicit statement about malformed
labels, 4.3/1 is a valid analogy. However, can we please drop the
reference to 8.1/2, and change the one to 5/10 into 5/10/3?

Then, given that "<64chars>.bar" is a valid <domain-spec>, what's the
point in having the indirection via some macro, if what we want to test
is just the handling of malformed (but syntax-valid) domain names?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWfkpwL7PKlBZWjsRAgOGAKCK2iB6TFme+qfrLnLNwqC5Bv3iWACcDQSZ
RNXkuRIctb+rTNmgESebT40=
=G+59
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73900887-b15176
Powered by Listbox: http://www.listbox.com

On Sat, 8 Dec 2007, Julian Mehnle wrote:

> I never understood it that way. Where do you get that from?
> (<http://www.openspf.org/Test_Suite/Schema> doesn't say it must be a
> one-liner. It merely says it should not be longer than one or two
> sentences, which my attempt isn't.)

Literally, you are correct. In George MacDonalds "The Lost Princess",
there is a single sentence in the first paragraph that goes on for
the rest of the page. That would pass literal muster also.
If it will help to say "lines" instead of "sentences", I'll change
the informal Schema.

> Then, given that "<64chars>.bar" is a valid <domain-spec>, what's the
> point in having the indirection via some macro, if what we want to test
> is just the handling of malformed (but syntax-valid) domain names?

Because the behaviour may be different? It could easily be a different
code path in an implementation. You are very good with improving the
explanations, but you really don't seem to get the value of testing
for all the stupid things we can think of. It is a big help to someone
creating an implementation.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73906032-4ef412
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> Julian Mehnle wrote:
> > Then, given that "<64chars>.bar" is a valid <domain-spec>, what's the
> > point in having the indirection via some macro, if what we want to
> > test is just the handling of malformed (but syntax-valid) domain
> > names?
>
> Because the behaviour may be different? It could easily be a different
> code path in an implementation.

OK, but where is the "a:<64chars>.bar" case being tested then?
No, "toolonglabel" is not it.

The "a:<64chars>.bar" case seems way more fundamental than "a:%{macro-
that-expands-to-64+chars}.bar", yet you seem to think that testing only
the latter is sufficient.

I have thought long about how a PermError (let alone TempError) could be
justified for those cases, but I couldn't find a reasonable rationale, so
I took the liberty of narrowing it down to "no-match". Is there any
dissent? If not, then do we really need an erratum (currently noted as
<http://www.openspf.org/RFC_4408/Errata#permerror-invalid-domains>)
codifying the "no-match" behavior, or can we just drop the draft erratum
entry?

What about the attached diff? It hopefully resolves all open concerns.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWq4FwL7PKlBZWjsRAjiCAKDiuSD110wTsCaXe4JTmQ6vIPPZjwCdGhUG
cKDZJUJgE06LPFmBaiiv3jQ=
=tmb6
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73935672-dbba29
Powered by Listbox: http://www.listbox.com

Mail kutumuzun yogunluguna gore size yanit verme surecimiz degisebilir. Gec kalinmis cevaplar icin bizi madur gorun.!

Sercan TAPSIN
GSM:
05447727177
05358583410
05558295252
ICQ : 706886
MSN : sercantapsin@hotmail.com
Yahoo : sercantapsin@yahoo.com
Google : sercantapsin@gmail.com
Mail : tapsin@hostunuz.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73936469-d28827
Powered by Listbox: http://www.listbox.com

Mail kutumuzun yogunluguna gore size yanit verme surecimiz degisebilir. Gec kalinmis cevaplar icin bizi madur gorun.!

Sercan TAPSIN
GSM:
05447727177
05358583410
05558295252
ICQ : 706886
MSN : sercantapsin@hotmail.com
Yahoo : sercantapsin@yahoo.com
Google : sercantapsin@gmail.com
Mail : tapsin@hostunuz.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73937146-266597
Powered by Listbox: http://www.listbox.com

Mail kutumuzun yogunluguna gore size yanit verme surecimiz degisebilir. Gec kalinmis cevaplar icin bizi madur gorun.!

Sercan TAPSIN
GSM:
05447727177
05358583410
05558295252
ICQ : 706886
MSN : sercantapsin@hotmail.com
Yahoo : sercantapsin@yahoo.com
Google : sercantapsin@gmail.com
Mail : tapsin@hostunuz.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73937461-7558f9
Powered by Listbox: http://www.listbox.com

Mail kutumuzun yogunluguna gore size yanit verme surecimiz degisebilir. Gec kalinmis cevaplar icin bizi madur gorun.!

Sercan TAPSIN
GSM:
05447727177
05358583410
05558295252
ICQ : 706886
MSN : sercantapsin@hotmail.com
Yahoo : sercantapsin@yahoo.com
Google : sercantapsin@gmail.com
Mail : tapsin@hostunuz.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73937819-7c98d2
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

tapsin@hostunuz.com wrote:
> Mail kutumuzun yogunluguna gore size yanit verme surecimiz degisebilir.
> Gec kalinmis cevaplar icin bizi madur gorun.!
>
> Sercan TAPSIN

*sigh*

I unsubscribed the guy.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWt1DwL7PKlBZWjsRAjG9AJ99ZuJJesizmbKRsWD0xMV7QCyGXgCg8hWz
GmYXiowyKf1tYIcNfFQmSHk=
=YWzH
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=73959487-4b9a3f
Powered by Listbox: http://www.listbox.com

Julian Mehnle wrote:

> The "a:<64chars>.bar" case seems way more fundamental than
> "a:%{macro-that-expands-to-64+chars}.bar", yet you seem to
> think that testing only the latter is sufficient.

IMO test both, it's in the spirit of Stuart's proposal to
cover all plausible code paths in a buggy implementation.

> I have thought long about how a PermError (let alone
> TempError) could be justified for those cases, but I
> couldn't find a reasonable rationale, so I took the
> liberty of narrowing it down to "no-match".

Does that agree with your proposal of a new erratum wrt
"Permerror after macro expansion" ? I haven't checked
this yet (later: now I did, see below).

> do we really need an erratum (currently noted as
> <http://www.openspf.org/RFC_4408/Errata#permerror-invalid-domains>)
> codifying the "no-match" behavior, or can we just drop
> the draft erratum entry?

That wannabe-erratum doesn't propose a fix, I guess what
I meant is "whatever you do, this is no TempError", or
"better let's say what it is" (no match or PermError).

But you found something about a PermError in the spec.,
http://permalink.gmane.org/gmane.mail.spam.spf.devel/1804
about http://www.openspf.org/RFC_4408#op-result-permerror
(= http://tools.ietf.org/html/rfc4408#section-2.5.7 )

What's a "checked identity having an unexpected format",
and where's Wayne when I want to kick his [guess]... ;-)

Have all syntatically valid identities expected formats ?
"good..dots"@example is syntactically valid, but simple
good..dots.example queries won't work for %{l}.example,
while good\.\.dots.example should work.

OTOH bad..dots.example is a syntactically invalid HELO,
does 2.5.7 mean that a:%{h} should throw a PermError ?

Likewise bad..dots@example is syntactically invalid,
implementations MUST NOT try to fix it for a possible
bad\.\.dots.example query.

We could say 2.5.7 got it wrong, and it should be "no
match", not PermError. Or we could remove the wannabe-
erratum keeping 2.5.7 and its obscure PermError as is.

In both cases I think that interpreting dots within a
quoted string (LHS) as "embedded dots" makes sense, it
is almost the same as "back\\slash"@example or other
odd creatures you might find in a local part for %{l}.

No erratum for the latter, but some ugly new test cases
where it would be a surprise if SPF implementations got
it right... :-( 2822upd will be somewhat simpler, no
more NO-WS-CTL. But still quoted dots and backslashes.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=74004467-3c1b06
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Julian Mehnle wrote:
> > The "a:<64chars>.bar" case seems way more fundamental than
> > "a:%{macro-that-expands-to-64+chars}.bar", yet you seem to think that
> > testing only the latter is sufficient.
>
> IMO test both, it's in the spirit of Stuart's proposal to cover all
> plausible code paths in a buggy implementation.

Fine, that's what my last proposed patch does.

> > I have thought long about how a PermError (let alone TempError) could
> > be justified for those cases, but I couldn't find a reasonable
> > rationale, so I took the liberty of narrowing it down to "no-match".
>
> Does that agree with your proposal of a new erratum wrt "Permerror after
> macro expansion" ?

Well, it doesn't agree, but I think 2.5.7/1/3 (the third sentence, which I
quoted in the message you are referring below) is actually a mistake.
After all, the entirety of RFC 4408 doesn't spell out the concept of
doing any syntax checking _after_ macro expansion, except for this one
sentence. Even if we think it would have been a good idea, it simply
isn't in the spec.

> > do we really need an erratum (currently noted as
> > <http://www.openspf.org/RFC_4408/Errata#permerror-invalid-domains>)
> > codifying the "no-match" behavior, or can we just drop the draft
> > erratum entry?
>
> That wannabe-erratum doesn't propose a fix, I guess what I meant is
> "whatever you do, this is no TempError", or "better let's say what it
> is" (no match or PermError).

Good. Can you think of a possible wording (and place in the RFC to put
it)?

> But you found something about a PermError in the spec.,
> http://permalink.gmane.org/gmane.mail.spam.spf.devel/1804
> about http://www.openspf.org/RFC_4408#op-result-permerror

See above.

> Have all syntatically valid identities expected formats ?
> "good..dots"@example is syntactically valid, but simple
> good..dots.example queries won't work for %{l}.example, while
> good\.\.dots.example should work.

I think that's exactly the case that 2.5.7/1/3 is referring to (even
though SPF doesn't know any concept of \-escaping, so your second example
would consist of four (4) labels, not two).

> OTOH bad..dots.example is a syntactically invalid HELO, does 2.5.7 mean
> that a:%{h} should throw a PermError ?

Yes, I think that's the intent. However, as I said, this is inconsistent
with the overall tune of the spec. I think it should be dropped for
consistency's sake. We can always revive the idea for SPFv3.

> We could say 2.5.7 got it wrong, and it should be "no match", not
> PermError. Or we could remove the wannabe-erratum keeping 2.5.7 and its
> obscure PermError as is.

I'd strongly prefer the former and would refrain from doing the latter.
What do others think?

> In both cases I think that interpreting dots within a quoted string
> (LHS) as "embedded dots" makes sense, it is almost the same as
> "back\\slash"@example or other odd creatures you might find in a local
> part for %{l}.

It might make sense, but the concept would be completely novel to RFC
4408, so I don't think we can do that.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHW+DBwL7PKlBZWjsRApBvAKCxx7RC0VfVLSCRpnVZuGKQ71jnPQCgm2F5
PBj/Re7WCtkEjNuKupiUtI4=
=Ztq4
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=74030371-267a80
Powered by Listbox: http://www.listbox.com

On Sat, 8 Dec 2007, Julian Mehnle wrote:

> OK, but where is the "a:<64chars>.bar" case being tested then?
> No, "toolonglabel" is not it.

Yes, that tests long label in mailfrom. You are right, we need a
too long label test for A: (and maybe MX and include) as well.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=74100955-7cc353
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Sat, 8 Dec 2007, Julian Mehnle wrote:
> > OK, but where is the "a:<64chars>.bar" case being tested then?
> > No, "toolonglabel" is not it.
>
> Yes, that tests long label in mailfrom. You are right, we need a
> too long label test for A: (and maybe MX and include) as well.

OK, but I don't think we need separate tests for "mx:" and "include:".
It can be reasonably expected that an SPF implementation uses the same
<domain-spec> validation code for all of them.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXHm8wL7PKlBZWjsRAkKZAJ9+uo1hpgqQgyZIkdqhSNzpLGHWSgCg6Zba
TP1wBgHUkJcTWPkCdUJdyog=
=C2wG
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=74118508-264e34
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Sat, 8 Dec 2007, Julian Mehnle wrote:
> > OK, but where is the "a:<64chars>.bar" case being tested then?
> > No, "toolonglabel" is not it.
>
> Yes, that tests long label in mailfrom. You are right, we need a
> too long label test for A: (and maybe MX and include) as well.

OK, but I don't think we need separate tests for "mx:" and "include:".
It can be reasonably expected that an SPF implementation uses the same
<domain-spec> validation code for all of them.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXHoxwL7PKlBZWjsRArIEAKCmGz1aslt3IvAYpzCxE1L73cWWCgCaAtvb
bnT/XUpxsZeLdUbS5z1gQEI=
=GJsC
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=74118938-005d13
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guy wrote:
> Does the test-suite test for extra TXT records? My domain has an extra
> TXT record from a test I did years ago. I left it in, but it has
> caused some SPF checkers to report invalid spf record errors. I
> reported such errors at least twice in the past 2 years. But it still
> comes up as an issue from time to time.
>
> To see my 2 records:
> dig watkins-home.com txt
>
> watkins-home.com. 360000 IN TXT "line1" "line2" "line3" "line4" "line5" "Line n"
> watkins-home.com. 360000 IN TXT "v=spf1 mx " "ip4:63.240.76.0/17 " "ip4:204.127.192.0/18 " "ip4:206.18.177.0/24 " "ip4:216.148.227.0/24 " "+exists:_h.%{h}._l.%{l}._o.%{o}._i.%{i}._spf.%{d}.spf-tracker.watkins-home. com " "?all"

I am interpreting your question as follows:

Does the test suite test whether an SPF implementation correctly selects a
singular "v=spf1 ..." record in the presence of other, non-"v=spf1" TXT
records?

Answer:

Yes, it does, via the "nospace2" test case, assuming the test suite is
being interpreted correctly. (The test suite is merely a specification of
scenarios and expected results. For it to "work" correctly, it depends on
a test suite "driver" feeding the data to an SPF implementation and
checking the results returned.)

A correct test suite driver will find the SPF-type records in the
"nospace2" test's zonedata and synthesize identical TXT-type records from
them (see <http://www.openspf.org/Test_Suite/Schema>). The "v=spf10"
record (SPF-type or TXT-type) must then be treated as any random non-
"v=spf1" record (just like your "line1" ... "line n" record) by the
implementation and ignored in favor of the other record, which starts
with "v=spf1".

An implementation that croaks on a non-"v=spf1" record even though another
record exists that starts with "v=spf1", will fail the test suite,
provided the test suite driver works correctly.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXdQ9wL7PKlBZWjsRAhILAJ98rIq9GB2yTgZpcA1KJ3GUH2RrrwCgwCKO
zXc1J1oUOcmVqo/es/yu+ug=
=mXwV
-----END PGP SIGNATURE-----

-------------------------------------------
Archives: http://v2.listbox.com/member/archive/1007/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/1007/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311533&id_secret=74434754-f5d56d
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Julian Mehnle wrote:
> [ "a:<64chars>.bar" / "a:%{macro-that-expands-to-64+chars}.bar" ]
>
> I have thought long about how a PermError (let alone TempError) could
> be justified for those cases, but I couldn't find a reasonable
> rationale, so I took the liberty of narrowing it down to "no-match".
> Is there any dissent?
> [...]
> What about the attached diff? It hopefully resolves all open concerns.

No opposition to that patch was raised, so I now committed exactly that,
as r95 of the test suite:

http://www.openspf.org/source/project/test-suite/rfc4408-tests.yml?view=log
http://www.openspf.org/source/project/test-suite/rfc4408-tests.yml?r1=94&r2=95

Can we now make a new test suite release?

> If not, then do we really need an erratum (currently noted as
> <http://www.openspf.org/RFC_4408/Errata#permerror-invalid-domains>)
> codifying the "no-match" behavior, or can we just drop the draft
> erratum entry?

Any comments on that?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFH6ss/wL7PKlBZWjsRAkQqAKCt0wzlX9w9h9h3qVbw7hxLmFejswCg2gns
oGmCozTB7DJTcmQ/jZ90QTo=
=Uprt
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle wrote:

> Any comments on that?

Yeah, in
<http://article.gmane.org/gmane.mail.spam.spf.devel/1831>
<http://article.gmane.org/gmane.mail.spam.spf.discuss/23632>
<http://article.gmane.org/gmane.mail.spam.spf.discuss/24048>
<http://article.gmane.org/gmane.mail.spam.spf.discuss/24052>

Inconclusive, two for "no match", two for "PermError", the
latter based on 2.5.7.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Thu, 27 Mar 2008, Frank Ellermann wrote:

> <http://article.gmane.org/gmane.mail.spam.spf.devel/1831>
> <http://article.gmane.org/gmane.mail.spam.spf.discuss/23632>
> <http://article.gmane.org/gmane.mail.spam.spf.discuss/24048>
> <http://article.gmane.org/gmane.mail.spam.spf.discuss/24052>
>
> Inconclusive, two for "no match", two for "PermError", the
> latter based on 2.5.7.

I was for TempError myself. But no match is ok. The test suite
allowed no match or TempError until the latest change. It never
allowed PermError. I guess we all though it was just "so wrong",
at least for the macro expansion case. There should probably be
an errata to modify 2.5.7 if we go with no match.

For reference, 2.5.7/3 says:
"Be aware that if the domain owner uses macros (Section 8), it is possible that
this result is due to the checked identities having an unexpected format."

This does not say specifically that 'example..com' should be a PermError,
just that some unspecified condition might result in one.

Section 5 says:
"For these DNS queries, except where noted, if the DNS server returns an error
(RCODE other than 0 or 3) or the query times out, the mechanism throws the
exception "TempError". If the server returns "domain does not exist" (RCODE 3),
then evaluation of the mechanism continues as if the server returned no error
(RCODE 0) and zero answer records."

Since 'example..com' cannot actually be sent to a DNS server, it cannot
result in an RCODE other than 0 or 3, and cannot result in a timeout,
and hence cannot be a TempError. Besides, in the non-macro case, the error
is decidedly permanent, no temporary. That leaves 'no match' or PermError.

A PermError is wrong because in the macro case, the error is *not*
permanent, but may depend on the sender.

So 'no match' is the only consistent result. But the spec is far from
clear on the point, despite what Julian says.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Stuart D. Gathman wrote:
> > I was for TempError myself. But no match is ok.
>
> Okay, let's say TempError is out, depending on how we read RFC 4408 that
> alone is not necessarily an erratum.

Right. The issues of how RFC 4408 as-is is to be interpreted and whether
RFC 4408 should be amended via an erratum are generally separate.

> > There should probably be an errata to modify 2.5.7 if we go with no
> > match.
>
> ACK, "no match" is not what 2.5.7 proposes, if we want "no match" it
> needs an erratum.

<RFC authoring mode>
Again, I'm pretty sure that 2.5.7/1/3 was a mistake. Nowhere else
does RFC 4408 suggest an "after-macro-expansion syntax checking"
concept.
</RFC authoring mode>

<RFC interpretation mode>
The question however is: Given the present wording of RFC 4408,
should we accept implementations generating a PermError on "broken"
(i.e., valid syntax per RFC 4408 but invalid per RFCs 1034/1035/1123)
<target-name>s, as wrong as it may occur to us based on the RFC 4408
authoring background knowledge we posses?
</RFC interpretation mode>

<RFC authoring mode>
If we decide to tolerate PermError on broken <target-name>s, we cannot
simply remove 2.5.7/1/3 ...

| Be aware that if the domain owner uses macros (Section 8), it is
| possible that this result is due to the checked identities having an
| unexpected format.

... because we will then want implementors to be aware of the issue as
well! So we will either have to leave it as it is or amend it to say
that PermError is not an intended result for those cases, but that
there may be implementations working like that due to a legitimate
misinterpretation of earlier versions of the spec (including the
unamended RFC 4408).

If, however, we decide NOT to tolerate PermError, the best amendment
is probably to simply delete 2.5.7/1/3.
</RFC authoring mode>

I think we should make decisions in this order. The RFC 4408 test suite
depends solely on how we interpret RFC 4408 as it is now. Any RFC 4408
errata should then amend the RFC to clarify what it already says (albeit
contradictorily or otherwise ambiguously). In any case, RFC 4408 errata
are not supposed to modify the (originally intended) semantics of RFC
4408.

> > A PermError is wrong because in the macro case, the error is *not*
> > permanent, but may depend on the sender.
>
> Here we have a disconnect, the same MAIL FROM and HELO combo from the
> same IP with the same policy will always, permanently, reproducible
> fail.

Let's face it, PermError is far from clearly defined per RFC 4408. From
an entirely utilitarian point of view, all that can be said is that it
covers all error conditions that are not already covered by TempError,
whose semantics are a lot better defined in RFC 4408.

The real question then is: Does a "broken" <target-name> (either verba-
tim, e.g., "<more-than-63-chars>.com", or via macro expansion) need to be
_flagged_as_an_error_condition_ in the first place?

Had you asked me that question two years ago, I'd have said: definitely
yes! You may recall me having fought long and hard for making
SPF(NXDOMAIN) to be considered a case of PermError. It was all in vain;
the rough consensus was that checking for whether the authority domain
(whose policy is to be evaluated) actually exists was not SPF's problem
and that None should be returned. Likewise, consider 5/10/3 in section
5, "Mechanism Definitions":

| Several mechanisms rely on information fetched from DNS. [...] If the
| server returns "domain does not exist" (RCODE 3), then evaluation of the
| mechanism continues as if the server returned no error (RCODE 0) and
| zero answer records.

Now I think we ought to stick to that school of thought and treat "broken"
<target-name>s as no-match, too.

After all, given a policy using %{h}, why should "HELO whacky..spammer"
result in "PermError" when "HELO whacky.spammer" results in "Fail" or
"SoftFail"? (If anyone argues that "whacky..spammer" isn't even a valid
HELO per RFC 2821, consider %{l} and "MAIL FROM:<whacky..spammer@...>".)

As for the "mech:<more-than-63-chars>.com" case, I could be convinced that
this is a separate issue from the macro expansion one, however since it
clearly is NOT a syntax error per RFC 4408, it would still be hard to
justify a PermError result.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFH7SEXwL7PKlBZWjsRAmLiAJ0bri0uFkfAHMJYkiwo4djI5McfPQCeLqSM
/gC6CxQ1qZgeB1fNXVm4D7Y=
=YsYr
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle wrote:

> I think we should make decisions in this order. The RFC 4408 test
> suite depends solely on how we interpret RFC 4408 as it is now.

Yes, and as I never tried to implement my interpretation it's only
that, an interpretation, not running code.

I'd disagree with the point where you say that RFC 4408 considered
adjacent dots and overlong labels as valid <target-name>, I think
it simply forgot to discuss invalid <target-name>s apart from the
2.5.7 note.

IOW we screwed up, fortunately only in a totally obscure corner
case mostly relevant for test suites and errata. Maybe it could
be a vulnerability if spammers find servers where they like an
intentionally provoked Permerror better than no match, or v.v.,
depending on the receiver policy.

> Any RFC 4408 errata should then amend the RFC to clarify what it
> already says (albeit contradictorily or otherwise ambiguously).

Yep, additional 2.5.7 clarification for PermError, or stripping
the questionable statement for "no match" are good. I'm less
thrilled by "clarification that implemenations are free to pick
either PermError or no match, certainly not TempError", but it's
also possible.

Based on that, one thing is sure, just removing the section from
the errata page as "rejected" is out, whatever we pick, it needs
some text.

I think I can come up with a proposal for the three choices:

PermError => add note that this is about a syntactically invalid
<target-name> for DNS, for example adjacent dots.

No match => say that the 2.5.7 statement was wrong and should
be ignored, reflecting common "no match" practice.

Pick => say that some implementations skip a syntactically
invalid DNS <target-name>, e.g., adjacent dots.

> In any case, RFC 4408 errata are not supposed to modify the
> (originally intended) semantics of RFC 4408.

I fear we simply forgot to think about this corner case, and so
"originally intended semantics" equals a blind spot in our pre-
4408 reviews.

> Let's face it, PermError is far from clearly defined per RFC
> 4408.

Yeah, things can get tricky if many receivers decide to accept
PermError. In that case the spammers have to "educate" them ;-)

After that education, if PermError for legit mail simply means
"fix your policy", it's IMO fine. Admittedly the plan was to
limit it to "fix the policy because it's syntactically broken".

"Fix your policy" covering invalid <target-name> is still KISS.

> You may recall me having fought long and hard for making
> SPF(NXDOMAIN) to be considered a case of PermError.

Yes, but your concept was not simply "fix your policy", or was
it ? At that time we also had discussions about rejecting a
PermError at all, until we finally decided that it is receiver
policy.

> Now I think we ought to stick to that school of thought and
> treat "broken" <target-name>s as no-match, too.

I don't insist on it, I take the easy way out: Whatever the
test suite decides, and I'm aware that it already decided it,
but you could still change your mind, I have an idea what to
do with the "last erratum" as noted above. For TempError I'd
have no idea, but we have consensus that TempError is out.

> given a policy using %{h}, why should "HELO whacky..spammer"
> result in "PermError" when "HELO whacky.spammer" results in
> "Fail" or "SoftFail"?

Or "Neutral". The best decision depends on what you consider
as the possible *future* best common practice for receivers:

A rejected PermError is better than an accepted Neutral, and
a rejected Fail is better than an accepted PermError. As you
know I like SPF FAIL and hope that more domains will use it.

It is okay if you decide that "no match" does the trick.

We were not wasting time, until today I didn't have it clear
that a consistent erratum or clarification for 2.5.7 matching
any decision in the test suite is possible and required.

> for the "mech:<more-than-63-chars>.com" case, I could be
> convinced that this is a separate issue from the macro
> expansion one

Different, you could find it without instantiating macros.

> since it clearly is NOT a syntax error per RFC 4408, it
> would still be hard to justify a PermError result.

In the sense of "fix your policy" that is not hard, and it
is clearly an invalid label. Either a typo or a malicious
policy attacking the time resources of folks discussing
SPF test suites and errata, or another seriously bad idea.

I don't see how this case could ever result in an erratum.

Better decide the <target-name> handling first ignoring
this. After that decision it's easier, e.g. it could make
sense to pick the same solution for KISS and consistency,
it could also make sense to allow PermError or "no match"
for the overlong label.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yet another attempt:

http://www.openspf.org/source/project/test-suite/rfc4408-tests.yml?view=diff&r1=98&r2=89
http://www.openspf.org/source/project/test-suite/rfc4408-tests.CHANGES?rev=98&view=auto

This should incorporate the resolutions of all the errata that were under
discussion since the 2007.05 test-suite release.

Any objections to making that a 2008.08 release?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkimIHoACgkQwL7PKlBZWju59ACgjyx423Mk6wH0C6RKn78IXuls
gIUAnRqLk+8JBMk8/QxN9FsTzKop7FAc
=fBIR
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle wrote:

> Any objections to making that a 2008.08 release?

Looking only in the change log I see no problem.

Copied from my "last erratum" reply, because the
end was wrong; please add the "macro mania" tests:

| <http://article.gmane.org/gmane.mail.spam.spf.devel/1938>
|
| The rationale for the new "macro mania" tests was a
| developer trying to use a seriously limited .NET DNS
| API.
|
| The "quoted string" + "quoted pair" business will be
| a bit more complex, it could go into an update of the
| test suite when we have that clear.
|
| Using upper case %{S} in a <domain-spec> can also have
| odd effects, especially in conjunction with the "last"
| erratum limit 63:
|
- mail from: <123...25@789...50.example.com>
+ mail from: <123...31@345...62.example.com>
|
| "v=spf1 exists:%{S} ?all"
|
- .example (8) + .com (4) + 25 + @ (1) + 24 => 62. But
+ local part (31) + @ (1) + left label (30) => 62. But
| if the @ is URL encoded for an upper case S it is 64.

The "..." is not for three dots, it is an abbreviation
for omitted LDH fill characters (31 + @ + 30)

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Julian Mehnle wrote:
> > Any objections to making that a 2008.08 release?
>
> Looking only in the change log I see no problem.

Good, thanks for lending your eyeballs. Release impending.

> Copied from my "last erratum" reply, because the end was wrong; please
> add the "macro mania" tests:

I added them, but rolled them up into a single test case ("macro-mania-
in-domain"). No need to be extra redundant.

Next stop: 2008.08 release announcement.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkioGeMACgkQwL7PKlBZWjtexgCg45tTST8dgzsdtFV4IJ56ZHxm
xeQAnR1TkiIRpooqhlzOZMSqfFQPi62F
=qhZP
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle wrote:

> Next stop: 2008.08 release announcement.

Good, maybe add it to the "news" on the Web site,
we didn't have many news so far this year. Just
in case I mentioned the test suite in an SPF EAI
draft some weeks ago, with a link to the errata.

My SEO instincts suggest that the errata should
now get a link to the test suite, after all they
are covered (or at least supposed to be covered).

Frank
--
http://tools.ietf.org/html/draft-ellermann-spf-eai-03#section-3



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Mon, 18 Aug 2008, Frank Ellermann wrote:

> > Next stop: 2008.08 release announcement.

IMO, any test-suite changes should be double checked by running
against as many existing implementations as possible. I am running
against pyspf right now. Any new failures need to be diagnosed
as an implementation or test-suite problem. It is not enough just
to eyeball the changes.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Mon, 18 Aug 2008, Stuart D. Gathman wrote:

> IMO, any test-suite changes should be double checked by running
> against as many existing implementations as possible. I am running
> against pyspf right now. Any new failures need to be diagnosed

Ok, looks like pyspf has picked up 3 new warnings - permerror is now
the preferred result for invalid domain. Ideally, the spec should
pick either nomatch or permerror so that results will be consistent. However,
what we ended up with isn't so bad. Most implementations will nomatch an
invalid domain, usually resulting in FAIL. Some will permerror instead,
hopefully providing some diagnostics to the domain owner. I am planning to add
a config option to pyspf for selecting the nomatch or permerror behaviour.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Stuart D. Gathman ha scritto:
> On Mon, 18 Aug 2008, Frank Ellermann wrote:
>
>>> Next stop: 2008.08 release announcement.
>
> IMO, any test-suite changes should be double checked by running
> against as many existing implementations as possible. I am running
> against pyspf right now. Any new failures need to be diagnosed
> as an implementation or test-suite problem. It is not enough just
> to eyeball the changes.

I tested 2008.08 against jSPF and everything works.
I also tested current head for the test suite and we fail the macromania
test.

We didn't correctly expanded the %_ and %- (they was explanded to "_"
and "-" :-( ).

I have already an easy fix for that (It's very good to have this test
suite!!) ready to be committed.

At Apache JAMES project we are also planning to extract the test suite
for jSPF and publish it as a separate project (we currently have it in
our tests).

The jSPF tester have the ability to load a zone configuration from the
yaml file, set up a "fake" dsn server on a given port, read tests, run
the spf checker and check the results.

One issue I've found when trying our tester against other
implementations is that it is not easy to tell other implementations to
use a specific dns server/port for the resolution.

IMHO the jSPF tester could be a good way for OpenSPF to certify the
implementations, otherwise you will have to manually check if the test
suite for each implementation is correct or not.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Mon, 18 Aug 2008, Frank Ellermann wrote:
> > > Next stop: 2008.08 release announcement.
>
> IMO, any test-suite changes should be double checked by running
> against as many existing implementations as possible. I am running
> against pyspf right now. Any new failures need to be diagnosed
> as an implementation or test-suite problem. It is not enough just
> to eyeball the changes.

I agree with your thought, and I had run it against Mail::SPF (2.005 and
2.006) as well as pyspf (2.0.4 and 2.0.5) before pushing it out.
I didn't run it against any other implementations, though, mostly because
I don't trust any others as much as I trust Mail::SPF and pyspf.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkip38YACgkQwL7PKlBZWjvn4ACeOBQ6aBBYf5FmXe2cMhEfbVfU
jz4Animy6hDegTlTW0IAxadabxEuIu3W
=Na3k
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> I tested 2008.08 against jSPF and everything works.

Very good.

> I also tested current head for the test suite and we fail the
> macromania test.
>
> We didn't correctly expanded the %_ and %- (they was explanded to "_"
> and "-" :-( ).
>
> I have already an easy fix for that (It's very good to have this test
> suite!!) ready to be committed.

Even better! Good to know that the test suite helped expose those
mistakes. Mail::SPF 2.005 had a similar bug by expanding "%-" to "-", by
the way.

> The jSPF tester have the ability to load a zone configuration from the
> yaml file, set up a "fake" dsn server on a given port, read tests, run
> the spf checker and check the results.
>
> One issue I've found when trying our tester against other
> implementations is that it is not easy to tell other implementations to
> use a specific dns server/port for the resolution.

FWIW, it's all but trivial with Mail::SPF. One can specify a custom DNS
resolver object, e.g., a Net::DNS::Resolver object configured to always
use a specific resolving server.

> IMHO the jSPF tester could be a good way for OpenSPF to certify the
> implementations, otherwise you will have to manually check if the test
> suite for each implementation is correct or not.

How will this help getting rid of any manual procedures in testing the
conformity of a specification?

Besides, running an SPF implementations self-tests could be made (by
authors) as easy as `make test`. It even almost works with Mail::SPF --
you just have to run `perl Makefile.PL` beforehand.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkip4sgACgkQwL7PKlBZWjsJvACg/qM5EDSQepznc3TwAajuqGo9
njoAoNWmS/iDrp4LVauY7p0XLq3v54P+
=wEag
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> I tested 2008.08 against jSPF and everything works.
>
> Very good.
>
>> I also tested current head for the test suite and we fail the
>> macromania test.
>>
>> We didn't correctly expanded the %_ and %- (they was explanded to "_"
>> and "-" :-( ).
>>
>> I have already an easy fix for that (It's very good to have this test
>> suite!!) ready to be committed.
>
> Even better! Good to know that the test suite helped expose those
> mistakes. Mail::SPF 2.005 had a similar bug by expanding "%-" to "-", by
> the way.
>
>> The jSPF tester have the ability to load a zone configuration from the
>> yaml file, set up a "fake" dsn server on a given port, read tests, run
>> the spf checker and check the results.
>>
>> One issue I've found when trying our tester against other
>> implementations is that it is not easy to tell other implementations to
>> use a specific dns server/port for the resolution.
>
> FWIW, it's all but trivial with Mail::SPF. One can specify a custom DNS
> resolver object, e.g., a Net::DNS::Resolver object configured to always
> use a specific resolving server.

I'm not a perl programmer, so I won't do this. A configuration option to
override the dns servers would do the trick.

>> IMHO the jSPF tester could be a good way for OpenSPF to certify the
>> implementations, otherwise you will have to manually check if the test
>> suite for each implementation is correct or not.
>
> How will this help getting rid of any manual procedures in testing the
> conformity of a specification?

This won't get rid of *any* manual procedure, but this at least save you
from checking the code quality correctness for the testsuite runner.

> Besides, running an SPF implementations self-tests could be made (by
> authors) as easy as `make test`. It even almost works with Mail::SPF --
> you just have to run `perl Makefile.PL` beforehand.

The difference between this test and "selftests" is big, IMHO.
Most "self-tests" I've seen around don't actually run a real UDP/TCP dns
query, but they mock the whole resolution mechanism at one or another layer.
Otherwise we could have a "selftest" returning "ok" for any test without
checking the real result.

e.g: jSPF supports synchronous, asynchronous and SEDA (Staged
Event-Driven Architecture) resolvers and testing against a real dns
server raised much more issues than our previous testsuite.

Having an external testsuite allow you to check bugs in the testsuites
too. Often people that wrote the code is the same that wrote the tester
code, this can result in obvious bugs to be left there because of
misunderstandings from the author.

BTW I don't care if you will or won't use the jSPF tester once we'll
publish it, but I'm a bit annoyed that OpenSPF page keep saying that the
only compliant implementations are pyspf 2 and Mail::SPF only because
OpenSPF members wrote that while jSPF has an "the jSPF library is
currently being evaluated by the project for RFC 4408 compliance" since
years. At the end of 2006 jSPF was listed as "The jSPF library is also
reported to pass all tests in the test suite, however this has not yet
been officially confirmed" and then on 2006-12-10 you removed the whole
sentence. On 2007-02-01 then the "currently being evaluated" has been
written there... 18 months passed since that...

IMHO it is bad for OpenSPF to fail to tell how implementations can be
listed between compliant implementation and ignore anyone but your own
implementations.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Julian Mehnle ha scritto:
> > Stefano Bagnara wrote:
> > > The jSPF tester have the ability to load a zone configuration from
> > > the yaml file, set up a "fake" dsn server on a given port, read
> > > tests, run the spf checker and check the results.
> > >
> > > One issue I've found when trying our tester against other
> > > implementations is that it is not easy to tell other implementations
> > > to use a specific dns server/port for the resolution.
> >
> > FWIW, it's all but trivial with Mail::SPF. One can specify a custom
> > DNS resolver object, e.g., a Net::DNS::Resolver object configured to
> > always use a specific resolving server.
>
> I'm not a perl programmer, so I won't do this. A configuration option
> to override the dns servers would do the trick.

Actually, I'd rather have the online version of the test suite driver
append a given zone name to every lookup performed, e.g.,
2008_08.rfc4408.test-suite.openspf.org. That way you wouldn't have to
specify a custom resolver server. That zone can then be delegated to
your fake DNS server.

> > Besides, running an SPF implementations self-tests could be made (by
> > authors) as easy as `make test`. It even almost works with Mail::SPF
> > -- you just have to run `perl Makefile.PL` beforehand.
>
> The difference between this test and "selftests" is big, IMHO.
> Most "self-tests" I've seen around don't actually run a real UDP/TCP
> dns query, but they mock the whole resolution mechanism at one or
> another layer. Otherwise we could have a "selftest" returning "ok" for
> any test without checking the real result.

I don't see the difference as being all that big, given that the test
suite drivers are probably rather trivial, but I certainly don't object
to have SPF implementations tested against live DNS. If you're willing
to help with setting this up, or even lead the effort, all the better!

> BTW I don't care if you will or won't use the jSPF tester once we'll
> publish it, but I'm a bit annoyed that OpenSPF page keep saying that
> the only compliant implementations are pyspf 2 and Mail::SPF only
> because OpenSPF members wrote that while jSPF has an "the jSPF library
> is currently being evaluated by the project for RFC 4408 compliance"
> since years. At the end of 2006 jSPF was listed as "The jSPF library is
> also reported to pass all tests in the test suite, however this has not
> yet been officially confirmed" and then on 2006-12-10 you removed the
> whole sentence. On 2007-02-01 then the "currently being evaluated" has
> been written there... 18 months passed since that...
>
> IMHO it is bad for OpenSPF to fail to tell how implementations can be
> listed between compliant implementation and ignore anyone but your own
> implementations.

As Stuart said, this isn't in any way due to a desire for self-promotion.
The reason that the website still says "The jSPF library is currently
being evaluated by the project for RFC 4408 compliance" simply is that
when I tried to run the (offline) test-suite against jSPF over a year
ago, it was very complicated to get it to work for someone who isn't an
experienced Java/Maven developer, and that the generated output was
practically unreadable, and no one else capable of editing the website
has since tried checking out jSPF's compliance. I admit that it was my
fault not to have announced that I had given up on trying to get useful
output from my jSPF test-suite runs, and hadn't asked that someone else
give it a try.

So please don't be mad, but let's now try to get this resolved instead.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkir3rIACgkQwL7PKlBZWjt+hQCgvaUqjODGqBHoBGCj6DbWJIzo
qQUAn1lV0LJR/sbXPowmNdrKSfu1YRG1
=jZrW
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> Stefano Bagnara wrote:
>> IMHO it is bad for OpenSPF to fail to tell how implementations can be
>> listed between compliant implementation and ignore anyone but your own
>> implementations.
>
> As Stuart said, this isn't in any way due to a desire for self-promotion.
> The reason that the website still says "The jSPF library is currently
> being evaluated by the project for RFC 4408 compliance" simply is that
> when I tried to run the (offline) test-suite against jSPF over a year
> ago, it was very complicated to get it to work for someone who isn't an
> experienced Java/Maven developer, and that the generated output was
> practically unreadable, and no one else capable of editing the website
> has since tried checking out jSPF's compliance. I admit that it was my
> fault not to have announced that I had given up on trying to get useful
> output from my jSPF test-suite runs, and hadn't asked that someone else
> give it a try.

Probably running java applications is difficult for you as running perl
applications for me.
How easy is for me (let's say I only understand java) to run the
testsuite for other implementations?? Hard. It's not jSPF that is hard,
it is simply that the test suite for a library is usually written in the
library own language and using that very language testing methodologies.
We use jUnit and we wrapped the yaml tests into jUnit tests, so it is
really easy to run our test suite graphically in any jUnit compatible
environment.

> So please don't be mad, but let's now try to get this resolved instead.

My complain is that either you publish a procedure that I will try to
follow or you say that jSPF is compliant. It doesn't work and it's not
fair for me that you say "I don't know Java so your implementation will
not be listed".

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Probably running java applications is difficult for you as running perl
> applications for me.
> How easy is for me (let's say I only understand java) to run the
> testsuite for other implementations?? Hard. It's not jSPF that is hard,
> it is simply that the test suite for a library is usually written in
> the library own language and using that very language testing
> methodologies. We use jUnit and we wrapped the yaml tests into jUnit
> tests, so it is really easy to run our test suite graphically in any
> jUnit compatible environment.
>
> > So please don't be mad, but let's now try to get this resolved
> > instead.
>
> My complain is that either you publish a procedure that I will try to
> follow or you say that jSPF is compliant. It doesn't work and it's not
> fair for me that you say "I don't know Java so your implementation will
> not be listed".

We don't have a formal procedure, and I don't think establishing one will
be worthwhile to any extent. We should simply strive to be fair. If we
aren't, please hit us on the head.

The issue with your implementation wasn't that I don't _speak_ Java, but
that there was no good documentation on how to set it up (cf.
http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL if you will)
for someone who isn't familiar with know how to _use_ Java, Maven, and
jUnit, and, even more importantly, the test output was incomprehensible
to me. Perhaps that has since changed, but back then that was just how
it was.

Can you produce somethink like the following?

http://julian.io.link-m.de/tmp/mail-spf-perl-rfc4408-tests.log

You'll see that this isn't exactly good proof that an implementation
passes the test suite, but it at least demonstrates that the test suite
has been run in some way.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkir9GwACgkQwL7PKlBZWjuJagCfYnpMHnUoz0aMjZuBdVpxHmAM
MJgAoLFZHUwmJck3W/GIgrxJ2Xp26xaj
=l5+w
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Probably running java applications is difficult for you as running perl
>> applications for me.
>> How easy is for me (let's say I only understand java) to run the
>> testsuite for other implementations?? Hard. It's not jSPF that is hard,
>> it is simply that the test suite for a library is usually written in
>> the library own language and using that very language testing
>> methodologies. We use jUnit and we wrapped the yaml tests into jUnit
>> tests, so it is really easy to run our test suite graphically in any
>> jUnit compatible environment.
>>
>>> So please don't be mad, but let's now try to get this resolved
>>> instead.
>> My complain is that either you publish a procedure that I will try to
>> follow or you say that jSPF is compliant. It doesn't work and it's not
>> fair for me that you say "I don't know Java so your implementation will
>> not be listed".
>
> We don't have a formal procedure, and I don't think establishing one will
> be worthwhile to any extent. We should simply strive to be fair. If we
> aren't, please hit us on the head.
>
> The issue with your implementation wasn't that I don't _speak_ Java, but
> that there was no good documentation on how to set it up (cf.
> http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL if you will)
> for someone who isn't familiar with know how to _use_ Java, Maven, and
> jUnit, and, even more importantly, the test output was incomprehensible
> to me. Perhaps that has since changed, but back then that was just how
> it was.

In every distribution we have BUILDING.txt:
-----
#############################################################################
# BUILDING JSPF
#############################################################################

You will need to install maven2 as well as acquire jSPF source from
subversion
or a source tarball.

Steps:

1) Install maven2 (v2.0.7 as of the time of this writing)

2) Add maven2 to your path. For me, I do the following:
$ tar zxvf maven-2.0.7.tar.gz
$ mv maven-2.0.7 /usr/local
$ ln -sf /usr/local/maven-2.0.7/bin/mvn /usr/local/bin/mvn

3) Change directory to jspf source dir
$ cd jspf-x/

4) Run the build (the "-Plocal" restrict maven to use bundled dependencies)
$ mvn -Plocal package
-----------------

It doesn't seems more difficult than the instructions you give for the
perl implementation.

> Can you produce somethink like the following?
>
> http://julian.io.link-m.de/tmp/mail-spf-perl-rfc4408-tests.log

A list of tests with an "ok" on their side??
The above "mvn package" command will run tests. The output will be more
verbose, but if any failure happens the package is not build and a
"BUILD FAILURE" appears.

> You'll see that this isn't exactly good proof that an implementation
> passes the test suite, but it at least demonstrates that the test suite
> has been run in some way.

We run it as part of our continuos integration build.
We run it at least 200 times per day and at least once per each commit.

If you just need a log of the test run to certificate jSPF I'm happy..
I'll send you the next output.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Stefano Bagnara ha scritto:
> Julian Mehnle ha scritto:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Stefano Bagnara wrote:
>>> Probably running java applications is difficult for you as running perl
>>> applications for me.
>>> How easy is for me (let's say I only understand java) to run the
>>> testsuite for other implementations?? Hard. It's not jSPF that is hard,
>>> it is simply that the test suite for a library is usually written in
>>> the library own language and using that very language testing
>>> methodologies. We use jUnit and we wrapped the yaml tests into jUnit
>>> tests, so it is really easy to run our test suite graphically in any
>>> jUnit compatible environment.
>>>
>>>> So please don't be mad, but let's now try to get this resolved
>>>> instead.
>>> My complain is that either you publish a procedure that I will try to
>>> follow or you say that jSPF is compliant. It doesn't work and it's not
>>> fair for me that you say "I don't know Java so your implementation will
>>> not be listed".
>>
>> We don't have a formal procedure, and I don't think establishing one
>> will be worthwhile to any extent. We should simply strive to be
>> fair. If we aren't, please hit us on the head.
>>
>> The issue with your implementation wasn't that I don't _speak_ Java,
>> but that there was no good documentation on how to set it up (cf.
>> http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL if you
>> will) for someone who isn't familiar with know how to _use_ Java,
>> Maven, and jUnit, and, even more importantly, the test output was
>> incomprehensible to me. Perhaps that has since changed, but back then
>> that was just how it was.
>
> In every distribution we have BUILDING.txt:
> -----
> #############################################################################
>
> # BUILDING JSPF
> #############################################################################
>
>
> You will need to install maven2 as well as acquire jSPF source from
> subversion
> or a source tarball.
>
> Steps:
>
> 1) Install maven2 (v2.0.7 as of the time of this writing)
>
> 2) Add maven2 to your path. For me, I do the following:
> $ tar zxvf maven-2.0.7.tar.gz
> $ mv maven-2.0.7 /usr/local
> $ ln -sf /usr/local/maven-2.0.7/bin/mvn /usr/local/bin/mvn
>
> 3) Change directory to jspf source dir
> $ cd jspf-x/
>
> 4) Run the build (the "-Plocal" restrict maven to use bundled dependencies)
> $ mvn -Plocal package
> -----------------
>
> It doesn't seems more difficult than the instructions you give for the
> perl implementation.
>
>> Can you produce somethink like the following?
>>
>> http://julian.io.link-m.de/tmp/mail-spf-perl-rfc4408-tests.log
>
> A list of tests with an "ok" on their side??
> The above "mvn package" command will run tests. The output will be more
> verbose, but if any failure happens the package is not build and a
> "BUILD FAILURE" appears.
>
>> You'll see that this isn't exactly good proof that an implementation
>> passes the test suite, but it at least demonstrates that the test
>> suite has been run in some way.
>
> We run it as part of our continuos integration build.
> We run it at least 200 times per day and at least once per each commit.
>
> If you just need a log of the test run to certificate jSPF I'm happy..
> I'll send you the next output.

Here is the last successful build test result. I think it's more fancy
than your perl log, isn't it?

http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/

We also run more tests as you can see here:
http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/

Because we also have an asynchronous resolver.

The test above is for our latest trunk tested against the latest yaml
testsuite from your trunk.

We only make releases when our product passes all of the last yaml suite
tests.

In fact our build tools reject to build jSPF at all if there are failures.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Stefano Bagnara ha scritto:
> > Julian Mehnle ha scritto:
> > > You'll see that this isn't exactly good proof that an implementation
> > > passes the test suite, but it at least demonstrates that the test
> > > suite has been run in some way.
> >
> > We run it as part of our continuos integration build.
> > We run it at least 200 times per day and at least once per each
> > commit.
> >
> > If you just need a log of the test run to certificate jSPF I'm
> > happy.. I'll send you the next output.
>
> Here is the last successful build test result. I think it's more fancy
> than your perl log, isn't it?

This isn't about being fancy.

> http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/

That looks acceptable (though as I said it isn't exactly good proof). Is
there a _release_ version of jSPF that passes _that_release_ of the test
suite? And which release of the test suite was that? 2007.05?

> The test above is for our latest trunk tested against the latest yaml
> testsuite from your trunk.

Please don't use "trunk" versions of our test suite (i.e., do not use the
file named "rfc4408-tests.yml"). That's the development version. Please
use the files named "rfc4408-tests-YYYY.MM.yml" instead.

> We only make releases when our product passes all of the last yaml
> suite tests.
>
> In fact our build tools reject to build jSPF at all if there are
> failures.

That's certainly good practice.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkisB0AACgkQwL7PKlBZWjvj5wCgm9HzVgFYUcmNN+ugjhmrK1d+
JRQAn1xdxtdfo3HR7rn7hESbpq9q1gph
=lJxP
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Stefano Bagnara ha scritto:
>>> Julian Mehnle ha scritto:
>>>> You'll see that this isn't exactly good proof that an implementation
>>>> passes the test suite, but it at least demonstrates that the test
>>>> suite has been run in some way.
>>> We run it as part of our continuos integration build.
>>> We run it at least 200 times per day and at least once per each
>>> commit.
>>>
>>> If you just need a log of the test run to certificate jSPF I'm
>>> happy.. I'll send you the next output.
>> Here is the last successful build test result. I think it's more fancy
>> than your perl log, isn't it?
>
> This isn't about being fancy.
>
>> http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/
>
> That looks acceptable (though as I said it isn't exactly good proof). Is
> there a _release_ version of jSPF that passes _that_release_ of the test
> suite? And which release of the test suite was that? 2007.05?

Out last release (jSPF 0.9.6) has been done when 2007.05 was out and the
tests passed against 2007.05. jSPF 0.9.6 also fully pass 2007.08, it has
1 issue in macromania that you added after 2007.08 in trunk.

Our macromania bug has been fixed in our trunk and the above report is
for the trunk version and your trunk testsuite including the macromania
test and our fix.

>> The test above is for our latest trunk tested against the latest yaml
>> testsuite from your trunk.
>
> Please don't use "trunk" versions of our test suite (i.e., do not use the
> file named "rfc4408-tests.yml"). That's the development version. Please
> use the files named "rfc4408-tests-YYYY.MM.yml" instead.

I use trunk in our trunk, so I can spot any issue as soon as possible.
In past it happened that openspf added tests that we didn't pass and we
complained for that and you finally removed them because they was
controversial, so I think it is good to test against trunk in our dev
snapshot.

I'll take care to add both the latest official and the latest trunk in
our test tree, so that we always check both suites.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Out last release (jSPF 0.9.6) has been done when 2007.05 was out and
> the tests passed against 2007.05. jSPF 0.9.6 also fully pass 2007.08,
> it has 1 issue in macromania that you added after 2007.08 in trunk.

There was no 2007.08. Do you mean 2008.08? If that's what you meant,
then there is nothing that has been added after the release of 2008.08
other than a cosmetic change to the spec reference field of the
"multitxt2" (not macromania) test.

> Our macromania bug has been fixed in our trunk and the above report is
> for the trunk version and your trunk testsuite including the macromania
> test and our fix.

Usually people use only release versions, so it is important that we tell
them which test suite release your latest release version conforms to.

Can the jSPF trunk version be downloaded other than by checking it out
from the Subversion repository? Or is the fix included in a nightly
build (assuming there even is a standalone nightly build for jSPF)?

If not, I suppose we should record it as follows:

2007.05: since 0.9.6

> >> The test above is for our latest trunk tested against the latest
> >> yaml testsuite from your trunk.
> >
> > Please don't use "trunk" versions of our test suite (i.e., do not use
> > the file named "rfc4408-tests.yml"). That's the development version.
> > Please use the files named "rfc4408-tests-YYYY.MM.yml" instead.
>
> I use trunk in our trunk, so I can spot any issue as soon as possible.
> In past it happened that openspf added tests that we didn't pass and we
> complained for that and you finally removed them because they was
> controversial, so I think it is good to test against trunk in our dev
> snapshot.
>
> I'll take care to add both the latest official and the latest trunk in
> our test tree, so that we always check both suites.

This may mislead you because sometimes we add or modify tests in the test
suite's trunk in a temporary fashion, e.g., when an issue is still under
discussion. Thus the change may get reversed later in a proper release
of the test suite. It has happened before.

In any case, compliance statements should only ever refer to release
versions of the test suite, so trying to comply with a trunk version
isn't going to be of much value.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkisK7sACgkQwL7PKlBZWju3xACgkmcToY1dKr8Smauumhu4ddAQ
macAnAqC7WO2toadGGQBE7pIIDBcLRxI
=xlt1
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Out last release (jSPF 0.9.6) has been done when 2007.05 was out and
>> the tests passed against 2007.05. jSPF 0.9.6 also fully pass 2007.08,
>> it has 1 issue in macromania that you added after 2007.08 in trunk.
>
> There was no 2007.08. Do you mean 2008.08? If that's what you meant,
> then there is nothing that has been added after the release of 2008.08
> other than a cosmetic change to the spec reference field of the
> "multitxt2" (not macromania) test.

sorry, you are right.

with 2008.08 jSPF 0.9.6 fails the macromania test because %- and %_ are
incorrectly expanded to - and _.
I fixed that as soon as I saw the macromania test in openspf testsuite.

>> Our macromania bug has been fixed in our trunk and the above report is
>> for the trunk version and your trunk testsuite including the macromania
>> test and our fix.
>
> Usually people use only release versions, so it is important that we tell
> them which test suite release your latest release version conforms to.

jSPF 0.9.6 conforms to (and includes) rfc4408-tests-2007.05.yml
The next release will pass the current specification. The only known
issue with jSPF 0.9.6 is the macro expansion above.

> Can the jSPF trunk version be downloaded other than by checking it out
> from the Subversion repository? Or is the fix included in a nightly
> build (assuming there even is a standalone nightly build for jSPF)?

Our continuos integration server produce "lastStableBuild" artifacts:
http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/lastStableBuild/org.apache.james.jspf$resolver/
They includes binaries and sources for the last version that passed
every test (in this case the current trunk is working).

At ASF (Apache Software Foundation) we have a strict release process and
we never recommend the use of snapshots and we never point users to
nightlies or snapshots.

> If not, I suppose we should record it as follows:
>
> 2007.05: since 0.9.6

If you tell me that it won't take 2 more years to update the page I'll
ask a new run as soon as we'll release 0.9.7. We'll probably wait a bit
more because we want to include newer dns libraries so to fix any
Kaminsky DNS vulnerability and a race issue in the asynchronous version
of our resolver.

>>>> The test above is for our latest trunk tested against the latest
>>>> yaml testsuite from your trunk.
>>> Please don't use "trunk" versions of our test suite (i.e., do not use
>>> the file named "rfc4408-tests.yml"). That's the development version.
>>> Please use the files named "rfc4408-tests-YYYY.MM.yml" instead.
>> I use trunk in our trunk, so I can spot any issue as soon as possible.
>> In past it happened that openspf added tests that we didn't pass and we
>> complained for that and you finally removed them because they was
>> controversial, so I think it is good to test against trunk in our dev
>> snapshot.
>>
>> I'll take care to add both the latest official and the latest trunk in
>> our test tree, so that we always check both suites.
>
> This may mislead you because sometimes we add or modify tests in the test
> suite's trunk in a temporary fashion, e.g., when an issue is still under
> discussion. Thus the change may get reversed later in a proper release
> of the test suite. It has happened before.

I know it happened. I was one of the people complaining to have it
reverted ;-)

I check trunk by purpose because I want to be aware of this issues ASAP
so that I can bring my opinion to this list before you release a bad
testsuite.

> In any case, compliance statements should only ever refer to release
> versions of the test suite

I agree.

> so trying to comply with a trunk version
> isn't going to be of much value.

I found it invaluable in past and it is not a big cost for us.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com