Mailing List Archive

On Mon, 22 May 2006, Stuart D. Gathman wrote:

> Starting with Wayne Schlitts test suite:
>
> http://prdownloads.sourceforge.net/pymilter/tests_v2.1.tar.gz?download
>
> we need to update it to rfc4408, and extract the DNS data it references to
> a text file.

Julian adds (about the same time I posted):

> I have set up a preliminary working group page for the official test suite:

> http://new.openspf.org/Test_Suite

> Maybe you'd like to mention it when you announce the test suite WG effort.

> In the meantime I'll try to collect the various old test suite bits out
> there (Wayne's old test suite, the libspf2 tests, etc.). However I think
> we should start fresh and only use the really good bits from the old stuff

So there is a vote to skip converting waynes test suite. I'm not sure
I agree (yet, anyway).

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> Tasks:
>
> 1) Decide text file format. Suggestions were XAML, JSON, CSV. Bind is
> not an option because it is tricky to parse and the point is *not* to
> use a real DNS server. My current vote is JSON - but then I don't know
> what XAML is or looks like.

s/XAML/YAML/. http://www.yaml.org/refcard.html describes YAML in itself as
an example.

> 2) Write script to extract DNS records referenced by current suite to
> text database. It is a one-off script, so we don't care what language
> you use - we just want the result. It may need to run multiple times
> and/or merge results to handle TEMP errors in the DNS servers holding
> the data.

What DNS zone are you talking about?

BTW, Shevek offered to provide a lot of test records in the "spftest.org."
zone (as well as delegating the zone to a name server of our choice in
case we want to host part of the test suite in DNS online).

The "spf1-test.mailzone.com." zone used by the Mail::SPF::Query test
script[1] is far from comprehensive and in part outright archaic.

> 3) Write a driver for your favorite SPF implementation (I will do
> pyspf). Run the suite, and review failures to determine whether the test
> suite or your implementation needs updating. Submit test suite patches
> to me. Post tricky cases to spf-discuss. Suggest changes to the test
> suite syntax that would make driving your implementation easier.

I think we should set up an issue tracker for the test suite mini-project.
Koen offered to set up additional queues for SPF-related tasks on his RT
installation, which already runs the contact form support queue.
Comments? Should such an issue tracker be publicly readable? (I think
it should.)

I'll care for Mail::SPF, which needs to be developed a bit further from its
current state (and documented!) anyway.

I think we should define a coverage strategy for the test suite:

* We should create at least one positive and one negative test for each
and every RFC 2119 keyword ("MUST", "MAY", etc.) in RFC 4408 that is
relevant to SPF implementations (some aren't). Thus we would have to
create an index of the spec's authoritative statements to determine
which are relevant and how to best test them.

* Besides, as Wayne did in his test suite, we should test for common
mistakes and tricky corner cases. We should look at the existing test
data to see what's already there.

* Should we create tests for "strict vs. lax" implementation behavior?
Those tests could be used in special modes (i.e. "strict"/"lax") of the
test suite.

Comments and additions?

References:
1. http://search.cpan.org/src/JMEHNLE/Mail-SPF-Query-1.999.1/t/test.dat

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEcilBwL7PKlBZWjsRAlijAJ9kSgrn0JpLYgoxAvxy21FYuDCKTACfbGcH
DNARZz6NMC9Q6BusiW7Sxw8=
=5uAa
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Mon, 22 May 2006, Julian Mehnle wrote:

> > 2) Write script to extract DNS records referenced by current suite to
> > text database. It is a one-off script, so we don't care what language
> > you use - we just want the result. It may need to run multiple times
> > and/or merge results to handle TEMP errors in the DNS servers holding
> > the data.
>
> What DNS zone are you talking about?

The records referenced in Wayne's test suite. (Some of which are supposed
to return NXDOMAIN.) I see you're campaigning to start from scratch,
in which case we skip that.

> BTW, Shevek offered to provide a lot of test records in the "spftest.org."
> zone (as well as delegating the zone to a name server of our choice in
> case we want to host part of the test suite in DNS online).

That means putting all the test DNS records in a single zone. Does
this preclude testing for some kinds of errors?

In any case, the the online copy should be identical to a specific
released text form of the data.

> The "spf1-test.mailzone.com." zone used by the Mail::SPF::Query test
> script[1] is far from comprehensive and in part outright archaic.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com