Mailing List Archive

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Monte Hansen wrote:
> I've tried to catch up on all the recent posts and didn't see any mention
> about anything related to the zone data with the new test suite. For
> instance, with the various live tests, I could run them without setup
> since they worked against live dns records in the mailzone.com tld. Not
> so in the new suite, it appears (example.com).
>
> I'm not clear what the plan is, but wouldn't it make sense to use that
> model. This way _each_ tester isn't required to stage an environment in
> accordance with every "zonedata:" test case from the yaml records. It
> would only need to be setup once for everyone to receive the benefit of
> it, and aligns with the Development Roadmap 1 c.

The new test suite will foremost and above all be an offline test suite,
the rationale being that often software, when being built and tested
automatically (such as by the Debian project's package builders system),
has no access to the Internet for practical and/or security reasons. IOW,
SPF implementations should be able to be tested offline.

You are not supposed to enter the zone data from the test suite into real
DNS. Any implementation should use a so-called adapter or test driver to
read the zonedata data from the test suite file instead of from real DNS.

However, we do plan on mirroring the test data in real DNS once a solid
release of the offline test suite has been finished and we thus no longer
need to concentrate our efforts on getting the test suite done ASAP. Then
implementations can test themselves using the live DNS if they want
(althouth that will not be the preferred modus operandi of testing.)

> That said, has anyone ported the live test cases to the new yaml layout?

No, not yet. Some of the old tests will be ported, some will be considered
obsolete. Also, some new tests will be developed.

Everyone who wants to contribute valuable tests is invited to do so.
Please post tests to spf-devel in the new test suite schema (I'll provide
an update shortly).

> I think I'm ready to release the COM version of SPF, I'm just trying to
> shake it down against as much test data I can gather, as well as
> implementing the new unit test model. Live data would help (so I don't
> hafta write code to add the dns records =).

Great to hear you're doing a COM implementation of SPF! Please keep us
posted on the status.

The new test suite is far from ready (although it might not take all that
long). You can use some of the older test data[1] until it is ready.

References:
1. http://www.openspf.org/svn/project/test-suite/contrib/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEzG0RwL7PKlBZWjsRAqhpAKChjsRZRD1K21kmlM7sDGJjC4DjXgCdHVOA
2C8RG+Aqlp8RhQHhRxgEAyM=
=l5KI
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

> The new test suite will foremost and above all be an offline test suite,
> the rationale being that often software, when being built and tested
> automatically (such as by the Debian project's package builders system),
> has no access to the Internet for practical and/or security reasons. IOW,
> SPF implementations should be able to be tested offline.
>

Wow, knock me over with a feather. I would think that would not be the
exception, not the rule. Was this a shift along the way? I mean, why the
effort to update the dns for mailzone.com and spftest.org?

> You are not supposed to enter the zone data from the test suite into real
> DNS. Any implementation should use a so-called adapter or test driver to
> read the zonedata data from the test suite file instead of from real DNS.

Again, I'm just so very surprised. So, every single implementation has to
construct their own dns driver just to implement the unit tests? This just
requires _so much more code_ from everyone to create a test arena. Moreover,
it also make the testing process more fallible. DNS already works, and we've
already written code to use it to support the framework. If
someone/something doesn't have access to the Internet for an automated test
suite, then let them install a local dns server for their test suite. I
strongly urge that you guys rethink that one. Make it the exception, not the
rule. It just seems like I'm being punished or something: the test suite is
my cross to bear, and I have to carry it with me inside the framework libary
code too ;-)

Please do not impose this on the implementations. It should be setup for the
implementations to use already -- it's part of the environment that should
already be in place. Each implementation shouldn't have to construct it's
own environment, unless they need to for their own reasons. Imposing this
will be a turnoff for developers on future implementations.

Respectfully submitted,

Monte Hansen


"Julian Mehnle" <julian@mehnle.net> wrote in message
news:200607300825.53749.julian@mehnle.net...
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Monte Hansen wrote:
>> I've tried to catch up on all the recent posts and didn't see any mention
>> about anything related to the zone data with the new test suite. For
>> instance, with the various live tests, I could run them without setup
>> since they worked against live dns records in the mailzone.com tld. Not
>> so in the new suite, it appears (example.com).
>>
>> I'm not clear what the plan is, but wouldn't it make sense to use that
>> model. This way _each_ tester isn't required to stage an environment in
>> accordance with every "zonedata:" test case from the yaml records. It
>> would only need to be setup once for everyone to receive the benefit of
>> it, and aligns with the Development Roadmap 1 c.
>
> The new test suite will foremost and above all be an offline test suite,
> the rationale being that often software, when being built and tested
> automatically (such as by the Debian project's package builders system),
> has no access to the Internet for practical and/or security reasons. IOW,
> SPF implementations should be able to be tested offline.
>
> You are not supposed to enter the zone data from the test suite into real
> DNS. Any implementation should use a so-called adapter or test driver to
> read the zonedata data from the test suite file instead of from real DNS.
>
> However, we do plan on mirroring the test data in real DNS once a solid
> release of the offline test suite has been finished and we thus no longer
> need to concentrate our efforts on getting the test suite done ASAP. Then
> implementations can test themselves using the live DNS if they want
> (althouth that will not be the preferred modus operandi of testing.)
>
>> That said, has anyone ported the live test cases to the new yaml layout?
>
> No, not yet. Some of the old tests will be ported, some will be
> considered
> obsolete. Also, some new tests will be developed.
>
> Everyone who wants to contribute valuable tests is invited to do so.
> Please post tests to spf-devel in the new test suite schema (I'll provide
> an update shortly).
>
>> I think I'm ready to release the COM version of SPF, I'm just trying to
>> shake it down against as much test data I can gather, as well as
>> implementing the new unit test model. Live data would help (so I don't
>> hafta write code to add the dns records =).
>
> Great to hear you're doing a COM implementation of SPF! Please keep us
> posted on the status.
>
> The new test suite is far from ready (although it might not take all that
> long). You can use some of the older test data[1] until it is ready.
>
> References:
> 1. http://www.openspf.org/svn/project/test-suite/contrib/
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
>
> iD8DBQFEzG0RwL7PKlBZWjsRAqhpAKChjsRZRD1K21kmlM7sDGJjC4DjXgCdHVOA
> 2C8RG+Aqlp8RhQHhRxgEAyM=
> =l5KI
> -----END PGP SIGNATURE-----
>
> -------
> To unsubscribe, change your address, or temporarily deactivate your
> subscription,
> please go to
> http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com
>



-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Monte Hansen wrote:
> > The new test suite will foremost and above all be an offline test
> > suite, the rationale being that often software, when being built and
> > tested automatically (such as by the Debian project's package builders
> > system), has no access to the Internet for practical and/or security
> > reasons. IOW, SPF implementations should be able to be tested
> > offline.
>
> Wow, knock me over with a feather. I would think that would not be the
> exception, not the rule. Was this a shift along the way? I mean, why the
> effort to update the dns for mailzone.com and spftest.org?

What effort to update the DNS for mailzone.com and spftest.org? I know of
no such effort.

> > You are not supposed to enter the zone data from the test suite into
> > real DNS. Any implementation should use a so-called adapter or test
> > driver to read the zonedata data from the test suite file instead of
> > from real DNS.
>
> Again, I'm just so very surprised. So, every single implementation has
> to construct their own dns driver just to implement the unit tests?

No. Like before, any implementation is free to implement testing on its
own. It's not like we were _removing_ anything from existence.

What we are doing is creating a _new_, _official_ test suite, which happens
to be developed in an offline form. Don't use it if you don't like it.

> This just requires _so much more code_ from everyone to create a test
> arena.

I doubt it.

Anyway, I don't understand why you're making such a fuss about this. As
soon as the offline test data is ready, anyone is free to copy it to a
real DNS zone of their choosing and exercise their implementation against
it. (Heck, even _we_ might do it, but who cares if _anybody_ can?)


The rest is redundant, or tangential only, but here we go:

> Moreover, it also make the testing process more fallible.

On the contrary. Offline testing eliminates all outside sources of prob-
lems in order to concentrate on the SPF core functionality. An implemen-
tation hardly needs an officially approved test suite in order to test
plain DNS functionality.

> If someone/something doesn't have access to the Internet for an automated
> test suite, then let them install a local dns server for their test
> suite.

Right, as if all the implementations were going to add bind9 (or even just
djbdns) to their build dependencies. No, that's not a good idea at all.
Besides, you are really missing the fact that you can always load the zone
data into your own DNS zone.

> Please do not impose this on the implementations.

We're not imposing anything that hasn't been imposed all the time already.
We're not removing anything from existence.

> Each implementation shouldn't have to construct it's own environment,
> unless they need to for their own reasons. Imposing this will be a
> turnoff for developers on future implementations.

I doubt it. Loading zone data into your own zone is trivial (but I'm
repeating myself), and writing an adapter for your implementation that
directly reads the offline data should be almost trivial as well.
Certainly a lot simpler than writing the SPF implementation itself anyway.

I guess you just need to think the situation over once more, then it should
all become clear. ;-)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEzNjDwL7PKlBZWjsRAjNPAJ4oRx2qGUkxLm6gC/hfWbhrPLwifgCgmRCB
HxNcKUKz9ewyhSsxCA0B3h4=
=1IAL
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

> What effort to update the DNS for mailzone.com and spftest.org? I know of
> no such effort.
The records exist to suit the live test cases, and are acknowledged as line
item 1(c) on the Development Roadmap. This is the only hint on the
http://new.openspf.org/Test_Suite page that infers an online/offline
methodology, and that speaks online to me.

Julian, I'll spare you the inline rebuts =) and sum up like so:

To impose an offline style test suite IS more work for everyone than it is
for an online style, merely to suit a condition that doesn't apply to
everyone. True? That's both dns administration and code to handle the
offline methodology (for all existing and non-existing implementations). An
online methodology means only 1 person takes the hit of setting up the live
records, for the benefit of a larger audience, no hooks or dns drivers
needed in the implementation.

Maybe to dns admins it's easy to stage the environment, but not every author
that implements spf is a dns admin class developer (like me). This would
also be locked down in (managed) corporate environments (running XP, which
most of the corprorate world runs). Not every workstation is capable of
doing so either. I'm a hellofa developer, and crappy dns admin =). But as a
developer I say to you it is considerably more effort to program hooks to
handle a dns driver, and every developer that assumes this task for future
implementations will have to do the same (if they want to use the entire
test suite). If you make the zone records live, then the existing dns
drivers in place will still work -- best of both worlds.

If I'm the only one that feels this way, then I'll go to the corner and chaw
on my humble pie.

Monte Hansen


"Julian Mehnle" <julian@mehnle.net> wrote in message
news:200607301605.23453.julian@mehnle.net...
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Monte Hansen wrote:
>> > The new test suite will foremost and above all be an offline test
>> > suite, the rationale being that often software, when being built and
>> > tested automatically (such as by the Debian project's package builders
>> > system), has no access to the Internet for practical and/or security
>> > reasons. IOW, SPF implementations should be able to be tested
>> > offline.
>>
>> Wow, knock me over with a feather. I would think that would not be the
>> exception, not the rule. Was this a shift along the way? I mean, why the
>> effort to update the dns for mailzone.com and spftest.org?
>
> What effort to update the DNS for mailzone.com and spftest.org? I know of
> no such effort.
>
>> > You are not supposed to enter the zone data from the test suite into
>> > real DNS. Any implementation should use a so-called adapter or test
>> > driver to read the zonedata data from the test suite file instead of
>> > from real DNS.
>>
>> Again, I'm just so very surprised. So, every single implementation has
>> to construct their own dns driver just to implement the unit tests?
>
> No. Like before, any implementation is free to implement testing on its
> own. It's not like we were _removing_ anything from existence.
>
> What we are doing is creating a _new_, _official_ test suite, which
> happens
> to be developed in an offline form. Don't use it if you don't like it.
>
>> This just requires _so much more code_ from everyone to create a test
>> arena.
>
> I doubt it.
>
> Anyway, I don't understand why you're making such a fuss about this. As
> soon as the offline test data is ready, anyone is free to copy it to a
> real DNS zone of their choosing and exercise their implementation against
> it. (Heck, even _we_ might do it, but who cares if _anybody_ can?)
>
>
> The rest is redundant, or tangential only, but here we go:
>
>> Moreover, it also make the testing process more fallible.
>
> On the contrary. Offline testing eliminates all outside sources of prob-
> lems in order to concentrate on the SPF core functionality. An implemen-
> tation hardly needs an officially approved test suite in order to test
> plain DNS functionality.
>
>> If someone/something doesn't have access to the Internet for an automated
>> test suite, then let them install a local dns server for their test
>> suite.
>
> Right, as if all the implementations were going to add bind9 (or even just
> djbdns) to their build dependencies. No, that's not a good idea at all.
> Besides, you are really missing the fact that you can always load the zone
> data into your own DNS zone.
>
>> Please do not impose this on the implementations.
>
> We're not imposing anything that hasn't been imposed all the time already.
> We're not removing anything from existence.
>
>> Each implementation shouldn't have to construct it's own environment,
>> unless they need to for their own reasons. Imposing this will be a
>> turnoff for developers on future implementations.
>
> I doubt it. Loading zone data into your own zone is trivial (but I'm
> repeating myself), and writing an adapter for your implementation that
> directly reads the offline data should be almost trivial as well.
> Certainly a lot simpler than writing the SPF implementation itself anyway.
>
> I guess you just need to think the situation over once more, then it
> should
> all become clear. ;-)
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
>
> iD8DBQFEzNjDwL7PKlBZWjsRAjNPAJ4oRx2qGUkxLm6gC/hfWbhrPLwifgCgmRCB
> HxNcKUKz9ewyhSsxCA0B3h4=
> =1IAL
> -----END PGP SIGNATURE-----
>
> -------
> To unsubscribe, change your address, or temporarily deactivate your
> subscription,
> please go to
> http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com
>



-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Monte Hansen wrote:
> > What effort to update the DNS for mailzone.com and spftest.org? I
> > know of no such effort.
>
> The records exist to suit the live test cases, and are acknowledged as
> line item 1(c) on the Development Roadmap. This is the only hint on the
> http://new.openspf.org/Test_Suite page that infers an online/offline
> methodology, and that speaks online to me.

It clearly says "old test data" in item 1, besides the two other sets of
old test data are offline test suites, so I find it difficult to follow
your argument.

> To impose an offline style test suite IS more work for everyone than it
> is for an online style, merely to suit a condition that doesn't apply to
> everyone. True?

No. Everyone can run offline tests, but only online people can run online
tests.

> An online methodology means only 1 person takes the hit of setting up
> the live records, for the benefit of a larger audience, no hooks or dns
> drivers needed in the implementation.

True. However note that "there is no we" (TINW). Simply take the offline
data as soon as it's "finished", load it into a public DNS zone of yours,
and announce it here. Then "we" (TINW) can check it for correctness and
declare it "official". ;-)

You see, this is an all volunteers project. Things get done when somebody
does them. Something being "official" means little more than that there
is consensus about it.

Perhaps hosting the test suite online data within the openspf.org zone
would give it an extra touch of officialness, but not much.

> But as a developer I say to you it is considerably more effort to program
> hooks to handle a dns driver, [...]

Well, I guess we'll just have to agree to disagree on that one. :-)

> If you make the zone records live, then the existing dns drivers in place
> will still work -- best of both worlds.

I told you it might happen. If not for a lot of good reasons (which I
don't see), then at least for the fact that doing it is trivial (for a DNS
expert).

No need to crawl into a corner. The project needs people who contribute.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEzPBWwL7PKlBZWjsRAg+QAKCsSHr5+YISTzgGP+xYBYYsrULxBACfaks+
eJ/uOfHoAcU8t6v8ywhpKaY=
=PB6u
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

> True. However note that "there is no we" (TINW). Simply take the offline
> data as soon as it's "finished", load it into a public DNS zone of yours,
> and announce it here. Then "we" (TINW) can check it for correctness and
> declare it "official". ;-)

OK, OK, got the message ;-)

Monte Hansen

"Julian Mehnle" <julian@mehnle.net> wrote in message
news:200607301745.58527.julian@mehnle.net...
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Monte Hansen wrote:
>> > What effort to update the DNS for mailzone.com and spftest.org? I
>> > know of no such effort.
>>
>> The records exist to suit the live test cases, and are acknowledged as
>> line item 1(c) on the Development Roadmap. This is the only hint on the
>> http://new.openspf.org/Test_Suite page that infers an online/offline
>> methodology, and that speaks online to me.
>
> It clearly says "old test data" in item 1, besides the two other sets of
> old test data are offline test suites, so I find it difficult to follow
> your argument.
>
>> To impose an offline style test suite IS more work for everyone than it
>> is for an online style, merely to suit a condition that doesn't apply to
>> everyone. True?
>
> No. Everyone can run offline tests, but only online people can run online
> tests.
>
>> An online methodology means only 1 person takes the hit of setting up
>> the live records, for the benefit of a larger audience, no hooks or dns
>> drivers needed in the implementation.
>
> True. However note that "there is no we" (TINW). Simply take the offline
> data as soon as it's "finished", load it into a public DNS zone of yours,
> and announce it here. Then "we" (TINW) can check it for correctness and
> declare it "official". ;-)
>
> You see, this is an all volunteers project. Things get done when somebody
> does them. Something being "official" means little more than that there
> is consensus about it.
>
> Perhaps hosting the test suite online data within the openspf.org zone
> would give it an extra touch of officialness, but not much.
>
>> But as a developer I say to you it is considerably more effort to program
>> hooks to handle a dns driver, [...]
>
> Well, I guess we'll just have to agree to disagree on that one. :-)
>
>> If you make the zone records live, then the existing dns drivers in place
>> will still work -- best of both worlds.
>
> I told you it might happen. If not for a lot of good reasons (which I
> don't see), then at least for the fact that doing it is trivial (for a DNS
> expert).
>
> No need to crawl into a corner. The project needs people who contribute.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
>
> iD8DBQFEzPBWwL7PKlBZWjsRAg+QAKCsSHr5+YISTzgGP+xYBYYsrULxBACfaks+
> eJ/uOfHoAcU8t6v8ywhpKaY=
> =PB6u
> -----END PGP SIGNATURE-----
>
> -------
> To unsubscribe, change your address, or temporarily deactivate your
> subscription,
> please go to
> http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com
>



-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

I can't say I've closely read this thread, or for that matter, closely
read this mailing list for quite a while.

I will, however, add my voice to the subject of having the test data
in a DNS zone.

I think it is *very* important to have an officially supported test
zone.

First, there are a huge number of implementations that simply can not
be unit-tested. Instead, they can only be tested as part of the
complete MTA. I personally don't agree with this practice at all, and
one of the first things I did to help out the development of SPF was
to create an "spfquery" program for James' libspf so that it could be
tested. That doesn't change the reality though, that you can't test
the majority of SPF implementations via something like "spfquery" or
whatever.

Secondly, Monte is right, requiring the support for offline testing
means more work for everyone. Again, the libspf2 implementation that
I initially wrote includes support for offline testing, so this isn't
a case of "I can't do it, so I don't think it should be done", but
rather that I acknowledge that it requires more work.

Thirdly, there are things that need to be tested that simply can't
easily be done in offline testing, such as making sure that timeouts
are handled correctly, that substrings in a single TXT record are
glued together correctly, that multiple TXT records are handled
correctly, that the SPF implementation has been correctly integrated
into the MTA, that the Received-SPF: header is correctly added, etc.


Personally, I think that the official test system should be online and
that the offline tests should be used as a backup, rather than vice
versa. That said, work in a volunteer project gets done by people who
are actually willing to do the work. If everyone who is interested
in creating test data things that offline stuff is the most important,
then, well, that's what will be developed. If the SPF Council decides
that only the offline test is "official", then I guess that is what
will be official. Future Councils could change that though.



-wayne

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wayne Schlitt wrote:
> [...]
> Thirdly, there are things that need to be tested that simply can't
> easily be done in offline testing, such as making sure that timeouts
> are handled correctly,

I think we have covered that nicely in our current test data schema, so it
should be possible to simulate timeouts provided that the test driver
translates the timeout representation into an internal timeout state
properly.

> that substrings in a single TXT record are glued together correctly,

A very good point, thank you! We need to provide for that. Stuart, what
do you think about allowing the following?

zonedata:
example.com:
- TXT: ['v=spf1 a ', 'mx -all']
- SPF: ['v=spf1 a', 'mx -all']

> that multiple TXT records are handled correctly,

That should be easily testable using our current schema.

> that the SPF implementation has been correctly integrated into the MTA,
> that the Received-SPF: header is correctly added, etc.

Absolutely true, there are things that cannot be tested easily using non-
Turing-complete test data.

> Personally, I think that the official test system should be online and
> that the offline tests should be used as a backup, rather than vice
> versa. That said, work in a volunteer project gets done by people who
> are actually willing to do the work. If everyone who is interested
> in creating test data things that offline stuff is the most important,
> then, well, that's what will be developed.

I really don't get why there's so much debate over whether the test suite
is developed in an "online" or "offline" form. Test data can be easily
converted between zone files and any custom format. And you certainly
wouldn't want to develop the test data directly in the configuration files
of a live DNS server, right? No revision control that way, and other
difficulties...

> If the SPF Council decides that only the offline test is "official", then
> I guess that is what will be official. Future Councils could change that
> though.

Please note that there's a difference between "official" and "authorita-
tive". Every set of data that is being developed needs an _authoritative_
instance for editing purposes and in case there are discrepancies between
instances. However multiple instances can very well be _official_ at the
same time.

Who said that only the offline instance could be "official"? (If I did, I
really meant "authoritative".)

I think that the official instance should be authoritative, though, because
(among other reasons) it can live in a Subversion repository.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEzQHNwL7PKlBZWjsRAlPWAJsEXFThC8m6mrSDFbaHsELHjmoTQQCfVxso
AWGPSEpO8lFT7aKF7WYbSi4=
=4ER4
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Julian Mehnle wrote:
> I think that the official instance should be authoritative, though,
> because (among other reasons) it can live in a Subversion repository.

"I think that the _offline_ instance..."

/me slaps forehead.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEzQKmwL7PKlBZWjsRAnKFAKCwWbP/9nOuxTT7Q10/atvwTfASuQCdF9dE
UGRcsT8tyTUnU9bY9SV5OuI=
=RTjC
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

In <200607301900.29378.julian@mehnle.net> Julian Mehnle <julian@mehnle.net> writes:

> Wayne Schlitt wrote:
>> [...]
>> Thirdly, there are things that need to be tested that simply can't
>> easily be done in offline testing, such as making sure that timeouts
>> are handled correctly,
>
> I think we have covered that nicely in our current test data schema, so it
> should be possible to simulate timeouts provided that the test driver
> translates the timeout representation into an internal timeout state
> properly.

Uh, ok. Well, even the very flexible DNS abstraction system that I
put into libspf2 can't deal with simulating timeouts, so I guess
libspf2 is yet another system that will have to have specialized code
written.


>> that substrings in a single TXT record are glued together correctly,
>> that multiple TXT records are handled correctly,
>
> That should be easily testable using our current schema.

What about EDNS0 and TCP fallbacks? DNS labels being too long? What
about misunderstandings about how real DNS servers work which
propogate into incorrect simulations?

>> that the SPF implementation has been correctly integrated into the MTA,
>> that the Received-SPF: header is correctly added, etc.
>
> Absolutely true, there are things that cannot be tested easily using non-
> Turing-complete test data.

Yes, unit testing works well, and I highly recommend it. It does not
replace integration testing. You simply can not develope a complete
test system by only checking isolated parts.


>> Personally, I think that the official test system should be online and
>> that the offline tests should be used as a backup, rather than vice
>> versa. That said, work in a volunteer project gets done by people who
>> are actually willing to do the work. If everyone who is interested
>> in creating test data things that offline stuff is the most important,
>> then, well, that's what will be developed.
>
> I really don't get why there's so much debate over whether the test suite
> is developed in an "online" or "offline" form.

Hmm... I guess the I thought the reasons were fairly obvious.


> is developed in an "online" or "offline" form. Test data can be easily
> converted between zone files and any custom format. And you certainly
> wouldn't want to develop the test data directly in the configuration files
> of a live DNS server, right? No revision control that way, and other
> difficulties...

You can put zone files under revision control, and, honestly, I don't
see what other difficulties there are. Meng and I managed just fine
doing it that way back when we were working on the SPF test system.

But, as I've said before, this stuff will get done by people doing the
work. When I was working on the test system, I actively maintained
the stuff for M:S:Q, libspf, and libspf2. I suspect that you are
going to support M:S and the python implementation. Neither supported
everyone. Such is life, I guess.


-wayne

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wayne Schlitt wrote:
> You can put zone files under revision control, and, honestly, I don't
> see what other difficulties there are.

E.g. everyone has to shell-log-in to the same server, even for development
purposes. Plus, in order to activate changes, you need to have sufficient
privileges for restarting the name server, which usually means root
access.

> But, as I've said before, this stuff will get done by people doing the
> work. When I was working on the test system, I actively maintained
> the stuff for M:S:Q, libspf, and libspf2. I suspect that you are
> going to support M:S and the python implementation. Neither supported
> everyone. Such is life, I guess.

True, in principle such is life.

But I still don't see the real problem here. You _can_ test against an
online instance of the test suite if you want... *sigh*

Personally, I'm not going to support _any_ implementations when contribu-
ting to the test suite. The suite is supposed to be implementation inde-
pendent. And even if you use it to do _offline_ testing, you can always
skip the tests that you can't easily (or don't want to) support.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEzRIIwL7PKlBZWjsRAr6yAJ4tnLY073Crpnd5a0VDlph72ENKqwCfZ45B
+Sxyo5159HOdLKt+6xQzMuA=
=KD6V
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Sun, 30 Jul 2006, wayne wrote:

> > I really don't get why there's so much debate over whether the test suite
> > is developed in an "online" or "offline" form.
>
> Hmm... I guess the I thought the reasons were fairly obvious.

You can blame me. While I greatly appreciated your test suite, and
it was the inspiration to begin this project, it was no end of trouble
having the test data online. I was rarely able to complete your entire
suite online due to DNS errors. The first step to using it reliably was to
copy it to an offline version.

You are correct that final testing should certainly be done online - because
the software needs to deal with those kind of errors in operation.
However, development is much faster offline.

The intention of the offline form of the test data is that it can be loaded
into a DNS server for online testing also.

> You can put zone files under revision control, and, honestly, I don't
> see what other difficulties there are. Meng and I managed just fine
> doing it that way back when we were working on the SPF test system.

Yes, zone files can be put under revision control, but then the
connection between the zone data and specific tests is very tenous.
That was another problem I ran into while using your test suite.
The tests and corresponding zonedata need to be kept together - like
the examples in the RFC.

My earlier iteration extended the zonefiles with a TEST class to
provide the test cases. Not sure if you would have liked that any
better, but I can generate it again from the YAML format :-)

There *is* an issue that needs to be addressed. Currently, the tests
are written with the expectation that an online version will be able
to append a base domain to queries. This allows resuse of
pedagogical domains such as "example.com". If an implementation can't
handle this, I figured that a programmed tool could statically add a base
domain to the test suite, but I haven't actually done this to verify
that modifying the SPF and TXT records is feasible.

It is important to keep each scenario independent, and not have to coordinate
domain names across the entire test suite. However, a convention might
simplify automatically adding a base domain. For instance, if all
test domains ended in example.com, then a sed replacing example.com with
a new base domain of testid.newbase.com might be all that is required.

> But, as I've said before, this stuff will get done by people doing the
> work. When I was working on the test system, I actively maintained
> the stuff for M:S:Q, libspf, and libspf2. I suspect that you are
> going to support M:S and the python implementation. Neither supported
> everyone. Such is life, I guess.

Other than not providing an online version of the data, was there any
other problem for libspf2? Is the yaml format a problem?

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Sun, 30 Jul 2006, Julian Mehnle wrote:

> A very good point, thank you! We need to provide for that. Stuart, what
> do you think about allowing the following?
>
> zonedata:
> example.com:
> - TXT: ['v=spf1 a ', 'mx -all']
> - SPF: ['v=spf1 a', 'mx -all']
>
> > that multiple TXT records are handled correctly,

I had already thought of that, and was going to suggest it as soon as
I verified it could get loaded in the DNS stub easily.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com