Mailing List Archive

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Mon, 22 May 2006, Stuart D. Gathman wrote:
> > 1) Decide text file format. Suggestions were XAML, JSON, CSV. Bind
> > is not an option because it is tricky to parse and the point is *not*
> > to use a real DNS server. My current vote is JSON - but then I don't
> > know what XAML is or looks like.
>
> I took a good look at XAML. It is more readable than JSON for business
> data, but not for long fields such as SPF records. Furthermore, JSON
> has a parsing library for every language known to machines, whereas XAML
> is limited at the moment.

Who said anything about "XAML"? I think this was a transcription error
from the council meeting where we discussed the test suite. To quote
myself from 2006-05-22:

Julian Mehnle wrote:
| s/XAML/YAML/. http://www.yaml.org/refcard.html describes YAML in itself
| as an example.

> So I still favor JSON, but to be sure, I will convert the existing
> pyspf test data to both formats, and see which I'd rather manually edit.

Can you please take a look at YAML and re-assess your comparison?

> QUESTION:
>
> Do we need to sanitize DNS data in the test suite? On the one hand,
> the data is public, so there should be no problem copying it to our
> test suite. On the other hand, real domains mentioned in the test suite
> might attract extra attacks.

What test data are you talking about? The only domains used by any test
data _I_ know about are spf1-test.mailzone.com and spftest.org.[1]

As we discussed on #spf, I think storing the data in an offline file, as
opposed to online DNS, would be best.

References:
1. http://new.openspf.org/source/project/test-suite/old-data/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEoqk/wL7PKlBZWjsRAqbkAJ0ccVNd8MhI82FVMTNYU6QBterbaACcCl0M
ZY+pUQaMO3zTWUk3fty/s2o=
=ON8T
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wed, 28 Jun 2006, Julian Mehnle wrote:

>
> Who said anything about "XAML"? I think this was a transcription error
> from the council meeting where we discussed the test suite. To quote
> myself from 2006-05-22:

Transcription error on my end. I can't get the name right. I looked
at YAML. After converting existing test data, I am actually leaning
toward CSV now. Actually I am leaning toward Whitespace Separated Values
that happen to be a subset of bind syntax. I.e., no parenthesis or
continuation lines, basically looking like the mailzone.com data posted
by Julian. The subset is as easily parsed as CSV, but can also be directly
loaded into bind if you want to test your library with local network DNS
queries:

80.spf1-test.mailzone.com. IN A 208.210.124.180
80.spf1-test.mailzone.com. IN TXT "v=spf1 a mx exists:%{ir}.%{v}._spf.80.spf1-test.mailzone.com ptr -all"

> > Do we need to sanitize DNS data in the test suite? On the one hand,
> > the data is public, so there should be no problem copying it to our
> > test suite. On the other hand, real domains mentioned in the test suite
> > might attract extra attacks.
>
> What test data are you talking about? The only domains used by any test
> data _I_ know about are spf1-test.mailzone.com and spftest.org.[1]

The test DNS data for the test suite. I have been copying real
problem SPF records to the pyspf test suite data. I have been manually
converting the top domain or so to some variant of example.com to
prevent drawing attacks to any domains mentioned. But this becomes harder
as the number of examples grows.

Julian has a better idea. Let each test case have its own self contained
DNS data. Then, the test cases can be selected, combined, and tweaked
independently without reference to a common DNS dataset.

> As we discussed on #spf, I think storing the data in an offline file, as
> opposed to online DNS, would be best.

There seems to be some misunderstanding. When I say "DNS data" in the
context of the test suite, I do *not* mean querying the actual internet
DNS via a network. But I am refering to some offline subset of the
data available via DNS, sanitized for publishing without drawing attention
to particular domains.

> References:
> 1. http://new.openspf.org/source/project/test-suite/old-data/

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wed, 28 Jun 2006, Stuart D. Gathman wrote:

> toward CSV now. Actually I am leaning toward Whitespace Separated Values
> that happen to be a subset of bind syntax. I.e., no parenthesis or
> continuation lines, basically looking like the mailzone.com data posted
> by Julian. The subset is as easily parsed as CSV, but can also be directly
> loaded into bind if you want to test your library with local network DNS
> queries:
>
> 80.spf1-test.mailzone.com. IN A 208.210.124.180
> 80.spf1-test.mailzone.com. IN TXT "v=spf1 a mx exists:%{ir}.%{v}._spf.80.spf1-test.mailzone.com ptr -all"

Here is a test case from pyspf in the proposed self-contained bind-like
but CSV compatible syntax:

example.co.uk IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
lyme.eater.example.co.uk.1.2.3.5.spf.example.net IN A 127.0.0.1
fail TEST 1.2.3.4 lyme.eater@example.co.uk mail.example.net
pass TEST 1.2.3.5 lyme.eater@example.co.uk mail.example.net

Test algorithm:
1. Initialize empty local DNS database
2. Read each line of test case parsing as CSV with space or tab as separator
3. If line is empty, clear local DNS database (new test case)
4. If 2nd field is "IN", add data to local DNS data. Field positions
are as in bind syntax, but no parenthesis, continuation lines, hex
data, etc, is allowed to remain compatible with CSV parsers.
Name in 1st field is always absolute. (May be extended later.)
4a. if 1st field is empty, repeat previous name
5. If 2nd field is "TEST", execute SPF query. 1st field is expecte
result. 3rd through 5th fields are ip,email,helo. 6th field,
if present, is expected explanation for testing exp modifier.

Comments?

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wed, 28 Jun 2006, Stuart D. Gathman wrote:

> Here is a test case from pyspf in the proposed self-contained bind-like
> but CSV compatible syntax:
>
> example.co.uk IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
> lyme.eater.example.co.uk.1.2.3.5.spf.example.net IN A 127.0.0.1
> fail TEST 1.2.3.4 lyme.eater@example.co.uk mail.example.net
> pass TEST 1.2.3.5 lyme.eater@example.co.uk mail.example.net

Test cases need a comment syntax. I propose that a 2st field of '#'
would introduce a comment. This will be compatible with CSV parsers,
provided quotes match, and is intuitive for those of us familiar with
common comment syntax:

# check exists with basic macro expansion
example.co.uk IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
lyme.eater.example.co.uk.1.2.3.5.spf.example.net IN A 127.0.0.1
fail TEST 1.2.3.4 lyme.eater@example.co.uk mail.example.net
pass TEST 1.2.3.5 lyme.eater@example.co.uk mail.example.net

> Test algorithm:
> 1. Initialize empty local DNS database
> 2. Read each line of test case parsing as CSV with space or tab as separator
> 3. If line is empty, clear local DNS database (new test case)
3a. If 1st field is '#', ignore the line
> 4. If 2nd field is "IN", add data to local DNS data. Field positions
> are as in bind syntax, but no parenthesis, continuation lines, hex
> data, etc, is allowed to remain compatible with CSV parsers.
> Name in 1st field is always absolute. (May be extended later.)
> 4a. if 1st field is empty, repeat previous name
> 5. If 2nd field is "TEST", execute SPF query. 1st field is expecte
> result. 3rd through 5th fields are ip,email,helo. 6th field,
> if present, is expected explanation for testing exp modifier.

Comments?

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wed, 28 Jun 2006, Stuart D. Gathman wrote:

> # check exists with basic macro expansion
> example.co.uk IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
> lyme.eater.example.co.uk.1.2.3.5.spf.example.net IN A 127.0.0.1
> fail TEST 1.2.3.4 lyme.eater@example.co.uk mail.example.net
> pass TEST 1.2.3.5 lyme.eater@example.co.uk mail.example.net

Here is my python code to run test cases. Should be equally simple in
other languages.

...
class SPFTestCase(unittest.TestCase):

def loadTest(self,fname):
global zonedata
zonedata = {}
fp = open(fname,'rb')
for a in csv.reader(fp,delimiter=' '):
if not a:
zonedata = {}
continue
if a[0].startswith('#'): continue
cmd = a[1].upper()
if cmd == 'IN':
#example.com IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
domain,t = a[0],a[2].upper()
if domain.endswith('.'):
domain = domain[:-1]
if t == 'MX':
v = t,(int(a[3]),a[4])
else:
v = t,a[3]
zonedata.setdefault(domain,[]).append(v)
elif cmd == 'TEST':
#fail TEST 1.2.3.4 lyme.eater@example.co.uk mail.example.net
i,s,h = a[2:5]
res = a[0]
q = spf.query(i=i, s=s, h=h)
self.assertEqual(q.check()[0],res)
# FIXME: check explanation if present

fp.close()

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> After converting existing test data, I am actually leaning toward CSV
> now. Actually I am leaning toward Whitespace Separated Values that
> happen to be a subset of bind syntax. I.e., no parenthesis or
> continuation lines, basically looking like the mailzone.com data posted
> by Julian. The subset is as easily parsed as CSV, but can also be
> directly loaded into bind if you want to test your library with local
> network DNS queries: [...]

Before we define the test data format, I think we should define _what_
data each test should contain in the first place. The examples on the
annotatable spec[1] are a first start, but I haven't completely thought it
through yet.

As Stuart said, I think that tests should use generally distinct DNS name
spaces for test records. However I also think that it should be possible
to share sets of test records between multiple tests. That leads us to
sets of test records having to have unique IDs, either _explicit_ or
_implicit_ ones (using a simple RFC2822-header style for merely expository
purposes):

Explicit:

Record-Set-ID: foo-records
Record: example.org. IN SPF "v=spf1 a -all"
Record: example.org. IN A 1.2.3.4

Test-ID: foo1-test
Record-Set: foo-records
...

Test-ID: foo2-test
Record-Set: foo-records
...

Implicit:

Record: example.org. IN SPF "v=spf1 a -all"
Record: example.org. IN A 1.2.3.4

Test-ID: foo1-test
# uses preceding record set
...

Test-ID: foo2-test
# uses preceding record set, too
...

The implicit style (where the preceding record set is always used) might be
problematic for tests where _no_ records should be defined.

Further, each test obviously must contain:

* an IP-address to be tested (w.l.o.g., MAIL FROM can be fixed to
"example.com", and HELO to "mta.example.com"),
* the expected result.

Besides that, I think it should contain:

* a definite reference to the part(s) of the spec that is (are) being
tested,
* at least a _concise_ comment that can be used as diagnostic output if
the test fails,
* possibly a longer comment that serves as full documentation for the
test.

Anything else?

References:
1. http://new.openspf.org/Test_Suite/Annotatable_Specification

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEouqNwL7PKlBZWjsRAhbNAKCl2w68lMs/PRp5vGfvuCjOmLtN9wCg53Aa
VPcTiBVSl73BcO7xQLK6Niw=
=TsUV
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wednesday 28 June 2006 16:46, Julian Mehnle wrote:

> Anything else?
>
I think it would be really cool to encode DNS data in XML ;):

http://www.microsoft.com/downloads/details.aspx?FamilyID=9a9e8a28-3e85-4d07-9d0f-6daeabd3b71b&displaylang=en

or

http://tinyurl.com/mo5jb

Scott K

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Julian Mehnle wrote:
> As Stuart said, I think that tests should use generally distinct DNS
> name spaces for test records. However I also think that it should be
> possible to share sets of test records between multiple tests. That
> leads us to sets of test records having to have unique IDs, either
> _explicit_ or _implicit_ ones (using a simple RFC2822-header style for
> merely expository purposes):
> [...]
> The implicit style (where the preceding record set is always used) might
> be problematic for tests where _no_ records should be defined.

Another problem with the implicit referencing style is that, in order to
actually share record sets, you have to group tests by shared record sets,
not by position in the spec or other critieria.

The problem with the explicit style is that it allows tests to refer to
record sets at the other end of the file, so when modifying some record
set you couldn't easily tell what tests will be affected (except by
searching for the record set's ID, of course).

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEou84wL7PKlBZWjsRAj+TAKDxWrmWC4VIbfOqIM1+FytUh1TNuACg9YV9
DzlOdDhSwsfFxXSNuAKZ6vs=
=DpC0
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wed, 28 Jun 2006, Julian Mehnle wrote:

> As Stuart said, I think that tests should use generally distinct DNS name
> spaces for test records. However I also think that it should be possible
> to share sets of test records between multiple tests. That leads us to
> sets of test records having to have unique IDs, either _explicit_ or
> _implicit_ ones (using a simple RFC2822-header style for merely expository
> purposes):

The proposed bind-like syntax supports that. The example test set has
two tests sharing the same DNS data. What does having record set ids
accomplish? It would be nice to have test ids for a shorthand way
to refer to failing tests. Perhaps the test-id should be the 1st field
for test records.

> Explicit:
>
> Record-Set-ID: foo-records
> Record: example.org. IN SPF "v=spf1 a -all"
> Record: example.org. IN A 1.2.3.4
>
> Test-ID: foo1-test
> Record-Set: foo-records
> ...
>
> Test-ID: foo2-test
> Record-Set: foo-records
> ...

The proposed format would have all DNS data sharing be implicit.

example.org. IN SPF "v=spf1 a -all"
example.org. IN A 1.2.3.4
#
foo1-test TEST ...
foo2-test TEST ...

> The implicit style (where the preceding record set is always used) might be
> problematic for tests where _no_ records should be defined.

The proposed format resets the DNS data on blank lines.

> Besides that, I think it should contain:
>
> * a definite reference to the part(s) of the spec that is (are) being
> tested,
> * at least a _concise_ comment that can be used as diagnostic output if
> the test fails,
> * possibly a longer comment that serves as full documentation for the
> test.

I think that the test-id serves as the concise comment. The longer
comment would consist of the test of the entire testcase.

Example of new proposed format (1st field of TEST is now testid):

# check basic exists with macro expansion
example.co.uk IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
lyme.eater.example.co.uk.1.2.3.5.spf.example.net IN A 127.0.0.1
exists1 TEST 1.2.3.4 lyme.eater@example.co.uk mail.example.net fail
exists2 TEST 1.2.3.5 lyme.eater@example.co.uk mail.example.net pass


--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wed, 28 Jun 2006, Julian Mehnle wrote:

> Another problem with the implicit referencing style is that, in order to
> actually share record sets, you have to group tests by shared record sets,
> not by position in the spec or other critieria.

Wouldn't tagging tests with spec references be sufficient when using
implicit referencing?

> The problem with the explicit style is that it allows tests to refer to
> record sets at the other end of the file, so when modifying some record
> set you couldn't easily tell what tests will be affected (except by
> searching for the record set's ID, of course).

I hate the explicit style. I'd rather repeat the test data.

However, the issue of adding tags points to the advantage of a more
general format like YAML or JSON. Additional tags could be defined later.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> The proposed bind-like syntax supports that. The example test set has
> two tests sharing the same DNS data.

Understood and granted.

> What does having record set ids accomplish?

Well, explicit referencing of record sets. Nothing else.

> I think that the test-id serves as the concise comment.

That's not at all what I had in mind. I thought more along the line of the
comments of the example tests in section 4.5[1] of the annotatable spec,
i.e. "Invalid version string", "SPF RR type overrides TXT RR type", "More
than one record applicable". Test IDs can hardly be as descriptive as a
short but free-text comment.

> The longer comment would consist of the test of the entire testcase.

I can see having test documentation as simple "#" comments in the file.
That would be unstructured, though, and thus more difficult to extract
automatically.

> > Another problem with the implicit referencing style is that, in order
> > to actually share record sets, you have to group tests by shared
> > record sets, not by position in the spec or other critieria.
>
> Wouldn't tagging tests with spec references be sufficient when using
> implicit referencing?

The term "implicit/explicit referencing", as I meant it, applies to the
tagging of record sets, not tests. But I can see embedding spec
references in test IDs and saving an extra "spec reference" item.

> > The problem with the explicit style is that it allows tests to refer
> > to record sets at the other end of the file, so when modifying some
> > record set you couldn't easily tell what tests will be affected
> > (except by searching for the record set's ID, of course).
>
> I hate the explicit style. I'd rather repeat the test data.

Agreed. I don't really like the explicit style.

> However, the issue of adding tags points to the advantage of a more
> general format like YAML or JSON. Additional tags could be defined
> later.

...see below.

> Example of new proposed format (1st field of TEST is now testid):
>
> # check basic exists with macro expansion
> example.co.uk IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
> lyme.eater.example.co.uk.1.2.3.5.spf.example.net IN A 127.0.0.1
> exists1 TEST 1.2.3.4 lyme.eater@example.co.uk mail.example.net fail
> exists2 TEST 1.2.3.5 lyme.eater@example.co.uk mail.example.net pass

Well, I must say I don't like this generalization of the BIND format at
all. Sure, let's keep test records in BIND format, but having all the
other test data in this same format as well seems a great bit too implicit
for my taste. It's hard to read if one doesn't know the test data fields
and field order by heart and thus invites mistakes in understanding the
tests, if not in modifying them. Also, as you pointed out, such a format
would be hard to extend.

Given that we want to use a format that supports structure, essentially
XML, JSON, and YAML remain.

I know little about JSON, so I'll compile a realistic example in YAML
tomorrow. Stuart, could you do one in JSON so we all see how that looks?
(I'm sure we all can imagine XML -- XML's advantages are others than
human readability, and I think we should consider them.)

References:
1. http://new.openspf.org/Test_Suite/Annotatable_Specification#version

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEowRqwL7PKlBZWjsRApyZAJ0VgM5XTItY38hYxQK7190/Ed3HJgCeNMx6
DA4zUH2ZbhcHfs+mB0p4VFs=
=VQte
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

On Wed, 28 Jun 2006, Julian Mehnle wrote:

> I know little about JSON, so I'll compile a realistic example in YAML
> tomorrow. Stuart, could you do one in JSON so we all see how that looks?
> (I'm sure we all can imagine XML -- XML's advantages are others than
> human readability, and I think we should consider them.)

Working on JSON example, but here is one more go at the extended bind.
It is now extensible and more like DNS:

; check basic exists with macros
example.co.uk IN SPF "v=spf1 mx/26 exists:%{l}.%{d}.%{i}.spf.example.net -all"
lyme.eater.example.co.uk.1.2.3.5.spf.example.net IN A 127.0.0.1
;
exists-fail TEST IP4 1.2.3.4
TEST MFROM lyme.eater@example.co.uk
TEST HELO mail.example.net
TEST RESULT fail
;
exists-pass TEST IP4 1.2.3.5
TEST MFROM lyme.eater@example.co.uk
TEST HELO mail.example.net
TEST RESULT pass

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com