Mailing List Archive: Test suite schema

Test suite schema

Jul 28, 2006, 11:56 AM

Post #1 of 7 (3039 views)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I do have a few concerns about the test suite schema[1]:

* Let's use descriptive test names, i.e. "trailing-dot-is-acceptable"
instead of "traildot1". Even if we're conservative, we'll have a lot
of tests. Let's be precise (not overly, though) when making spec
references ("8.1/16" vs "8.1"). And let's not be overly redundant when
wording test descriptions, i.e. avoid saying "Check that..." every
time.

* There's usually no reason to use _literal_ YAML blocks[2] for comments.
Instead, one would usually use _folded_ blocks[3], and thus the > (or
>-) style indicator instead of the | (or |-) indicator. I fixed that
in the informal schema sample already.

* Let's call the comment field "description" instead of "comment", and
use "comment" only for non-essential comments about the item in
question. I.e.:

tests:
domain-spec-trailing-dot-is-acceptable:
spec: 8.1/16
comment: a trailing dot in a domain-spec must be accepted
...
explanation-trailing-dot-is-preserved:
spec: 8.1
comment: >-
a trailing dot in an explanation string must be preserved
...
explanation: This is a test.

* I have been thinking about the various identities that can be checked
by SPF and S-ID (and possibly later SPF revisions). Currently, there
are at least HELO, MAIL FROM, and PRA. I think we should allow each
test to check any one of those identities. What I did in Mail::SPF is
allow two basic parameters to be specified: the scope and the identity
value. The scopes would be named after the scope tags they would use
in a real SPF record, i.e. "helo", "mfrom", "pra", etc. If the scope
is omitted, it would default to "mfrom" In addition to that, any
parameters relevant to the given scope should be specified ("host" for
all the currently defined scopes). I.e.:

tests:
helo-test:
scope: helo
helo: mta.example.com
host: 10.0.0.1
result: ...
...
mfrom-test:
scope: mfrom # optional, as "mfrom" is the default
mfrom: joe@example.com
host: 10.0.0.1
result: ...
...

I think this scheme is still very simple but significantly more
versatile.

References:
1. http://new.openspf.org/Test_Suite/Schema
2. http://yaml.org/spec/current.html#id2540046
3. http://yaml.org/spec/current.html#id2540863

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEyl36wL7PKlBZWjsRAgXtAKCz+FJ82LZScPrBhT6gtV6lbaLmRgCdFNbs
ZMoHgaeuLYYFeGfF3KsxX6w=
=HTb4
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Test suite schema [ In reply to ]

julian at mehnle

Jul 28, 2006, 12:10 PM

Post #2 of 7 (2954 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Julian Mehnle wrote:
> I do have a few concerns about the test suite schema[1]:
> [...]
> * Let's call the comment field "description" instead of "comment", and
> use "comment" only for non-essential comments about the item in
> question. I.e.:
>
> tests:
> domain-spec-trailing-dot-is-acceptable:
> spec: 8.1/16
> comment: a trailing dot in a domain-spec must be accepted
> ...
> explanation-trailing-dot-is-preserved:
> spec: 8.1
> comment: >-
> a trailing dot in an explanation string must be preserved
> ...
> explanation: This is a test.

Sorry, that was supposed to read:

tests:
domain-spec-trailing-dot-is-acceptable:
spec: 8.1/16
description: a trailing dot in a domain-spec must be accepted
comment: (some secondary comments about the test case)
...
explanation-trailing-dot-is-preserved:
spec: 8.1
description: >-
a trailing dot in an explanation string must be preserved
comment: >-
(some lengthy secondary comments about the test case...
...foo, bar, zip)
...
explanation: This is a test.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEymEswL7PKlBZWjsRAr3aAJ98gfeU80QMsh488sE002rgKhM9CACg1E4Y
37/GlVTf44Cx6Qb5xSIMaBs=
=cREH
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Test suite schema [ In reply to ]

stuart at bmsi

Jul 29, 2006, 10:38 AM

Post #3 of 7 (2938 views)

Permalink

On Fri, 28 Jul 2006, Julian Mehnle wrote:

> * Let's use descriptive test names, i.e. "trailing-dot-is-acceptable"
> instead of "traildot1". Even if we're conservative, we'll have a lot
> of tests. Let's be precise (not overly, though) when making spec
> references ("8.1/16" vs "8.1"). And let's not be overly redundant when
> wording test descriptions, i.e. avoid saying "Check that..." every
> time.

Agree. How about you select some samples from rfc4408-tests.yml for the
informal schema to replace my pyspf samples? Revise the test ids to
reflect this style.

+1

> * There's usually no reason to use _literal_ YAML blocks[2] for comments.
> Instead, one would usually use _folded_ blocks[3], and thus the > (or
> >-) style indicator instead of the | (or |-) indicator. I fixed that
> in the informal schema sample already.

I used '|' because it looks prettier. I am not a YAML expert yet.
Whatever you say. Does it hurt to use |?

0

> * Let's call the comment field "description" instead of "comment", and
> use "comment" only for non-essential comments about the item in
> question. I.e.:

Renaming comment -> description is no problem. Why do you want two
levels? Are you thinking of some form of automated directory or index with
short vs. long entries? We already have lexical comments, which are ignored
entirely.

?0

> * I have been thinking about the various identities that can be checked
> by SPF and S-ID (and possibly later SPF revisions). Currently, there
> are at least HELO, MAIL FROM, and PRA. I think we should allow each

The current schema already checks HELO in the v=spf1 style. Just make
mailfrom empty. While a scope field will be handy for SPF2.5, I do
not want to add it for the v=spf1 test suite.

I vote NO on the scope field for v=spf1 test suite.

-1

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Test suite schema [ In reply to ]

julian at mehnle

Jul 29, 2006, 12:04 PM

Post #4 of 7 (2958 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Scott Kitterman wrote:
> How do you derive the correct result for a scope PRA without using
> Microsoft's patent pending algorithm? IANAL, so I don't know. I'd hate
> to have US based test suite contributors have to get a license from
> Microsoft....

The test suite doesn't have anything to do with the PRA algorithm itself.
It is _implementations_ that have to implement it. Implementations that
don't support PRA can easily ignore the "scope: pra" tests (if we create
any in the first place).

You make a good point with regard to Mail::SPF, though. Personally, I have
always been convinced that the PRA patent is (1) trivial, and (2) covered
by prior art. But I might make Mail::SPF modular enough so that PRA
support can, and _has_ to be, plugged in using a module distributed sepa-
rately (or not at all).

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEy7FHwL7PKlBZWjsRAo3LAKCd327ERK51m1pPcnp283tUj32CUQCfZhxQ
SwhxmIvxaTkXQbMVQF5GzX4=
=icac
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Test suite schema [ In reply to ]

julian at mehnle

Jul 29, 2006, 12:06 PM

Post #5 of 7 (2950 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Fri, 28 Jul 2006, Julian Mehnle wrote:
> > * Let's use descriptive test names, i.e. "trailing-dot-is-acceptable"
> > instead of "traildot1". Even if we're conservative, we'll have a
> > lot of tests. Let's be precise (not overly, though) when making
> > spec references ("8.1/16" vs "8.1"). And let's not be overly
> > redundant when wording test descriptions, i.e. avoid saying "Check
> > that..." every time.
>
> Agree. How about you select some samples from rfc4408-tests.yml for the
> informal schema to replace my pyspf samples? Revise the test ids to
> reflect this style.

Will do.

> > * There's usually no reason to use _literal_ YAML blocks[2] for
> > comments. Instead, one would usually use _folded_ blocks[3], and
> > thus the > (or >-) style indicator instead of the | (or |-)
> > indicator. I fixed that in the informal schema sample already.
>
> I used '|' because it looks prettier. I am not a YAML expert yet.
> Whatever you say. Does it hurt to use |?

foo: |
blah
blah

means "blah\nblah\n" (literal block style[1]).

foo: |-
blah
blah

means "blah\nblah" (literal block style with stripping[3]).

foo: >
blah
blah

means "blah blah\n" (folded block style[2]).

foo: >-
blah
blah

means "blah blah" (folded block style with stripping).

I think the literal block style is not generally suited for our uses of
free-form text. We're mostly using blocks for overly long values,
especially for long TXT/SPF records.

> > * Let's call the comment field "description" instead of "comment",
> > and use "comment" only for non-essential comments about the item in
> > question. I.e.:
>
> Renaming comment -> description is no problem.

Ok, let's do it.

> Why do you want two levels? Are you thinking of some form of automated
> directory or index with short vs. long entries? We already have lexical
> comments, which are ignored entirely.

You make a good point with regard to lexical comments. Those are not
extractable in line with the logical structure of the document, though.
If we decide we don't need that, then I'm fine with using lexical comments
for additional comments beyond the short descriptions. I was thinking of
a verbose mode of testing, though, where those additional (optional!)
comments would be output in order to explain tricky test cases.

> > * I have been thinking about the various identities that can be
> > checked by SPF and S-ID (and possibly later SPF revisions).
> > Currently, there are at least HELO, MAIL FROM, and PRA. I think we
> > should allow each
>
> The current schema already checks HELO in the v=spf1 style. Just make
> mailfrom empty. While a scope field will be handy for SPF2.5, I do
> not want to add it for the v=spf1 test suite.
>
> I vote NO on the scope field for v=spf1 test suite.

AFAICT, my proposal is backwards compatible with the current schema,
i.e. "scope" would default to "mfrom" if absent. It would probably be a
good idea, however, to take the proposed scheme into account when writing
tools, i.e. read the "scope" field if it exists and die if != "mfrom",
etc.

Please note that I devised this extended scheme not primarily with PRA in
mind, but for things such as PGP, S/MIME, or DKIM.

References:
1. http://yaml.org/spec/current.html#id2540046
2. http://yaml.org/spec/current.html#id2540863
3. http://yaml.org/spec/current.html#id2538656

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEy7G/wL7PKlBZWjsRAjcYAJ4wcR0EpEWGHiEL69990f3urMlPhgCghUTb
ux5xdNtHAPlhiEH+DlraYuE=
=kKHP
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Test suite schema [ In reply to ]

scott at kitterman

Jul 29, 2006, 12:22 PM

Post #6 of 7 (2948 views)

Permalink

On Sat, 29 Jul 2006 13:38:57 -0400 (EDT) "Stuart D. Gathman"
<stuart@bmsi.com> wrote:
>On Fri, 28 Jul 2006, Julian Mehnle wrote:
>> * I have been thinking about the various identities that can be checked
>> by SPF and S-ID (and possibly later SPF revisions). Currently, there
>> are at least HELO, MAIL FROM, and PRA. I think we should allow each
>
>The current schema already checks HELO in the v=spf1 style. Just make
>mailfrom empty. While a scope field will be handy for SPF2.5, I do
>not want to add it for the v=spf1 test suite.
>
>I vote NO on the scope field for v=spf1 test suite.
>
>-1
>
+1 to Stuart's -1 (I don't like it either).

One other point...

How do you derive the correct result for a scope PRA without using
Microsoft's patent pending algorithm? IANAL, so I don't know. I'd hate to
have US based test suite contributors have to get a license from
Microsoft....

Scott K

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Re: Test suite schema [ In reply to ]

scott at kitterman

Jul 29, 2006, 2:27 PM

Post #7 of 7 (2946 views)

Permalink

On Sat, 29 Jul 2006 19:04:39 +0000 Julian Mehnle <julian@mehnle.net> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Scott Kitterman wrote:
>> How do you derive the correct result for a scope PRA without using
>> Microsoft's patent pending algorithm? IANAL, so I don't know. I'd hate
>> to have US based test suite contributors have to get a license from
>> Microsoft....
>
>The test suite doesn't have anything to do with the PRA algorithm itself.
>It is _implementations_ that have to implement it. Implementations that
>don't support PRA can easily ignore the "scope: pra" tests (if we create
>any in the first place).
>
>You make a good point with regard to Mail::SPF, though. Personally, I have
>always been convinced that the PRA patent is (1) trivial, and (2) covered
>by prior art. But I might make Mail::SPF modular enough so that PRA
>support can, and _has_ to be, plugged in using a module distributed sepa-
>rately (or not at all).
>
Trivial or not those of us in the US may be stuck with it for a while.

I should have also mentioned that it applies to distributors as well as
developers, so yes, particularly for Mail::SPF.

Scott sk

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com