Mailing List Archive: DNS errors in test suite

DNS errors in test suite

Jul 28, 2006, 8:57 AM

Post #1 of 9 (4071 views)

For certain sections of the RFC, the test-suite should simulate a DNS
error. In the pyspf tests, missing names that begin with 'error.' result
in a simulated timeout rather than NXDOMAIN. There could be a more
elegant convention in the YAML format. For instance, a zonedata entry
could be RCODE: 2 (RCODE: n), or TIMEOUT instead of a list of maps.

Here is a proposed example:
# This is the test suite based on RFC4408.
---
comment: |
Initial processing
tests:
longlabel:
spec: 4.3/1
helo: mail.example.net
host: 1.2.3.5
mailfrom: lyme.eater@A12345678901234567890123456789012345678901234567890123.example.com
result: none
nolocalpart:
spec: 4.3/2
helo: mail.example.net
host: 1.2.3.4
mailfrom: @example.net
result: fail
explanation: postmaster
zonedata:
example.com: TIMEOUT
example.net:
- SPF: v=spf1 -all exp=exp.example.net
exp.example.net:
- TXT: %{l}

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: DNS errors in test suite [ In reply to ]

stuart at bmsi

Jul 28, 2006, 9:05 AM

Post #2 of 9 (4015 views)

Permalink

On Fri, 28 Jul 2006, Stuart D. Gathman wrote:

> Here is a proposed example:
> zonedata:
> example.com: TIMEOUT

This is no good. Real servers can respond to TXT query, but not
to SPF query for the same query name. So it has to be something like
this:

zonedata:
example.com:
- TXT: v=spf1 +all
- TIMEOUT # any other types get timeout

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: DNS errors in test suite [ In reply to ]

julian at mehnle

Jul 28, 2006, 10:16 AM

Post #3 of 9 (4005 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> For certain sections of the RFC, the test-suite should simulate a DNS
> error. In the pyspf tests, missing names that begin with 'error.'
> result in a simulated timeout rather than NXDOMAIN. There could be a
> more elegant convention in the YAML format. For instance, a zonedata
> entry could be RCODE: 2 (RCODE: n), or TIMEOUT instead of a list of
> maps.
>
> Here is a proposed example:
> [...]
> zonedata:
> example.com: TIMEOUT
> example.net:
> - SPF: v=spf1 -all exp=exp.example.net
> exp.example.net:
> - TXT: %{l}

Good, let's do it that way.

Stuart D. Gathman wrote:
> Stuart D. Gathman wrote:
> > Here is a proposed example:
> > zonedata:
> > example.com: TIMEOUT
>
> This is no good. Real servers can respond to TXT query, but not
> to SPF query for the same query name. So it has to be something like
> this:
>
> zonedata:
> example.com:
> - TXT: v=spf1 +all
> - TIMEOUT # any other types get timeout

No. You are correct that in reality a server could time out on a FOO query
but not on a BAR query. However we don't have to represent in the test
suite _all_ the situations that can happen in reality. The point of
representing RCODEs and time-outs in the test suite is merely to check for
how implementations map those to SPF record evaluation behavior and SPF
results. This can be done entirely using the scheme you proposed first.

IOW, what information about an implementation's correctness could be gained
from your second scheme that cannot be gained from the first one?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEykZVwL7PKlBZWjsRArOAAJwMqIohoe5v4dnwtG8JL5zFWSVnMQCgxSEG
pP1Q1e5QBPfRX7Z+GcMghWg=
=6ttW
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Re: DNS errors in test suite [ In reply to ]

stuart at bmsi

Jul 28, 2006, 10:33 AM

Post #4 of 9 (4016 views)

Permalink

On Fri, 28 Jul 2006, Julian Mehnle wrote:

> No. You are correct that in reality a server could time out on a FOO query
> but not on a BAR query. However we don't have to represent in the test
> suite _all_ the situations that can happen in reality. The point of
> representing RCODEs and time-outs in the test suite is merely to check for
> how implementations map those to SPF record evaluation behavior and SPF
> results. This can be done entirely using the scheme you proposed first.
>
> IOW, what information about an implementation's correctness could be gained
> from your second scheme that cannot be gained from the first one?

Checking for correct operation when a (braindead but real) DNS server
returns TXT records, but times out on SPF queries.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: DNS errors in test suite [ In reply to ]

julian at mehnle

Jul 28, 2006, 11:18 AM

Post #5 of 9 (4021 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Fri, 28 Jul 2006, Julian Mehnle wrote:
> > No. You are correct that in reality a server could time out on a FOO
> > query but not on a BAR query. However we don't have to represent in
> > the test suite _all_ the situations that can happen in reality. The
> > point of representing RCODEs and time-outs in the test suite is merely
> > to check for how implementations map those to SPF record evaluation
> > behavior and SPF results. This can be done entirely using the scheme
> > you proposed first.
> >
> > IOW, what information about an implementation's correctness could be
> > gained from your second scheme that cannot be gained from the first
> > one?
>
> Checking for correct operation when a (braindead but real) DNS server
> returns TXT records, but times out on SPF queries.

First, the test suite is not meant for testing DNS servers, but for testing
SPF implementations.

But you still _can_ simulate that with your first scheme:

zonedata:
txt.example.com:
- SPF: v=spf1 include:txt.inc.example.com -all
txt.inc.example.com:
- SPF: v=spf1 +all
spf.example.com:
- SPF: v=spf1 include:spf.inc.example.com -all
spf.inc.example.net: TIMEOUT
tests:
mech-lookup-no-timeout:
spec: 5/10/2
mailfrom: @txt.example.com
result: pass
mech-lookup-timeout:
spec: 5/10/2
mailfrom: @spf.example.com
result: temperror

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEylUJwL7PKlBZWjsRAvIhAKDn5roUHQwoqwnSR+VwNMOfqk3BnACgtmU0
HXUZ4C0ODIvikW4jntD+o4k=
=hP3J
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Re: DNS errors in test suite [ In reply to ]

stuart at bmsi

Jul 28, 2006, 11:39 AM

Post #6 of 9 (4024 views)

Permalink

On Fri, 28 Jul 2006, Julian Mehnle wrote:

> > > IOW, what information about an implementation's correctness could be
> > > gained from your second scheme that cannot be gained from the first
> > > one?
> >
> > Checking for correct operation when a (braindead but real) DNS server
> > returns TXT records, but times out on SPF queries.
>
> First, the test suite is not meant for testing DNS servers, but for testing
> SPF implementations.

Exactly. The *implementation* needs to work correctly even when the
DNS server doesn't.

> But you still _can_ simulate that with your first scheme:
>
> zonedata:
> txt.example.com:
> - SPF: v=spf1 include:txt.inc.example.com -all
> txt.inc.example.com:
> - SPF: v=spf1 +all
> spf.example.com:
> - SPF: v=spf1 include:spf.inc.example.com -all
> spf.inc.example.net: TIMEOUT
> tests:
> mech-lookup-no-timeout:
> spec: 5/10/2
> mailfrom: @txt.example.com
> result: pass
> mech-lookup-timeout:
> spec: 5/10/2
> mailfrom: @spf.example.com
> result: temperror

That does *not* simulate the real-life situation that SPF implementation
need to handle. Braindead DNS servers are *very* popular (they come
from a very popular but braindead software company).

The test suite I've committed does address this common situation:

zonedata:
spftimeout.example.net:
- TXT: v=spf1 mx -all
- TIMEOUT

A correct implementation must not return temperror for this setup,
as braindead as it may be on the server side. Not because of any
practical considerations (though, of course they apply too), but
because the RFC demands it. See the comments and section/paragraph
references for the test cases that need TIMEOUT.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: DNS errors in test suite [ In reply to ]

julian at mehnle

Jul 28, 2006, 12:06 PM

Post #7 of 9 (4002 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> That does *not* simulate the real-life situation that SPF implementation
> need to handle. Braindead DNS servers are *very* popular (they come
> from a very popular but braindead software company).
>
> The test suite I've committed does address this common situation:
>
> zonedata:
> spftimeout.example.net:
> - TXT: v=spf1 mx -all
> - TIMEOUT
>
> A correct implementation must not return temperror for this setup,
> as braindead as it may be on the server side. Not because of any
> practical considerations (though, of course they apply too), but
> because the RFC demands it. See the comments and section/paragraph
> references for the test cases that need TIMEOUT.

If you mean to test the response of an implementation that queries for SPF
and TXT records in parallel and only gets the TXT record because the SPF
query times out (or a similar scenario), then you can't, because any
required behavior in this case is simply not specified. An implementation
is not even required to query for SPF RRs. (Oh, what box of Pandora have
I opened now?)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEymBMwL7PKlBZWjsRAhJpAJ4spk0W7vRBUW03MJbcnwd+RTxs4ACg/4lZ
jL356oygucJP3SwiBW8yZYY=
=k/yy
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: DNS errors in test suite [ In reply to ]

julian at mehnle

Jul 28, 2006, 12:22 PM

Post #8 of 9 (4010 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Julian Mehnle wrote:
> If you mean to test the response of an implementation that queries for
> SPF and TXT records in parallel and only gets the TXT record because the
> SPF query times out (or a similar scenario), then you can't, because any
> required behavior in this case is simply not specified. An
> implementation is not even required to query for SPF RRs. (Oh, what box
> of Pandora have I opened now?)

While I'm not sure how this stuff can be put back in Pandora's box, I think
the solution has something to do with specifying certain behavior for
implementation adaptors for the test suite.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEymP8wL7PKlBZWjsRAs1/AKCQsP5oYtzyk/RLrGJnhWLggaqU3wCfSQyY
VNMBZRO/rtE3oDi5bKu7eMs=
=4Joz
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com

Re: Re: DNS errors in test suite [ In reply to ]

stuart at bmsi

Jul 29, 2006, 10:28 AM

Post #9 of 9 (4009 views)

Permalink

On Fri, 28 Jul 2006, Julian Mehnle wrote:

> If you mean to test the response of an implementation that queries for SPF
> and TXT records in parallel and only gets the TXT record because the SPF
> query times out (or a similar scenario), then you can't, because any
> required behavior in this case is simply not specified. An implementation
> is not even required to query for SPF RRs. (Oh, what box of Pandora have
> I opened now?)

My test cases simply reflect what the RFC says. The RFC requires certain
behaviour in case of DNS errors, and the test suite tests for it. I noted
the section and paragraph in the tests. If you don't like it, remember
that you signed off on it too.

You do make one valid point, that conforming implentations may query
TXT records only, or SPF records only. The DNS error edge cases are
mainly concered with querying both. I HIGHLY recommend that any
currently maintained implementations query both, hence the test cases
I provided cover that choice. However, it is not required.

I suggest that sections 4.3 - 4.5 be provided in three profiles. (Hear
the lid of the box beginning to creak.) TXT only, SPF only, and BOTH.
Rather than introduce a profile tag, I would simply have three source
files for the tests.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com