Mailing List Archive

Stuart D. Gathman ha scritto:
> On Tue, 19 Aug 2008, Stefano Bagnara wrote:
>
>> Having an external testsuite allow you to check bugs in the testsuites too.
>> Often people that wrote the code is the same that wrote the tester code, this
>> can result in obvious bugs to be left there because of misunderstandings from
>> the author.
>
> I have already put running your tester on my TODO list (although I don't
> know when I'll get to it). Effective testing is an area of software
> that is often ignored because of its difficulty once you get beyond
> the simple input/process/output library routine.
>
>> BTW I don't care if you will or won't use the jSPF tester once we'll publish
>> it, but I'm a bit annoyed that OpenSPF page keep saying that the only
>> compliant implementations are pyspf 2 and Mail::SPF only because OpenSPF
>> members wrote that while jSPF has an "the jSPF library is currently being
>> evaluated by the project for RFC 4408 compliance" since years. At the end of
>> 2006 jSPF was listed as "The jSPF library is also reported to pass all tests
>> in the test suite, however this has not yet been officially confirmed" and
>> then on 2006-12-10 you removed the whole sentence. On 2007-02-01 then the
>> "currently being evaluated" has been written there... 18 months passed since
>> that...
>
>> IMHO it is bad for OpenSPF to fail to tell how implementations can be listed
>> between compliant implementation and ignore anyone but your own
>> implementations.
>
> Agreed. But I'm pretty sure it is due to not having come up with an
> actual procedure rather than self promotion. Maybe you could volunteer
> to test implementations using your live DNS test framework? That would
> at least get yours done :-)

I'm already testing jSPF with my tester :-)

My effort to make the tester "implementation agnostic" is really to gain
some more trust from you when you see that my tester correctly check
your implementations.

It is OpenSPF that have to tell how an implementation is certified. What
tests have Mail::SPF and pyspf 2.0 passed in order to be listed there??
I bet jSPF passed the same tests.

In fact we always had selftests based on the yaml file you publish and
we have some more unit test in place to increase the coverage, too.

I'd happily complete the live dns tester tool but in order to check
implementations they have to return an "spfquery" like result, 4 lines
where the 1st is the result, the 2nd is the explanation, the 4th is the
Received-SPF: header. Then they also have the ability to use a specific
dns server for their queries (e.g: --dnsserver|-s <IP>[:PORT]).
I can make parameters configurable, but that dnsserver option is needed
in order to run my tester. AFAIK no implementation currently support
this (jSPF will support this in the next release).

Using the "commandline interface" is the only way I found to test
different implementations using a single tester.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Tue, 19 Aug 2008, Stefano Bagnara wrote:

> My effort to make the tester "implementation agnostic" is really to gain some
> more trust from you when you see that my tester correctly check your
> implementations.

What about it guys? I would "certainly" vote for jSPF as "certified", but what
about the next implementation that comes up? What is the procedure? Do
we elect a review panel? Bring back the council? Have a general vote of
everyone on spf-devel? Did we have a procedure written, but forgot about it?

> Using the "commandline interface" is the only way I found to test different
> implementations using a single tester.

The test suite is certainly a critical component, but how do we determine that
it is properly applied? We don't have a volunteer to do the testing
independently. Stefano's suggestion to offer a test via the spfquery
interface and his live DNS framework is an excellent solution. I can trivially
add a DNS port option to spfquery in pyspf. That will help define the precise
spfquery requirements (which are something of a vague convention at this
point).

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Tue, 19 Aug 2008, Stuart D. Gathman wrote:

> What about it guys? I would "certainly" vote for jSPF as "certified", but
> what about the next implementation that comes up? What is the procedure? Do
> we elect a review panel? Bring back the council? Have a general vote of
> everyone on spf-devel? Did we have a procedure written, but forgot about it?

Maybe any implementation that passes the test suite should be conditionally
certified - with the person(s) or organization doing the test listed
as part of the certification. That is probably good enough. There are
already lots of ways to screw up SPF even if you get all the
ip,mfrom,helo + DNS -> result mappings correct. (Like the every popular
"check behind an MX".)

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Tue, 19 Aug 2008, Stuart D. Gathman wrote:
> On Tue, 19 Aug 2008, Stuart D. Gathman wrote:
> > What about it guys? I would "certainly" vote for jSPF as "certified", but
> > what about the next implementation that comes up? What is the procedure? Do
> > we elect a review panel? Bring back the council? Have a general vote of
> > everyone on spf-devel? Did we have a procedure written, but forgot about it?
>
> Maybe any implementation that passes the test suite should be conditionally
> certified - with the person(s) or organization doing the test listed
> as part of the certification. That is probably good enough. There are
> already lots of ways to screw up SPF even if you get all the
> ip,mfrom,helo + DNS -> result mappings correct. (Like the every popular
> "check behind an MX".)

I really do not carry much weight but I agree with this. I think it is
the right thing to do (TM)[or what ever is put for a common saying]. I
like this idea and I would really like all the cerified to have such a
statement/feature.

just my 2 cents.

--
Boyd Gerber <gerberb@zenez.com>
ZENEZ 1042 East Fort Union #135, Midvale Utah 84047


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Stuart D. Gathman wrote:
> Maybe any implementation that passes the test suite should be
> conditionally certified - with the person(s) or organization doing
> the test listed as part of the certification.

For actually running the tests, one still needs to know the tuples
required for each test. Those fields have to be in some order, e.g.

mailfrom, tcpremoteip, helodomain, mydomain.

SPF implementations integrated in an MTA may take "tcpremotehost" as a
parameter, in case the origin has been validated already when the SPF
check starts. In that case the test function should link with the
relevant function in order to obtain the validation for test cases.
(It is not _exactly_ as it would be in real life, but acceptable.)

I'm thinking about Courier. BTW, it has its own testsuite, see e.g.
http://www.koders.com/cpp/fidAF68AE4ACFB883D5D3585D26D35899C84E7502AA.aspx?s=md5

To add a --dnsserver|-s <IP>[:PORT] option to the test function is
even easier.

> That is probably
> good enough. There are already lots of ways to screw up SPF even
> if you get all the ip,mfrom,helo + DNS -> result mappings correct.
> (Like the every popular "check behind an MX".)

Agreed, the real concern is to discover actual bugs.

IMHO, the best thing would be to have those tuples published on a web
page. If it were also possible to post the results in order to have a
response, much like html validation pages, that would be great!

In the latter case, if the submitter authenticates and specifies name
and version of the implementation, it will be trivial to build a
database of verified implementations, along with the date and test
suite version.

Stefano Bagnara wrote:
> I'd happily complete the live dns tester tool but in order to check
> implementations they have to return an "spfquery" like result, 4
> lines where the 1st is the result, the 2nd is the explanation, the
> 4th is the Received-SPF:

What's the 3rd line?

Validating Received-SPF can be slightly more difficult, since order
and indentation may vary.

An empty line to separate one test from the next might suffice;
however, repeating the tuple as a key is more robust and allows
results to be reviewed even after new tests have been inserted,
provided the DNS won't vary.



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Stuart D. Gathman ha scritto:
> > Agreed. But I'm pretty sure it is due to not having come up with an
> > actual procedure rather than self promotion. Maybe you could
> > volunteer to test implementations using your live DNS test framework?
> > That would at least get yours done :-)
>
> I'm already testing jSPF with my tester :-)
>
> My effort to make the tester "implementation agnostic" is really to
> gain some more trust from you when you see that my tester correctly
> check your implementations.

Testing implementations from an additional angle is good; any disparity
between testing results from the different angles is likely to help
exposing bugs. The reason that I, personally, haven't invested the
necessary time to implement an online-DNS tester yet is simply that I
felt the benefit to be expected wouldn't justify the (time) cost. But if
you're willing to set this up, please go for it!

> It is OpenSPF that have to tell how an implementation is certified.
> What tests have Mail::SPF and pyspf 2.0 passed in order to be listed
> there?? I bet jSPF passed the same tests.

You have to pass the RFC 4408 test suite that can be downloaded from

http://www.openspf.org/Test_Suite

More precisely, you have to convince someone who is able to edit the SPF
project's website that the implementation passes the test suite.

> In fact we always had selftests based on the yaml file you publish and
> we have some more unit test in place to increase the coverage, too.

That's good, but unfortunately it was too difficult (for me, who had first
given it a try) to verify that. See my previous mail.

> I'd happily complete the live dns tester tool but in order to check
> implementations they have to return an "spfquery" like result, 4 lines
> where the 1st is the result, the 2nd is the explanation, the 4th is the
> Received-SPF: header. Then they also have the ability to use a specific
> dns server for their queries (e.g: --dnsserver|-s <IP>[:PORT]).
> I can make parameters configurable, but that dnsserver option is needed
> in order to run my tester. AFAIK no implementation currently support
> this (jSPF will support this in the next release).

Can we instead agree instead on having a public DNS zone delegated to your
server and have it serve test records from there? That seems cleaner to
me than forcing implementations to use a specific resolver server.

> Using the "commandline interface" is the only way I found to test
> different implementations using a single tester.

Or you could use the spfd interface: pipe test data into STDIN, get
results from STDOUT. See http://search.cpan.org/dist/Mail-SPF/sbin/spfd
for a good documentation of that interface. (spfd uses a TCP or UNIX
socket, but you could just as well implement the interface using a pipe.)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkir6ykACgkQwL7PKlBZWjtv9ACgpcJ/E6TMzLBe9OpQLFaYkZ3p
v5sAoO7cFoqQ5DZmK7A1Jn6iDrPnMVWL
=xd8F
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alessandro Vesely wrote:
> I'm thinking about Courier. BTW, it has its own testsuite, see e.g.
> http://www.koders.com/cpp/fidAF68AE4ACFB883D5D3585D26D35899C84E7502AA.aspx?s=md5

That looks a bit too simplistic for a proper test suite.

> IMHO, the best thing would be to have those tuples published on a web
> page. If it were also possible to post the results in order to have a
> response, much like html validation pages, that would be great!
>
> In the latter case, if the submitter authenticates and specifies name
> and version of the implementation, it will be trivial to build a
> database of verified implementations, along with the date and test
> suite version.

Given that mostly just library implementations will be able to be tested
systematically (not on principal grounds, but let's just wait and see how
many non-library implementations, such as MTA plugins, come up with a
command-line testing interface!), I don't see how the effort required to
implement the system you suggest could be justified by the few
implementations that would be using it. Do you want to do it?

> Stefano Bagnara wrote:
> > I'd happily complete the live dns tester tool but in order to check
> > implementations they have to return an "spfquery" like result, 4
> > lines where the 1st is the result, the 2nd is the explanation, the
> > 4th is the Received-SPF:
>
> What's the 3rd line?

The 2nd line is the authority domain's explanation (via the exp= modifier)
if available, and the local explanation (generated by the SPF client)
otherwise.

The 3rd line is always the local explanation.

> Validating Received-SPF can be slightly more difficult, since order
> and indentation may vary.

Indeed. This probably isn't something that the official test suite's YAML
file can check. The best we could do is provide an algorithm (and perhaps
command-line tool) for checking it.

> An empty line to separate one test from the next might suffice;
> however, repeating the tuple as a key is more robust and allows
> results to be reviewed even after new tests have been inserted,
> provided the DNS won't vary.

Please check out the spfd interface, it probably fits:

http://search.cpan.org/dist/Mail-SPF/sbin/spfd

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkir7ywACgkQwL7PKlBZWjvejACdEfyE6YeO+3WrhrUhF0CIgSjU
xxQAoNTz8ZXjk6dqpE5EIB3c/gsHNoyR
=26UX
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> Stefano Bagnara wrote:
>> Stuart D. Gathman ha scritto:
>>> Agreed. But I'm pretty sure it is due to not having come up with an
>>> actual procedure rather than self promotion. Maybe you could
>>> volunteer to test implementations using your live DNS test framework?
>>> That would at least get yours done :-)
>> I'm already testing jSPF with my tester :-)
>>
>> My effort to make the tester "implementation agnostic" is really to
>> gain some more trust from you when you see that my tester correctly
>> check your implementations.
>
> Testing implementations from an additional angle is good; any disparity
> between testing results from the different angles is likely to help
> exposing bugs. The reason that I, personally, haven't invested the
> necessary time to implement an online-DNS tester yet is simply that I
> felt the benefit to be expected wouldn't justify the (time) cost. But if
> you're willing to set this up, please go for it!
>
>> It is OpenSPF that have to tell how an implementation is certified.
>> What tests have Mail::SPF and pyspf 2.0 passed in order to be listed
>> there?? I bet jSPF passed the same tests.
>
> You have to pass the RFC 4408 test suite that can be downloaded from
>
> http://www.openspf.org/Test_Suite

We pass it since 2 years I guess..

> More precisely, you have to convince someone who is able to edit the SPF
> project's website that the implementation passes the test suite.

That's unfair. IMHO.
You didn't have to convince anyone that pyspf and Mail::SPF pass the
test suite.

>> In fact we always had selftests based on the yaml file you publish and
>> we have some more unit test in place to increase the coverage, too.
>
> That's good, but unfortunately it was too difficult (for me, who had first
> given it a try) to verify that. See my previous mail.

The same difficulties I find with Mail::SPF testsuites and pyspf
testsuites..

So you simply tell me that to be listed in the certification you want
need a perl or python test suite or you won't list it?

What happens if I create an implementation for an operative system you
don't have access to? No certification??

>> I'd happily complete the live dns tester tool but in order to check
>> implementations they have to return an "spfquery" like result, 4 lines
>> where the 1st is the result, the 2nd is the explanation, the 4th is the
>> Received-SPF: header. Then they also have the ability to use a specific
>> dns server for their queries (e.g: --dnsserver|-s <IP>[:PORT]).
>> I can make parameters configurable, but that dnsserver option is needed
>> in order to run my tester. AFAIK no implementation currently support
>> this (jSPF will support this in the next release).
>
> Can we instead agree instead on having a public DNS zone delegated to your
> server and have it serve test records from there? That seems cleaner to
> me than forcing implementations to use a specific resolver server.

No. The Yaml tests do not share the same zone. In fact every single yaml
test declare its own zone (zonedata:) and I reconfigure the live tester
and empty the caches at each test to make it work.

If you make sure that the "zonedata:" from the rfc4408 can be merged in
a single zonedata without conflicts then we can use this way.

In this case maybe you should rename the "example.com" in the testsuite
to "testsuite.openspf.org" and then have that ptr pointing to some host
where we run the live zone... not sure how feasible it is (I'm not ready
to host a public service for this, I can manage if someone offer a box).

>> Using the "commandline interface" is the only way I found to test
>> different implementations using a single tester.
>
> Or you could use the spfd interface: pipe test data into STDIN, get
> results from STDOUT. See http://search.cpan.org/dist/Mail-SPF/sbin/spfd
> for a good documentation of that interface. (spfd uses a TCP or UNIX
> socket, but you could just as well implement the interface using a pipe.)

Interesting! How standard/used is this protocol?

Wouldn't it worth to publish command line conventions and spfd
"protocols" in the OpenSPF website so that implementations can easily
see what are the "suggested" interfaces?

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Julian Mehnle ha scritto:
> > More precisely, you have to convince someone who is able to edit the
> > SPF project's website that the implementation passes the test suite.
>
> That's unfair. IMHO.
> You didn't have to convince anyone that pyspf and Mail::SPF pass the
> test suite.

Right, I didn't _have_ to. I could have easily faked my test suite
results or simply edited the website and declared Mail::SPF as compliant
without any proof whatsoever.

However, it should be trivial for anyone who doesn't trust me to follow
http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL
and check whether I have lied.

The same was, and still is, NOT true for jSPF, however:

I just gave it another try. I installed Sun Java 6 and Maven 2 on my
Debian machine, downloaded and unpacked apache-jspf-0.9.6-src.tar.gz, and
tried following BUILDING.txt in the hope that it would automatically run
tests during building (there was no other documentation to be found in the
tarball on how to run the tests).

I do not have the James source installed, so I edited pom.xml as per
BUILDING.txt (see the attached diff), and then ran `mvn -Plocal package`.
It downloaded a lot of packages, and then this is what I got:

- ----------8<----------8<----------8<----------8<----------8<----------
...
[INFO] Setting property: classpath.resource.loader.class
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO] [remote-resources:process {execution: default}]
Downloading:
file:///home/julian/tmp/apache-jspf-0.9.6/stage/org.apache/jars/apache-jar-resource-bundle-1.2.jar
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Resources JAR cannot be found.

Embedded error: Requested download does not exist.
Unable to download the artifact from any repository
- ---------->8---------->8---------->8---------->8---------->8----------

Conversely, _I_ can easily convince anyone who wants to know that
Mail::SPF passes the test suite, and I'm sure I have already convinced at
least one or two other participants of the SPF project.

> > That's good, but unfortunately it was too difficult (for me, who had
> > first given it a try) to verify that. See my previous mail.
>
> The same difficulties I find with Mail::SPF testsuites and pyspf
> testsuites..

What difficulties? I'd very much like to make testing Mail::SPF as easy
as possible.

> So you simply tell me that to be listed in the certification you want
> need a perl or python test suite or you won't list it?

No.

> What happens if I create an implementation for an operative system you
> don't have access to? No certification??

No, in that case I (or someone else on the project) will first try to get
access to a platform where it can be run (you could provide such access).
If all fails, I (or they) will probably certify compliance based on some
cursory evidence (such as a log file demonstrating that the implemen-
tation passed the test suite).

My personal view aside on the requirements for listing on the website an
implementation as fully compliant, I am not the owner of the SPF project.
It's very much possible that someone else capabable of editing the website
is less demanding.

See my next, separate message for my comments on building a live-DNS
testing infrastructure.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkisAxQACgkQwL7PKlBZWjs/YgCeKak2l30QLcGSEhr9t1UsBJ2c
Ga0An2OMIBF+CdMCgNgGKDOVVmQw7oym
=BtYX
-----END PGP SIGNATURE-----



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Julian Mehnle ha scritto:
>>> More precisely, you have to convince someone who is able to edit the
>>> SPF project's website that the implementation passes the test suite.
>> That's unfair. IMHO.
>> You didn't have to convince anyone that pyspf and Mail::SPF pass the
>> test suite.
>
> Right, I didn't _have_ to. I could have easily faked my test suite
> results or simply edited the website and declared Mail::SPF as compliant
> without any proof whatsoever.
>
> However, it should be trivial for anyone who doesn't trust me to follow
> http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL
> and check whether I have lied.
>
> The same was, and still is, NOT true for jSPF, however:
>
> I just gave it another try. I installed Sun Java 6 and Maven 2 on my
> Debian machine, downloaded and unpacked apache-jspf-0.9.6-src.tar.gz, and
> tried following BUILDING.txt in the hope that it would automatically run
> tests during building (there was no other documentation to be found in the
> tarball on how to run the tests).
>
> I do not have the James source installed, so I edited pom.xml as per
> BUILDING.txt (see the attached diff), and then ran `mvn -Plocal package`.
> It downloaded a lot of packages, and then this is what I got:
>
> - ----------8<----------8<----------8<----------8<----------8<----------
> ...
> [INFO] Setting property: classpath.resource.loader.class
> => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
> [INFO] Setting property: velocimacro.messages.on => 'false'.
> [INFO] Setting property: resource.loader => 'classpath'.
> [INFO] Setting property: resource.manager.logwhenfound => 'false'.
> [INFO] [remote-resources:process {execution: default}]
> Downloading:
> file:///home/julian/tmp/apache-jspf-0.9.6/stage/org.apache/jars/apache-jar-resource-bundle-1.2.jar
> [INFO] ------------------------------------------------------------------------
> [ERROR] BUILD ERROR
> [INFO] ------------------------------------------------------------------------
> [INFO] Resources JAR cannot be found.
>
> Embedded error: Requested download does not exist.
> Unable to download the artifact from any repository
> - ---------->8---------->8---------->8---------->8---------->8----------
>
> Conversely, _I_ can easily convince anyone who wants to know that
> Mail::SPF passes the test suite, and I'm sure I have already convinced at
> least one or two other participants of the SPF project.

Try to remove the "-Plocal" and give it one more try, please.
Here it works with and without "-Plocal", but maybe in your environment
it is not possible to have an offline build for this.

>>> That's good, but unfortunately it was too difficult (for me, who had
>>> first given it a try) to verify that. See my previous mail.
>> The same difficulties I find with Mail::SPF testsuites and pyspf
>> testsuites..
>
> What difficulties? I'd very much like to make testing Mail::SPF as easy
> as possible.

The difficulties of running a perl environment on my machine.
Not knowing perl is not so easy as you describe it in the INSTALL, but I
understand this is extremely easy for a perl developer that is the
target of your product. For a java developer is easy to run tests for
jspf: there is nothing more standard than our tests in a java world. If
they don't want to use maven they simply can use any junit tester around
the world or an IDE supporting junits (I use eclipse).

>> So you simply tell me that to be listed in the certification you want
>> need a perl or python test suite or you won't list it?
>
> No.

That's good to know.

>> What happens if I create an implementation for an operative system you
>> don't have access to? No certification??
>
> No, in that case I (or someone else on the project) will first try to get
> access to a platform where it can be run (you could provide such access).
> If all fails, I (or they) will probably certify compliance based on some
> cursory evidence (such as a log file demonstrating that the implemen-
> tation passed the test suite).
>
> My personal view aside on the requirements for listing on the website an
> implementation as fully compliant, I am not the owner of the SPF project.
> It's very much possible that someone else capabable of editing the website
> is less demanding.

That's ok. Just put yourself in my shoes and look that the OpenSPF site
keeps telling the only compliant implementations are pyspf and Mail::SPF
and you can find in this list I already tried in past to have it listed.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Wed, 20 Aug 2008, Alessandro Vesely wrote:

> IMHO, the best thing would be to have those tuples published on a web page. If
> it were also possible to post the results in order to have a response, much
> like html validation pages, that would be great!

That is what the test suite is: a collection of such tuples (actually
maps so that the order is explicit).

> In the latter case, if the submitter authenticates and specifies name and
> version of the implementation, it will be trivial to build a database of
> verified implementations, along with the date and test suite version.

Just as important is who did the verification. Ideally, an independent
3rd party such inspect and run the tests. But that requires volunteers
or a budget. In the interim, just documenting who ran the tests - even
if it is someone from the same project - would be better than nothing.

> Validating Received-SPF can be slightly more difficult, since order and
> indentation may vary.

The official test suite doesn't validate received-spf for this reason.
However, the schema supports adding such tests (and pyspf uses them).
It is trivial for a given implementation since you can check for an
exact received-spf output.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Try to remove the "-Plocal" and give it one more try, please.
> Here it works with and without "-Plocal", but maybe in your environment
> it is not possible to have an offline build for this.

Now I got somewhat further, but it still failed:

- ----------8<----------8<----------8<----------8<----------8<----------
...
[INFO] [surefire:test]
[INFO] Surefire report directory: /home/julian/tmp/apache-jspf-0.9.6/target/surefire-reports
org.apache.maven.surefire.booter.SurefireExecutionException: Unable to create test class 'org.apache.james.jspf.SPF1ParserTest'; nested exception is java.lang.ClassNotFoundException:
org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}; nested
exception is org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test class 'org.apache.james.jspf.SPF1ParserTest'; nested exception is java.lang.ClassNotFoundException:
org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test class 'org.apache.james.jspf.SPF1ParserTest'; nested exception is java.lang.ClassNotFoundException:
org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
java.lang.ClassNotFoundException: org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar],
parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
at java.net.URLClassLoader.findClass(libgcj.so.90)
at java.lang.ClassLoader.loadClass(libgcj.so.90)
at java.lang.ClassLoader.loadClass(libgcj.so.90)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.locateTestSets(AbstractDirectoryTestSuite.java:87)
at org.apache.maven.surefire.Surefire.createSuiteFromDefinition(Surefire.java:209)
at org.apache.maven.surefire.Surefire.run(Surefire.java:156)
at java.lang.reflect.Method.invoke(libgcj.so.90)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:338)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:997)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] There are test failures.

Please refer to /home/julian/tmp/apache-jspf-0.9.6/target/surefire-reports for the individual test results.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19 seconds
[INFO] Finished at: Wed Aug 20 13:54:54 GMT 2008
[INFO] Final Memory: 34M/58M
[INFO] ------------------------------------------------------------------------
- ---------->8---------->8---------->8---------->8---------->8----------

> > > The same difficulties I find with Mail::SPF testsuites and pyspf
> > > testsuites..
> >
> > What difficulties? I'd very much like to make testing Mail::SPF as
> > easy as possible.
>
> The difficulties of running a perl environment on my machine.
> Not knowing perl is not so easy as you describe it in the INSTALL, but
> I understand this is extremely easy for a perl developer that is the
> target of your product.

No, this has nothing to do with Perl other than you need to have Perl
installed. Everything else is just what INSTALL tells you.

I appreciate your helping me getting jSPF to build, though.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkisI4MACgkQwL7PKlBZWjvDegCfVi/+Yp7+7qoz0lXyEFidzyY+
mTcAnRamKAcdUtw7bG3jAPIe+XD/gIlf
=kFQG
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Stuart D. Gathman wrote:

> What about it guys? I would "certainly" vote for jSPF as
> "certified", but what about the next implementation that
> comes up? What is the procedure?

Self-certification is one way to tackle it. Third party
testing is likely beyond our (voluntary) capabilities.

If a distribution contains its own test suite users could
check that it works also on their platform. Ideally this
self test can handle any test suite version as input.

Some weeks somebody used his own "manual" yaml to XML (or
similar) input transformation, and posted his results...

> Did we have a procedure written, but forgot about it?

No, we discussed the problem in early 2007 or late 2006,
but didn't arrive at any convincing ideas, let alone any
written rule.

> Stefano's suggestion to offer a test via the spfquery
> interface and his live DNS framework is an excellent
> solution.

Indeed, this also covers an open point in the test suite
wish list. For corner cases like an IDNA <toplabel> it
cannot emulate this effect below a real non-IDNA domain,
but that's as it is.

Frank



-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Try to remove the "-Plocal" and give it one more try, please.
>> Here it works with and without "-Plocal", but maybe in your environment
>> it is not possible to have an offline build for this.
>
> Now I got somewhat further, but it still failed:
>
> - ----------8<----------8<----------8<----------8<----------8<----------
> ...
> [INFO] [surefire:test]
> [INFO] Surefire report directory: /home/julian/tmp/apache-jspf-0.9.6/target/surefire-reports
> org.apache.maven.surefire.booter.SurefireExecutionException: Unable to create test class 'org.apache.james.jspf.SPF1ParserTest'; nested exception is java.lang.ClassNotFoundException:
> org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}; nested
> exception is org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test class 'org.apache.james.jspf.SPF1ParserTest'; nested exception is java.lang.ClassNotFoundException:
> org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
> org.apache.maven.surefire.testset.TestSetFailedException: Unable to create test class 'org.apache.james.jspf.SPF1ParserTest'; nested exception is java.lang.ClassNotFoundException:
> org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
> java.lang.ClassNotFoundException: org.apache.james.jspf.SPF1ParserTest not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/tmp/surefirebooterxp0dwr.jar],
> parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
> at java.net.URLClassLoader.findClass(libgcj.so.90)
> at java.lang.ClassLoader.loadClass(libgcj.so.90)
> at java.lang.ClassLoader.loadClass(libgcj.so.90)
> at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.locateTestSets(AbstractDirectoryTestSuite.java:87)
> at org.apache.maven.surefire.Surefire.createSuiteFromDefinition(Surefire.java:209)
> at org.apache.maven.surefire.Surefire.run(Surefire.java:156)
> at java.lang.reflect.Method.invoke(libgcj.so.90)
> at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:338)
> at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:997)
> [INFO] ------------------------------------------------------------------------
> [ERROR] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] There are test failures.
>
> Please refer to /home/julian/tmp/apache-jspf-0.9.6/target/surefire-reports for the individual test results.
> [INFO] ------------------------------------------------------------------------
> [INFO] For more information, run Maven with the -e switch
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 19 seconds
> [INFO] Finished at: Wed Aug 20 13:54:54 GMT 2008
> [INFO] Final Memory: 34M/58M
> [INFO] ------------------------------------------------------------------------
> - ---------->8---------->8---------->8---------->8---------->8----------

gnu.gcj.runtime ??? I guess you are using gcj and not a JAVA(tm) virtual
machine. Please use a compliant virtual machine (I usually use openjdk
or any sun implementation, but bea, ibm, harmony jvms should work too).

>>>> The same difficulties I find with Mail::SPF testsuites and pyspf
>>>> testsuites..
>>> What difficulties? I'd very much like to make testing Mail::SPF as
>>> easy as possible.
>> The difficulties of running a perl environment on my machine.
>> Not knowing perl is not so easy as you describe it in the INSTALL, but
>> I understand this is extremely easy for a perl developer that is the
>> target of your product.
>
> No, this has nothing to do with Perl other than you need to have Perl
> installed. Everything else is just what INSTALL tells you.
>
> I appreciate your helping me getting jSPF to build, though.

This is the same for jSPF: you should only need a proper java virtual
machine (1.4+) installed.

I am happy you decided to try this. Let me know if you have any more
issue and I'll help.

Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

On Wed, 20 Aug 2008, Stefano Bagnara wrote:

> > [INFO] Surefire report directory:
> > /home/julian/tmp/apache-jspf-0.9.6/target/surefire-reports
> > org.apache.maven.surefire.booter.SurefireExecutionException: Unable to
> > create test class 'org.apache.james.jspf.SPF1ParserTest'; nested exception
> > is java.lang.ClassNotFoundException: org.apache.james.jspf.SPF1ParserTest
> > not found in

> gnu.gcj.runtime ??? I guess you are using gcj and not a JAVA(tm) virtual
> machine. Please use a compliant virtual machine (I usually use openjdk or any
> sun implementation, but bea, ibm, harmony jvms should work too).

The gcj should get past this once you figure out how to set the CLASSPATH
to find SPF1ParserTest. It will probably work to set CLASSPATH=.
if you wrote the compiled classes to current directory.

I downloaded jSPF, but am stuck on installing dnsjava (I know where to look,
but don't have time for the process. It isn't in jpackage.)

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> gnu.gcj.runtime ??? I guess you are using gcj and not a JAVA(tm)
> virtual machine. Please use a compliant virtual machine (I usually use
> openjdk or any sun implementation, but bea, ibm, harmony jvms should
> work too).

http://packages.debian.org/lenny/sun-java6-jre
^^^ This looks like the Java(TM) virtual machine to me, right?

It seems that for the JDK (!= JRE) I had erroneously installed the
default-jdk (=> java-gcj-compat-dev) package rather than the
sun-java6-jdk one.

After correcting that, the build succeeded. Unfortunately, the test
output is still LOTS of raw stuff that I cannot ever hope to comprehend.
But at least it looked legitimate. :-)

> > I appreciate your helping me getting jSPF to build, though.
>
> This is the same for jSPF: you should only need a proper java virtual
> machine (1.4+) installed.

Not exactly.

I recommend that you include a note in your BUILDING.txt file that one may
have to use `mvn package` instead of `mvn -Plocal`, and also have it say
that tests will be run automatically when doing a build.

I also encountered two other show stoppers (one see above, the other being
that some part of Maven seems to parse the output of `env` incorrectly
when the environment contains shell functions -- WTF?), but they are not
specific to jSPF so there's nothing you can do about them.

Finally, it would be good to get the testing output in a more readable and
much less cluttered format, or at least include a way to filter out the
noise.

> I am happy you decided to try this. Let me know if you have any more
> issue and I'll help.

Please tell me what troubles with testing Mail::SPF you were referring to:

> >>>> The same difficulties I find with Mail::SPF testsuites and pyspf
> >>>> testsuites..

Perhaps I can improve something there.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkisZeEACgkQwL7PKlBZWjviJgCgh/QxO8KJjyBF5ibVXTtD7kJU
kYIAmQGP8No4CeRg2+dWERZP6AGMSh8O
=1NA4
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> gnu.gcj.runtime ??? I guess you are using gcj and not a JAVA(tm)
>> virtual machine. Please use a compliant virtual machine (I usually use
>> openjdk or any sun implementation, but bea, ibm, harmony jvms should
>> work too).
>
> http://packages.debian.org/lenny/sun-java6-jre
> ^^^ This looks like the Java(TM) virtual machine to me, right?
>
> It seems that for the JDK (!= JRE) I had erroneously installed the
> default-jdk (=> java-gcj-compat-dev) package rather than the
> sun-java6-jdk one.

Sure sorry. I forgot you are not a java guy. To compile packages you
need a JDK and not a JRE.
However to run jSPF you only need the JRE.
Testing the library without building will only require the JRE (we don't
have bytecode-compiled distribution for our tester yet).

> After correcting that, the build succeeded. Unfortunately, the test
> output is still LOTS of raw stuff that I cannot ever hope to comprehend.
> But at least it looked legitimate. :-)

The DEBUG output is verbose. It is useful to have it verbose to have any
chance to understand non-repeatable bugs and you run the build in a
continuous integration environment.

In java there are a number of Junit runners and they will produce fancy
and cleaner output.

I'm not going to alter our build for this. IMHO there is no real need
for this. Instead I prefer to waste my time working on the live tester
so that I can see the same output for any implementation.

>>> I appreciate your helping me getting jSPF to build, though.
>> This is the same for jSPF: you should only need a proper java virtual
>> machine (1.4+) installed.
>
> Not exactly.
>
> I recommend that you include a note in your BUILDING.txt file that one may
> have to use `mvn package` instead of `mvn -Plocal`, and also have it say
> that tests will be run automatically when doing a build.

Ok, but this is not about compliance. Users use the compiled version.
This is not perl, java bytecode works on any platform without the need
to be compiled. Users of our library won't compile it.

I'll add an hint, anyway.

> I also encountered two other show stoppers (one see above, the other being
> that some part of Maven seems to parse the output of `env` incorrectly
> when the environment contains shell functions -- WTF?), but they are not
> specific to jSPF so there's nothing you can do about them.

I want to point out that it should not require you to be able to build a
library in order to declare it compliant. A library could even be closed
source while still pass any test.

> Finally, it would be good to get the testing output in a more readable and
> much less cluttered format, or at least include a way to filter out the
> noise.

"noise" is something depending on the information you are looking for.

You can use the "mvn package site" command. It will create a full
website for the jSPF tool and this will include the surfire test report
in the target/site/surefire-report.html file.

I don't expect a perl developer and a java developer to agree on best
practices ;-)

>> I am happy you decided to try this. Let me know if you have any more
>> issue and I'll help.
>
> Please tell me what troubles with testing Mail::SPF you were referring to:

I don't remember, it's 1.5 years ago. I admit I had not too much
interest in this so I stopped at the first issues, but I was not
successful in compiling/installing/running it in our Solaris box because
I had a different perl installed and I was not able to change that and
on my windows box (but there I have perl installation issues more often
than not).

>>>>>> The same difficulties I find with Mail::SPF testsuites and pyspf
>>>>>> testsuites..
>
> Perhaps I can improve something there.

If I'll try again in the near future to build on the 2 problematic boxes
I'll report a better issue.

Thank you,
Stefano


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com