Mailing List Archive

Inconsistent behavior between pyspf and libspf2
I'm trying to use SPF in a custom MTA, and I see some inconsistent behavior.
I've used libspf2 (v 1.2.9) and pyspf (v2.005) on Ubuntu 9.04, and see
different behavior from both.

I attached the output from both below.
I want to know which is correct and why they give different results.

The IP: 209.11.164.66, sender: xxx@delivery.net, EHLO: xxx.delivery.net

Any help in resolving this would be great.
I would prefer to use libspf2, as I can make it a part of my executable,
without having to depend on external run-times.

Thanks.
-SJ

-----------------------------------------------------------------------------
Using: spfquery.libspf2 (v1.2.9)
-----------------------------------------------------------------------------

$ /usr/bin/spfquery.libspf2 --ip 209.11.164.66 --sender blah@delivery.net \
--debug=10

spf_compile.c:523 Debug: Parsing macro starting at
Please%_see%_http://www.openspf.org/Why?id=%{S}&ip=%{C}&receiver=%{R}
spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
spf_compile.c:554 Debug: Adding string literal (6): 'Please'
spf_compile.c:589 Debug: Adding literal space
spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
spf_compile.c:554 Debug: Adding string literal (3): 'see'
spf_compile.c:589 Debug: Adding literal space
spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
spf_compile.c:554 Debug: Adding string literal (30):
'http://www.openspf.org/Why?id='
spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d524
spf_compile.c:641 Debug: Next data is at 0xbfb5d528
spf_compile.c:543 Debug: Current data is at 0xbfb5d528
spf_compile.c:554 Debug: Adding string literal (4): '&ip='
spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d530
spf_compile.c:641 Debug: Next data is at 0xbfb5d534
spf_compile.c:543 Debug: Current data is at 0xbfb5d534
spf_compile.c:554 Debug: Adding string literal (10): '&receiver='
spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d544
spf_compile.c:641 Debug: Next data is at 0xbfb5d548
spf_compile.c:1210 Debug: Compiling record v=spf1
spf_dns.c:54 Debug: DNS[cache] lookup: delivery.net SPF (99)
spf_dns.c:54 Debug: DNS[resolv] lookup: delivery.net SPF (99)
spf_dns_resolv.c:311 Debug: query failed: err = -1 Host name lookup failure
(2): delivery.net
spf_dns.c:66 Debug: DNS[resolv] found record
spf_dns.c:69 Debug: DOMAIN: delivery.net TYPE: SPF (99)
spf_dns.c:76 Debug: TTL: 0 RR found: 0 herrno: 2 source: resolv
spf_dns.c:66 Debug: DNS[cache] found record
spf_dns.c:69 Debug: DOMAIN: delivery.net TYPE: SPF (99)
spf_dns.c:76 Debug: TTL: 0 RR found: 0 herrno: 2 source: resolv
spf_server.c:384 Debug: get_record(delivery.net): TRY_AGAIN
--vv--
Context: Main query
Response result: (invalid)
Response reason: none
Response err: No errors
StartError
Error: [UNRETURNED] Temporary DNS failure for 'delivery.net'.
EndError
--^^--
StartError
Context: Failed to query MAIL-FROM
ErrorCode: (26) DNS lookup failure
Error: Temporary DNS failure for 'delivery.net'.
EndError
(invalid)

-----------------------------------------------------------------------------
Using: spfquery.pyspf (v2.005)
-----------------------------------------------------------------------------

$ /usr/bin/spfquery.pyspf --ip 209.11.164.66 --sender blah@delivery.net \
--debug=1

pass

spfquery: domain of delivery.net designates 209.11.164.66 as permitted sender
Received-SPF: Pass (spfquery: domain of delivery.net designates 209.11.164.66
as permitted sender) client-ip=209.11.164.66; envelope-from="blah@delivery.net";
receiver=spfquery; mechanism="ip4:209.11.164.0/22";
identity=mailfrom

-----------------------------------------------------------------------------




-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com
Re: Inconsistent behavior between pyspf and libspf2 [ In reply to ]
On Wed, Nov 25, 2009 at 19:36, stijay <stijay@hotmail.com> wrote:
>
> I'm trying to use SPF in a custom MTA, and I see some inconsistent behavior.
> I've used libspf2 (v 1.2.9) and pyspf (v2.005) on Ubuntu 9.04, and see
> different behavior from both.
>
> I attached the output from both below.
> I want to know which is correct and why they give different results.

It's much, much, easier to help you if you don't try to hide the domain name...

>    Using: spfquery.libspf2  (v1.2.9)
<---SNIP--->
> spf_dns.c:54         Debug: DNS[cache] lookup: delivery.net SPF (99)
> spf_dns.c:54         Debug: DNS[resolv] lookup: delivery.net SPF (99)
<---SNIP--->
> Error: [UNRETURNED] Temporary DNS failure for 'delivery.net'.

That only looked for the record type SPF, which few people publish.
It didn't find one so you saw a temporary error.

>    Using: spfquery.pyspf  (v2.005)
<---SNIP--->
> spfquery: domain of delivery.net designates 209.11.164.66 as permitted sender
> Received-SPF: Pass (spfquery: domain of delivery.net designates 209.11.164.66
> as permitted sender) client-ip=209.11.164.66; envelope-from="blah@delivery.net";
> receiver=spfquery; mechanism="ip4:209.11.164.0/22";
> identity=mailfrom

That looked at the TXT record and found it.

--
Please keep list traffic on the list.

Rob MacGregor
Whoever fights monsters should see to it that in the process he
doesn't become a monster. Friedrich Nietzsche


-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com
Re: Inconsistent behavior between pyspf and libspf2 [ In reply to ]
At 19:36 25/11/2009 Wednesday, stijay wrote:

>I'm trying to use SPF in a custom MTA, and I see some inconsistent behavior.
>I've used libspf2 (v 1.2.9) and pyspf (v2.005) on Ubuntu 9.04, and see
>different behavior from both.
>
>I attached the output from both below.
>I want to know which is correct and why they give different results.
>
>The IP: 209.11.164.66, sender: xxx@delivery.net, EHLO: xxx.delivery.net
>
>Any help in resolving this would be great.
>I would prefer to use libspf2, as I can make it a part of my executable,
>without having to depend on external run-times.
>
>Thanks.
>-SJ

at a guess the one that passes is correctly passing and stopping at the first match

the second failing software is bypassing the pass record and testing each/all expressions
and thus finding/failing at the later syntax error
+exists:CL.%{i}.FR.%{s}.spf.mmtrack.net
as this expands to CL.209.11.164.66.FR.blah@delivery.net.spf.mmtrack.net
which of course has no hope of being resolved due to the %(s) macro mis-use {ie the @ in a hostname}
if they were using this for some sort of sender/ip whitelisting it should have been say

+exists:CL.%{i}.FR.%{l}.AT.%{o}.spf.mmtrack.net

now i don't know spf implementation spec well enough to know if the @ should/would be escaped via -- encoding and this isn't happening?
or if the SPF record is just borked

as only
%(c) %(r) and %(t) are explicitly only allowed in exp macros {maybe %(s) should also be only allowed there as i can see no other use}



>-----------------------------------------------------------------------------
> Using: spfquery.libspf2 (v1.2.9)
>-----------------------------------------------------------------------------
>
>$ /usr/bin/spfquery.libspf2 --ip 209.11.164.66 --sender blah@delivery.net \
>--debug=10
>
>spf_compile.c:523 Debug: Parsing macro starting at
>Please%_see%_http://www.openspf.org/Why?id=%{S}&ip=%{C}&receiver=%{R}
>spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
>spf_compile.c:554 Debug: Adding string literal (6): 'Please'
>spf_compile.c:589 Debug: Adding literal space
>spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
>spf_compile.c:554 Debug: Adding string literal (3): 'see'
>spf_compile.c:589 Debug: Adding literal space
>spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
>spf_compile.c:554 Debug: Adding string literal (30):
> 'http://www.openspf.org/Why?id='
>spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d524
>spf_compile.c:641 Debug: Next data is at 0xbfb5d528
>spf_compile.c:543 Debug: Current data is at 0xbfb5d528
>spf_compile.c:554 Debug: Adding string literal (4): '&ip='
>spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d530
>spf_compile.c:641 Debug: Next data is at 0xbfb5d534
>spf_compile.c:543 Debug: Current data is at 0xbfb5d534
>spf_compile.c:554 Debug: Adding string literal (10): '&receiver='
>spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d544
>spf_compile.c:641 Debug: Next data is at 0xbfb5d548
>spf_compile.c:1210 Debug: Compiling record v=spf1
>spf_dns.c:54 Debug: DNS[cache] lookup: delivery.net SPF (99)
>spf_dns.c:54 Debug: DNS[resolv] lookup: delivery.net SPF (99)
>spf_dns_resolv.c:311 Debug: query failed: err = -1 Host name lookup failure
> (2): delivery.net
>spf_dns.c:66 Debug: DNS[resolv] found record
>spf_dns.c:69 Debug: DOMAIN: delivery.net TYPE: SPF (99)
>spf_dns.c:76 Debug: TTL: 0 RR found: 0 herrno: 2 source: resolv
>spf_dns.c:66 Debug: DNS[cache] found record
>spf_dns.c:69 Debug: DOMAIN: delivery.net TYPE: SPF (99)
>spf_dns.c:76 Debug: TTL: 0 RR found: 0 herrno: 2 source: resolv
>spf_server.c:384 Debug: get_record(delivery.net): TRY_AGAIN
>--vv--
>Context: Main query
>Response result: (invalid)
>Response reason: none
>Response err: No errors
>StartError
>Error: [UNRETURNED] Temporary DNS failure for 'delivery.net'.
>EndError
>--^^--
>StartError
>Context: Failed to query MAIL-FROM
>ErrorCode: (26) DNS lookup failure
>Error: Temporary DNS failure for 'delivery.net'.
>EndError
>(invalid)
>
>-----------------------------------------------------------------------------
> Using: spfquery.pyspf (v2.005)
>-----------------------------------------------------------------------------
>
>$ /usr/bin/spfquery.pyspf --ip 209.11.164.66 --sender blah@delivery.net \
>--debug=1
>
>pass
>
>spfquery: domain of delivery.net designates 209.11.164.66 as permitted sender
>Received-SPF: Pass (spfquery: domain of delivery.net designates 209.11.164.66
>as permitted sender) client-ip=209.11.164.66; envelope-from="blah@delivery.net";
>receiver=spfquery; mechanism="ip4:209.11.164.0/22";
>identity=mailfrom
>
>-----------------------------------------------------------------------------
>
>
>
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/1020/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/1020/
>Powered by Listbox: http://www.listbox.com



-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com
Re: Inconsistent behavior between pyspf and libspf2 [ In reply to ]
rob i right my first analysis was wrong

so it looks like lookups of type 99 {spf} are receiving an unexpected error response from your local dns server
instead of the expected no such record, where the library would then move on and try a TXT lookup next

so next try repeating with another make/model of dns server/cache and see if its specific to that one {bug in dns server code}
or is it repeatable on many {bug in libraries failure SPF vs TXT detection/handling}
i suspect it could be dns server/cache handling of unknown RR types {as many do not yet support/understand type=SPF
but the library should probably also be modified to try TXT
even after the unexpected response to type=SPF queries in light of existence of the broken dns servers



At 19:36 25/11/2009 Wednesday, stijay wrote:

>I'm trying to use SPF in a custom MTA, and I see some inconsistent behavior.
>I've used libspf2 (v 1.2.9) and pyspf (v2.005) on Ubuntu 9.04, and see
>different behavior from both.
>
>I attached the output from both below.
>I want to know which is correct and why they give different results.
>
>The IP: 209.11.164.66, sender: xxx@delivery.net, EHLO: xxx.delivery.net
>
>Any help in resolving this would be great.
>I would prefer to use libspf2, as I can make it a part of my executable,
>without having to depend on external run-times.
>
>Thanks.
>-SJ
>
>-----------------------------------------------------------------------------
> Using: spfquery.libspf2 (v1.2.9)
>-----------------------------------------------------------------------------
>
>$ /usr/bin/spfquery.libspf2 --ip 209.11.164.66 --sender blah@delivery.net \
>--debug=10
>
>spf_compile.c:523 Debug: Parsing macro starting at
>Please%_see%_http://www.openspf.org/Why?id=%{S}&ip=%{C}&receiver=%{R}
>spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
>spf_compile.c:554 Debug: Adding string literal (6): 'Please'
>spf_compile.c:589 Debug: Adding literal space
>spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
>spf_compile.c:554 Debug: Adding string literal (3): 'see'
>spf_compile.c:589 Debug: Adding literal space
>spf_compile.c:543 Debug: Current data is at 0xbfb5d4f4
>spf_compile.c:554 Debug: Adding string literal (30):
> 'http://www.openspf.org/Why?id='
>spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d524
>spf_compile.c:641 Debug: Next data is at 0xbfb5d528
>spf_compile.c:543 Debug: Current data is at 0xbfb5d528
>spf_compile.c:554 Debug: Adding string literal (4): '&ip='
>spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d530
>spf_compile.c:641 Debug: Next data is at 0xbfb5d534
>spf_compile.c:543 Debug: Current data is at 0xbfb5d534
>spf_compile.c:554 Debug: Adding string literal (10): '&receiver='
>spf_compile.c:620 Debug: Adding macro, data is at 0xbfb5d544
>spf_compile.c:641 Debug: Next data is at 0xbfb5d548
>spf_compile.c:1210 Debug: Compiling record v=spf1
>spf_dns.c:54 Debug: DNS[cache] lookup: delivery.net SPF (99)
>spf_dns.c:54 Debug: DNS[resolv] lookup: delivery.net SPF (99)
>spf_dns_resolv.c:311 Debug: query failed: err = -1 Host name lookup failure
> (2): delivery.net
>spf_dns.c:66 Debug: DNS[resolv] found record
>spf_dns.c:69 Debug: DOMAIN: delivery.net TYPE: SPF (99)
>spf_dns.c:76 Debug: TTL: 0 RR found: 0 herrno: 2 source: resolv
>spf_dns.c:66 Debug: DNS[cache] found record
>spf_dns.c:69 Debug: DOMAIN: delivery.net TYPE: SPF (99)
>spf_dns.c:76 Debug: TTL: 0 RR found: 0 herrno: 2 source: resolv
>spf_server.c:384 Debug: get_record(delivery.net): TRY_AGAIN
>--vv--
>Context: Main query
>Response result: (invalid)
>Response reason: none
>Response err: No errors
>StartError
>Error: [UNRETURNED] Temporary DNS failure for 'delivery.net'.
>EndError
>--^^--
>StartError
>Context: Failed to query MAIL-FROM
>ErrorCode: (26) DNS lookup failure
>Error: Temporary DNS failure for 'delivery.net'.
>EndError
>(invalid)
>
>-----------------------------------------------------------------------------
> Using: spfquery.pyspf (v2.005)
>-----------------------------------------------------------------------------
>
>$ /usr/bin/spfquery.pyspf --ip 209.11.164.66 --sender blah@delivery.net \
>--debug=1
>
>pass
>
>spfquery: domain of delivery.net designates 209.11.164.66 as permitted sender
>Received-SPF: Pass (spfquery: domain of delivery.net designates 209.11.164.66
>as permitted sender) client-ip=209.11.164.66; envelope-from="blah@delivery.net";
>receiver=spfquery; mechanism="ip4:209.11.164.0/22";
>identity=mailfrom
>
>-----------------------------------------------------------------------------
>
>
>
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/1020/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/1020/
>Powered by Listbox: http://www.listbox.com



-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com
Re: Inconsistent behavior between pyspf and libspf2 [ In reply to ]
On Wed, 25 Nov 2009 21:11:11 +0000 Rob MacGregor <rob.macgregor@gmail.com> wrote:
>On Wed, Nov 25, 2009 at 19:36, stijay <stijay@hotmail.com> wrote:
>>
>> I'm trying to use SPF in a custom MTA, and I see some inconsistent behavior.
>> I've used libspf2 (v 1.2.9) and pyspf (v2.005) on Ubuntu 9.04, and see
>> different behavior from both.
>>
>> I attached the output from both below.
>> I want to know which is correct and why they give different results.
>
>It's much, much, easier to help you if you don't try to hide the domain name...
>
>>    Using: spfquery.libspf2  (v1.2.9)
><---SNIP--->
>> spf_dns.c:54         Debug: DNS[cache] lookup: delivery.net SPF (99)
>> spf_dns.c:54         Debug: DNS[resolv] lookup: delivery.net SPF (99)
><---SNIP--->
>> Error: [UNRETURNED] Temporary DNS failure for 'delivery.net'.
>
>That only looked for the record type SPF, which few people publish.
>It didn't find one so you saw a temporary error.
>
>>    Using: spfquery.pyspf  (v2.005)
><---SNIP--->
>> spfquery: domain of delivery.net designates 209.11.164.66 as permitted sender
>> Received-SPF: Pass (spfquery: domain of delivery.net designates 209.11.164.66
>> as permitted sender) client-ip=209.11.164.66; envelope-from="blah@delivery.net";
>> receiver=spfquery; mechanism="ip4:209.11.164.0/22";
>> identity=mailfrom
>
>That looked at the TXT record and found it.

It's slightly more complex than this. Delivery.net's DNS server returns servefail for Type SPF queries (pyspf will do this as well if you set it to query for Type SPF and not just Type TXT).

The pyspf maintainers (of which I am one) take the pragmatic approach that since Type SPF has almost no deployment, routinely querying for it is a waste of time.

If you can control which record types you query for using libspf2's API, then I wouldn't worry about it.

Scott K


-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com
Re: Inconsistent behavior between pyspf and libspf2 [ In reply to ]
Scott Kitterman <scott <at> kitterman.com> writes:

<...SNIP...>

> It's slightly more complex than this. Delivery.net's DNS server returns
servefail for Type SPF queries
> (pyspf will do this as well if you set it to query for Type SPF and not just
Type TXT).
>
> The pyspf maintainers (of which I am one) take the pragmatic approach that
since Type SPF has almost no
> deployment, routinely querying for it is a waste of time.
>
> If you can control which record types you query for using libspf2's API, then
I wouldn't worry about it.
>
> Scott K
>
>


Thank you. That explains the behavioral difference.
But, it looks like there is no direct method of changing the query type from the
libspf API.
I'll probably have to change the library to start with TXT record.

Thank you.
-sj




-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com
Re: Inconsistent behavior between pyspf and libspf2 [ In reply to ]
Hello!

On Wed, Nov 25, 2009 at 09:52:21PM +0000, alan wrote:

>rob i right my first analysis was wrong

>so it looks like lookups of type 99 {spf} are receiving an unexpected error response from your local dns server
>instead of the expected no such record, where the library would then move on and try a TXT lookup next

>so next try repeating with another make/model of dns server/cache and see if its specific to that one {bug in dns server code}
>or is it repeatable on many {bug in libraries failure SPF vs TXT detection/handling}
>i suspect it could be dns server/cache handling of unknown RR types {as many do not yet support/understand type=SPF
>but the library should probably also be modified to try TXT
>even after the unexpected response to type=SPF queries in light of existence of the broken dns servers

IMO it might probably be related to a specific broken DNS implementation,
as DNS (the specifications) *does* provide for handling of unknown RR
types, so a conformant implementation should be able to cope with SPF
records even if it has been made without having SPF records *specifically*
in mind.

At our site, we deploy libspf2 (mostly version 1.2.9) and usually it
works quite well, and we haven't patched its DNS resolution strategy in
this respect.

Kind regards,

Hannah.


-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/1020/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1020/
Powered by Listbox: http://www.listbox.com