Greetings,
I've been doing some research on the reliability of SPF with respect to
UDP vs. TCP queries, and I found some interesting facts and some risks
that were not considered before.
Below I will present them, along with some recommendations for implementers.
Some background
===============
SPF is published in TXT records. When a TXT record is needed, an
application typically calls res_query() to resolve the domain name and
fetch the TXT record from a name server.
By default, the resolver library sends a query as a UDP packet when
res_query() is invoked. The server will respond to a UDP query with a
UDP answer. If the answer takes more than 512-bytes to encode in the
reponse packet, the "truncated answer" bit is set by the name server in
the answer UDP packet.
When the resolver library receives a UDP response with the "truncated
answer" bit set, it usually discards that answer and attempts a TCP
query for the same record. TCP allows arbitrary length of response, so
no matter how long the answer is, it will not be trucated.
UDP is a connection-less protocol, and thus a query/answer conversation
takes exactly two packets (a query packet and a response packet).
TCP is a connection protocol, and every query takes several packets, and
a longer dialog. First, a connection must be made, then the query is
sent, the answer returned, and the connection must be torn down. The
connection bits take several packets transfered back and forth. Each
time a packet is transfered, it is delayed by the latency of the network.
For instance, in order to transfer the same data that a UDP conversation
does in 2 packets, TCP takes 13 packets to complete (3 for connection, 6
for data, 4 for tear down)
On top of this, both kernels (the client and server sides) have to
maintain state in TCP, but not in UDP. Maintaining state takes memory.
Thus TCP is much more expensive than UDP and should be avoided when
possible, for performance reasons.
Very frequently, firewalls block port 53 TCP. This means that the only
way for DNS queries to be answered successfully is if they can be done
over UDP.
Thus, DNS over TCP should be avoided for reliability reasons also.
The problem
===========
SPF records are currently entered manually into a zone file. The
publisher (a human) works with the guideline of 450-bytes as a maximum
record size to publish.
The problem is that when a name server responds to a query, it also
packs the question in the reponse packet, as well as an "authority
section" listing name servers that are responsible for that query, and
an "additional information" section. In the case of TXT records, there
is probably only an empty "additional information" section if the answer
is found. Some servers put the SOA in the "additional information"
section if the answer does not exist.
Due to the extra information that must be packed in the same response
packet, the real space available for a TXT record is variable.
As the question (domain name being queried) is longer, the payload
available for the answer decreases (ie, in a response packet from
ohmi.org, 450-bytes of payload might be available, but a response for
subdomain.alongerdomainname.com has less payload space available).
Also, as a domain employs more authoritative name servers, the authority
section grows, leaving even less space for the TXT record. As the names
of those name servers are longer (ns1.zoneedit.com vs. ns1.dns.uk), the
space available for the TXT record is even less.
Unfortunately, if an administrator has to publish an SPF record
manually, it is very tedious and error prone to calculate how much space
is actually available. Also, if the domain's name-server list changes
later, the SPF space available changes also, even though the SPF record
itself does not.
So the longer an SPF record a domain has, the more likely it is that it
will not fit into a UDP packet, and will require the resolver library to
use TCP. This should be avoided, for the reasons I explained in the
background section (performance and reliability).
While your applications may run fine even with TCP queries in your test
environment, if that software is ever deployed behind a firewall or some
other device which blocks port 53 TCP, the MTA will not be able to
accept mail from the domains that publish such long records:
If the TCP query times out because the connection cannot be made due to
the firewall, the MTA will have to "temporarily" reject the mail, until
the SPF record is resolved. Unfortuantely, this is a permanent failure,
as the firewall be always block port 53 TCP. So the MTA will never
accept any mail from that domain (probably not even the administrator or
postmaster).
While it can be argued that it's the fault of the record with the long
SPF record, I believe the problem lies in two places:
1. The firewall configuration.
2. The SPF checking software
Note that this was never a problem with MTA's before, because they never
needed to do TCP queries until this SPF method was invented.
The solution
============
A two part solution is necessary to deal with this problem:
1. The SPF standard will be ammended to specify the TXT record size
limit as a function of the domain's FDQN length.
2. The SPF software must give careful consideration to UDP vs. TCP queries.
I will first look into the SPF software design, and then at the
record-size limit that is necessary.
The DNS response packet is organized as follows:
+---------------------+
| Header |
+---------------------+
| Question | the question for the name server
+---------------------+
| Answer | RRs answering the question
+---------------------+
| Authority | RRs pointing toward an authority
+---------------------+
| Additional | RRs holding additional information
+---------------------+
The SPF implementation does not actually need the Authority or
Additional sections in the answer. Thus, if a UDP response is received
that contais a complete "Answer section", a follow-up TCP query is not
necessary. The header contains information on the length of the Answer,
so by looking at the header, one can tell if a truncated DNS reply
contains a complete "answer section".
In order to keep the resolver library from automatically doing a TCP
query when a UDP response is truncated, one must set the RES_IGNTC
option before calling res_query. This will cause the resolver to ignore
truncation errors.
Also, in order to remove the dealock involving firewalls that block TCP
53, the SPF software should do the following:
set RES_IGNTC // ie, UDP only.
res_query()
if (timeout)
set SPF_RESULT "TEMPFAIL"
return
end
if (reply_truncated() && answer_incomplete())
reset RES_IGNTC
set RES_USEVC // Force TCP, no point retrying UDP
res_query()
if (timeout)
evaluate the truncated UDP response
set SPF_DEFAULT_RESULT "NONE"
else
use TCP response
end
end
In this case, if the UDP answer is incomplete but the TCP query is
impossible, the SPF client will evaluate as much of the received answer
as is possible, but:
1. Ignore the last mechanism which is not followed by a whitespace (ie,
if the incomplete answer ends with "... ip4:203.45.3". This should not
result in a PermError. Just because the software cannot fetch the
complete SPF record doesn't mean it's syntactically wrong.
2. If none of the mechanisms in the evaluated incomplete record match,
the SPF evaluation should return "none", as if no SPF record was found
at all. Returning "neutral" is incorrect, as the publisher may have
never allowed any of his mechanisms to return "neutral".
Portability
===========
Unfortunately, not all OS's implement the RES_IGNTC feature. Linux does
not (neither glibc nor BIND implement it). However, HPUX and all BSDs do
implement it.
My information source is the respective res_query MAN pages, so it may
not be up to date.
This means that on the systems that do not support the RES_IGNTC, you
should substitute the system-provided res_query() with your own version
(or perhaps a copy of BSD's version)
How you can help
================
It would be great if all SPF implementations would be reliable with
respect this UDP vs. TCP issue, regardless if they're open source or
commercial.
You can help by making a patch openly available for Linux glibc, or a
res_query() function with RES_IGNTC support.
You can also help by providing information about which OS's support
RES_IGNTC (and starting with which version) and which do not.
I imagine the patch would be very simple, as it would simply have to
return a truncated answer instead of branching to the TCP attempt.
If you take up the patching task, please also fix the Linux Man page for
resolver(3) to remove the "Not Currently Implemented" note affixed to
this feature.
Comments and suggestions are welcome.
Regards,
Radu.
-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com
I've been doing some research on the reliability of SPF with respect to
UDP vs. TCP queries, and I found some interesting facts and some risks
that were not considered before.
Below I will present them, along with some recommendations for implementers.
Some background
===============
SPF is published in TXT records. When a TXT record is needed, an
application typically calls res_query() to resolve the domain name and
fetch the TXT record from a name server.
By default, the resolver library sends a query as a UDP packet when
res_query() is invoked. The server will respond to a UDP query with a
UDP answer. If the answer takes more than 512-bytes to encode in the
reponse packet, the "truncated answer" bit is set by the name server in
the answer UDP packet.
When the resolver library receives a UDP response with the "truncated
answer" bit set, it usually discards that answer and attempts a TCP
query for the same record. TCP allows arbitrary length of response, so
no matter how long the answer is, it will not be trucated.
UDP is a connection-less protocol, and thus a query/answer conversation
takes exactly two packets (a query packet and a response packet).
TCP is a connection protocol, and every query takes several packets, and
a longer dialog. First, a connection must be made, then the query is
sent, the answer returned, and the connection must be torn down. The
connection bits take several packets transfered back and forth. Each
time a packet is transfered, it is delayed by the latency of the network.
For instance, in order to transfer the same data that a UDP conversation
does in 2 packets, TCP takes 13 packets to complete (3 for connection, 6
for data, 4 for tear down)
On top of this, both kernels (the client and server sides) have to
maintain state in TCP, but not in UDP. Maintaining state takes memory.
Thus TCP is much more expensive than UDP and should be avoided when
possible, for performance reasons.
Very frequently, firewalls block port 53 TCP. This means that the only
way for DNS queries to be answered successfully is if they can be done
over UDP.
Thus, DNS over TCP should be avoided for reliability reasons also.
The problem
===========
SPF records are currently entered manually into a zone file. The
publisher (a human) works with the guideline of 450-bytes as a maximum
record size to publish.
The problem is that when a name server responds to a query, it also
packs the question in the reponse packet, as well as an "authority
section" listing name servers that are responsible for that query, and
an "additional information" section. In the case of TXT records, there
is probably only an empty "additional information" section if the answer
is found. Some servers put the SOA in the "additional information"
section if the answer does not exist.
Due to the extra information that must be packed in the same response
packet, the real space available for a TXT record is variable.
As the question (domain name being queried) is longer, the payload
available for the answer decreases (ie, in a response packet from
ohmi.org, 450-bytes of payload might be available, but a response for
subdomain.alongerdomainname.com has less payload space available).
Also, as a domain employs more authoritative name servers, the authority
section grows, leaving even less space for the TXT record. As the names
of those name servers are longer (ns1.zoneedit.com vs. ns1.dns.uk), the
space available for the TXT record is even less.
Unfortunately, if an administrator has to publish an SPF record
manually, it is very tedious and error prone to calculate how much space
is actually available. Also, if the domain's name-server list changes
later, the SPF space available changes also, even though the SPF record
itself does not.
So the longer an SPF record a domain has, the more likely it is that it
will not fit into a UDP packet, and will require the resolver library to
use TCP. This should be avoided, for the reasons I explained in the
background section (performance and reliability).
While your applications may run fine even with TCP queries in your test
environment, if that software is ever deployed behind a firewall or some
other device which blocks port 53 TCP, the MTA will not be able to
accept mail from the domains that publish such long records:
If the TCP query times out because the connection cannot be made due to
the firewall, the MTA will have to "temporarily" reject the mail, until
the SPF record is resolved. Unfortuantely, this is a permanent failure,
as the firewall be always block port 53 TCP. So the MTA will never
accept any mail from that domain (probably not even the administrator or
postmaster).
While it can be argued that it's the fault of the record with the long
SPF record, I believe the problem lies in two places:
1. The firewall configuration.
2. The SPF checking software
Note that this was never a problem with MTA's before, because they never
needed to do TCP queries until this SPF method was invented.
The solution
============
A two part solution is necessary to deal with this problem:
1. The SPF standard will be ammended to specify the TXT record size
limit as a function of the domain's FDQN length.
2. The SPF software must give careful consideration to UDP vs. TCP queries.
I will first look into the SPF software design, and then at the
record-size limit that is necessary.
The DNS response packet is organized as follows:
+---------------------+
| Header |
+---------------------+
| Question | the question for the name server
+---------------------+
| Answer | RRs answering the question
+---------------------+
| Authority | RRs pointing toward an authority
+---------------------+
| Additional | RRs holding additional information
+---------------------+
The SPF implementation does not actually need the Authority or
Additional sections in the answer. Thus, if a UDP response is received
that contais a complete "Answer section", a follow-up TCP query is not
necessary. The header contains information on the length of the Answer,
so by looking at the header, one can tell if a truncated DNS reply
contains a complete "answer section".
In order to keep the resolver library from automatically doing a TCP
query when a UDP response is truncated, one must set the RES_IGNTC
option before calling res_query. This will cause the resolver to ignore
truncation errors.
Also, in order to remove the dealock involving firewalls that block TCP
53, the SPF software should do the following:
set RES_IGNTC // ie, UDP only.
res_query()
if (timeout)
set SPF_RESULT "TEMPFAIL"
return
end
if (reply_truncated() && answer_incomplete())
reset RES_IGNTC
set RES_USEVC // Force TCP, no point retrying UDP
res_query()
if (timeout)
evaluate the truncated UDP response
set SPF_DEFAULT_RESULT "NONE"
else
use TCP response
end
end
In this case, if the UDP answer is incomplete but the TCP query is
impossible, the SPF client will evaluate as much of the received answer
as is possible, but:
1. Ignore the last mechanism which is not followed by a whitespace (ie,
if the incomplete answer ends with "... ip4:203.45.3". This should not
result in a PermError. Just because the software cannot fetch the
complete SPF record doesn't mean it's syntactically wrong.
2. If none of the mechanisms in the evaluated incomplete record match,
the SPF evaluation should return "none", as if no SPF record was found
at all. Returning "neutral" is incorrect, as the publisher may have
never allowed any of his mechanisms to return "neutral".
Portability
===========
Unfortunately, not all OS's implement the RES_IGNTC feature. Linux does
not (neither glibc nor BIND implement it). However, HPUX and all BSDs do
implement it.
My information source is the respective res_query MAN pages, so it may
not be up to date.
This means that on the systems that do not support the RES_IGNTC, you
should substitute the system-provided res_query() with your own version
(or perhaps a copy of BSD's version)
How you can help
================
It would be great if all SPF implementations would be reliable with
respect this UDP vs. TCP issue, regardless if they're open source or
commercial.
You can help by making a patch openly available for Linux glibc, or a
res_query() function with RES_IGNTC support.
You can also help by providing information about which OS's support
RES_IGNTC (and starting with which version) and which do not.
I imagine the patch would be very simple, as it would simply have to
return a truncated answer instead of branching to the TCP attempt.
If you take up the patching task, please also fix the Linux Man page for
resolver(3) to remove the "Not Currently Implemented" note affixed to
this feature.
Comments and suggestions are welcome.
Regards,
Radu.
-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel@v2.listbox.com