Mailing List Archive

Quoted pairs in the test suite
Hi, does the SPF test suite cover oddities like
MAIL FROM:<"\b\a\c\k\s\l\a\s\h"@test.example"> ?

In the SPF I18N entry of the FAQ I claim that
implementations will evaluate a %{l} local part
macro as "backslash" (without the quotes), but
actually I'm far from sure that implementations
get <quoted-pair> within <quoted-string> right.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
On Mon, 14 Apr 2008, Frank Ellermann wrote:

> Hi, does the SPF test suite cover oddities like
> MAIL FROM:<"\b\a\c\k\s\l\a\s\h"@test.example"> ?
>
> In the SPF I18N entry of the FAQ I claim that
> implementations will evaluate a %{l} local part
> macro as "backslash" (without the quotes), but
> actually I'm far from sure that implementations
> get <quoted-pair> within <quoted-string> right.

Good question. So what is the right expansion for %{l}:

1) backslash
2) \b\a\c\k\s\l\a\s\h
3) "\b\a\c\k\s\l\a\s\h"
4) any of the above

You could argue that %{l}@%{d} (in an exp, for example) should be a
syntactically valid email. And leaving the quotes intact would be the
easiest way to ensure that.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Hi, does the SPF test suite cover oddities like
> MAIL FROM:<"\b\a\c\k\s\l\a\s\h"@test.example"> ?
>
> In the SPF I18N entry of the FAQ I claim that implementations will
> evaluate a %{l} local part macro as "backslash" (without the quotes),
> but actually I'm far from sure that implementations get <quoted-pair>
> within <quoted-string> right.

There's a difference between the _value_ of the localpart in a RFC 2821
e-mail address and its _representation_ in the e-mail address. RFC 2821,
section 4.1.2, "Command Argument Syntax", in combination with RFC 2822,
sections 3.2.2, "Quoted characters", and 3.2.5, "Quoted strings", define
how to derive the _value_ of a localpart from its representation (and
vice versa).

According to this definition, the localpart value of the representation in
the above MAIL FROM command is "backslash" without the quotes. And this
is exactly what MTAs should pass to SPF implementations.

The test suite does not (and cannot, and should not) cover parsing of SMTP
command arguments in RFC 2821/2822 _representation_. This is the MTA's
duty. The test suite in YAML format encodes localpart _values_ in YAML
_representation_, but again, parsing those representations isn't up to
the SPF implementation. It's the YAML parsing library's duty. To sum it
up, this isn't something that the test suite can or should or does do.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIA6V7wL7PKlBZWjsRAlWCAKD6iesYOGkoqKkTIMdGCmOXm/kd4ACg0OW/
ouUebgIIujx7VXwkPVtSU5Y=
=A2pi
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Mon, 14 Apr 2008, Frank Ellermann wrote:
> > Hi, does the SPF test suite cover oddities like
> > MAIL FROM:<"\b\a\c\k\s\l\a\s\h"@test.example"> ?
> >
> > In the SPF I18N entry of the FAQ I claim that
> > implementations will evaluate a %{l} local part
> > macro as "backslash" (without the quotes), but
> > actually I'm far from sure that implementations
> > get <quoted-pair> within <quoted-string> right.
>
> Good question. So what is the right expansion for %{l}:
>
> 1) backslash
> 2) \b\a\c\k\s\l\a\s\h
> 3) "\b\a\c\k\s\l\a\s\h"
> 4) any of the above
>
> You could argue that %{l}@%{d} (in an exp, for example) should be a
> syntactically valid email. And leaving the quotes intact would be the
> easiest way to ensure that.

You cannot expect SPF implementations to safely generate valid RFC 2821/
2822 local-parts. You could say "%{l}"@%{d} (with the quotes) in an
explanation string, but if the local-part's _value_ was "foo (with the
quote), the SPF implementation would generate ""foo"@<domain>, whereas
the proper RFC 2821/2822 representation would be "\"foo"@<domain>.

It cannot be done reliably. SPF wasn't designed to do it.

(Also note that RFC 2821, section 4.1.2, sort of states quoted-string
local-parts as being bad practice.)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIA6cAwL7PKlBZWjsRAjj4AJ9HRPumciuKHM7GCGQg3Pzyi1SLIgCgxbZR
ekGarPoEDdwvph7uDRdxkEY=
=76HJ
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Re: Quoted pairs in the test suite [ In reply to ]
On Mon, 14 Apr 2008, Julian Mehnle wrote:

> > You could argue that %{l}@%{d} (in an exp, for example) should be a
> > syntactically valid email. And leaving the quotes intact would be the
> > easiest way to ensure that.
>
> You cannot expect SPF implementations to safely generate valid RFC 2821/
> 2822 local-parts. You could say "%{l}"@%{d} (with the quotes) in an
> explanation string, but if the local-part's _value_ was "foo (with the
> quote), the SPF implementation would generate ""foo"@<domain>, whereas
> the proper RFC 2821/2822 representation would be "\"foo"@<domain>.
>
> It cannot be done reliably. SPF wasn't designed to do it.

You missed the point. If %{l} simply echoes the localpart with quoting
intact, then if the original email was <"\"foo"@example.com>, %{l}
would expand to "\"foo" (with the quotes), rather than try to undo the
quoting. If mail from was <"foo"@example.com> (with the redundant quotes),
then %{l} per my suggestion would be "foo". A similar question goes
for %{s}, which can leave quoting alone, or try to remove it.

I'm saying that removing the quotes is a lot of trouble, and leaving them
alone leave %{s} as a valid email (if mail from was to begin with).

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Mon, 14 Apr 2008, Julian Mehnle wrote:
> > You cannot expect SPF implementations to safely generate valid RFC
> > 2821/ 2822 local-parts. You could say "%{l}"@%{d} (with the quotes)
> > in an explanation string, but if the local-part's _value_ was "foo
> > (with the quote), the SPF implementation would generate
> > ""foo"@<domain>, whereas the proper RFC 2821/2822 representation
> > would be "\"foo"@<domain>.
> >
> > It cannot be done reliably. SPF wasn't designed to do it.
>
> You missed the point. If %{l} simply echoes the localpart with quoting
> intact, then if the original email was <"\"foo"@example.com>, %{l}
> would expand to "\"foo" (with the quotes), rather than try to undo the
> quoting.

True.

> If mail from was <"foo"@example.com> (with the redundant quotes),
> then %{l} per my suggestion would be "foo". A similar question goes
> for %{s}, which can leave quoting alone, or try to remove it.

SPF implementations aren't supposed to perform quote removal in e-mail
addresses or localparts.

> I'm saying that removing the quotes is a lot of trouble, and leaving
> them alone leave %{s} as a valid email (if mail from was to begin
> with).

Agreed.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIA7HqwL7PKlBZWjsRAkfLAKD9BH1e3m4Xk3f4QrSCiEc0SdwXcACfbOmU
+Q0nwUSiOijE/d9G4uMPBEs=
=MKkh
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
Julian Mehnle wrote:

> There's a difference between the _value_ of the localpart
> in a RFC 2821 e-mail address and its _representation_ in
> the e-mail address.

Yep, I know the fine print, 2822upd has this as "semantical
content" IIRC. But in practice these things tend to vary:

RFC 2617 (HTTP AUTH) and its predecessors strip the quotes
from a quoted-string, leaving quoted-pairs alone. RFC 2831
inherited this oddity.

RFC.ietf-usefor-usefor defines a canonical Message-ID, that
boils down to "use a quoted-pair only if you must, and after
that leave it alone, i.e. keep quotes, case, and backlashes
as is no matter what you're up to". The news + nntp URI I-D
sticks to that dogma, unsurprisingly ;-)

The mailto-bis I-D and AFAIK the RFC about tag: URI scheme
also follow a "keep as is" dogma. Of course none of these
I-Ds and RFCs are remotely relevant for SPF - I just want
to point out that "doing the right thing" isn't always the
same thing in different contexts.

IIRC you said here that SPF implementations strip the quotes
from <quoted-string> in a discussion about "embedded dots".

After that they can't escape "embedded dots" even if DNS in
theory supports any octet. After that it's also logical to
get rid of backslashes introducing <quoted-pair>, and so I
arrived at the same result as you, again unsurprisingly. :-)

***BUT***

> this is exactly what MTAs should pass to SPF implementations.

I don't see why MTAs should suddenly start to be smart. They
get an address with a <quoted-string> as local part, and they
pass this on to SPF with a clear directive "now do your magic,
I want PASS or FAIL, a.s.a.p.".

MTAs have no reason to figure an odd "semantical content" out,
that's hardcore standardista stuff, not practical programming.

SPF has to do this - something above checkhost() is fine, but
not so far above checkhost() that it is outside of SPF. There
is no "duty" for MTAs or other SPF users to parse a MAIL FROM
address into its "semantical content".

I'd even fear that nothing removes the reverse route in cases
where it's present. RFC 4408 clearly says "strip it". It's
the job of SPF to strip a reverse route, therefore it's also
the job of SPF to deal with <quoted-string> + <quoted-pair>.

And it's not something that still works if done twice, if an
MTA really already did it SPF must not try it again, example:

MAIL FROM:<@example.com:"\"back\\slash\""@example.org>

The semantical content of the local part is "back\slash" with
the quotes and the backslash.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
Julian Mehnle wrote:

> SPF implementations aren't supposed to perform quote removal
> in e-mail addresses or localparts.

I don't believe it, we discussed this for months here, it was
you who said that ".do..ts." degenerate into .do..ts. and hit
the res_mkquery() wall.

If ".do..ts." are kept as is (quoted), the query for the label
\"\.do\.\.ts\.\" would work, details TBD for the resolver API.

Various errata resolutions and FAQ entries build on your idea
that ".do..ts." degenerate into .do..ts. And they are plain
nonsense if you suddenly change your mind about this.

If SPF implementations are supposed to deal with ".do..ts."
or "\b\a\c\k\s\l\a\s\h" as is that should be also tested, IMO.

What is the query for a literal "\b\a\c\k\s\l\a\s\h" label
with a resolver API, is that \"\\b\\a\\c\\k\\s\\a\\s\\h\" ?

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Re: Quoted pairs in the test suite [ In reply to ]
On Mon, 14 Apr 2008, Frank Ellermann wrote:

> Julian Mehnle wrote:
>
> > SPF implementations aren't supposed to perform quote removal
> > in e-mail addresses or localparts.
>
> I don't believe it, we discussed this for months here, it was
> you who said that ".do..ts." degenerate into .do..ts. and hit
> the res_mkquery() wall.

I think Julian is forgetting about the use of %{l} in exists
queries, and similar things. You are correct that handling of
quoted localpart needs to be consistent for %{l} to work with
such queries and a quoted localpart.

You could argue that a sender (who controls valid localparts) should
avoid such quoted localparts to avoid any inconsistencies in
receivers. However, it *is* our job to specify the correct behaviour
in the test suite - provided the spec is unambiguous (however obscure).

I will admit that I am still trying to figure out what the required
behaviour should be. Which is why I asked.

I think you are saying that all quoting should be completely removed for %{l}
so as to have a consistent DNS label regardless of how the sender (redundantly)
quoted the localpart. It doesn't matter if %{l}@%{o} is a valid email anymore.

But what about %{s}? Should that preserve quoting so as to remain a valid
email? I am assuming that %{s} would never be used as an actual DNS label.
But maybe I'm wrong. The spec says, "The 's' macro expands to the <sender>
argument. It is an E-Mail address ...", which says to me that it should
remain a valid email address and preserve quoting.

It is possible that you have uncovered yet another errata (fortunately
just another obscure corner case).

I think we are missing a test case to ensure that %{o} (and %{s}) stays
unchanged across recursive queries.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
Stuart D. Gathman wrote:

> You could argue that a sender (who controls valid localparts)
> should avoid such quoted localparts to avoid any inconsistencies
> in receivers.

Oh yes, we all support the relevant "SHOULD avoid" in 2821bis,
and it is exactly the same "SHOULD avoid" as in RFC 2821.

> However, it *is* our job to specify the correct behaviour in
> the test suite

Yep. And get this right in the FAQ, the errata, and I-D.spf-eai.

From Julian's *old* line of arguments it was clear that removing
quotes while keeping backslashes makes no sense. Removing both
and then say "oops" to .do..ts. is acceptable (or maybe not, see
below).

Keeping both and then do the right thing for ".do..ts." as well
as "d\.o\\t\s" might be tricky. IOW I've no clue how to do the
right thing in these cases - but it is clearly possible, because
DNS permits any octet in a label.

So far all errata etc. went for .do..ts. and "oops". That does
not mean that "unquoting" + "unescaping" happens outside of SPF,
it only means that it happens pecisely once.

So "d\.o\\t\s" somehow arrived as d.o\ts in checkhost(), treated
as two labels "d" and "o\ts", and how implementations managed to
query DNS for a domain with label "o\ts" was their problem.

IMO they MUST get this right for conformance, there have to be
tests in the test suite checking that local part "d\.o\\t\s"
(or similar) works.

> I think you are saying that all quoting should be completely
> removed for %{l}

Well, actually I thought that's what you guys were saying after
I mentioned that "embedded dots" are allowed in DNS. You and
Julian among others did not like this, and from there we went
in the opposite direction, something already removed the quotes
from ".do..ts.", and an empty label in .do..ts. won't fly with
res_mkquery().

You cannot have quotes removed or kept as you see fit, if they
are ugly for ".do..ts."@ they are also ugly for "okay"@ or for
"back\\slash"@

Removing quotes while keeping quoted pairs is RFC 2617 magic,
one of several factors why RFC 2831 will be moved to HISTORIC,
the 2831bis author was annoyed, google for qdstr-val.

Therefore we got "remove quotes and unquote <quoted-pair>" -
until yesterday, before Julian found that this of course also
has some drawbacks. It was a shaky decision, but you did not
want "embedded dots", for a "keep as is" approach.

> The spec. says, "The 's' macro expands to the <sender>
> argument. It is an E-Mail address ...", which says to me that
> it should remain a valid email address and preserve quoting.

Yes, and 'l' is supposed to be the original local part, and
'h' the original helo, as broken as it might be if receivers
accept syntactically invalid (SMTP syntax) HELOs.

> It is possible that you have uncovered yet another errata
> (fortunately just another obscure corner case).

IMO this is the same old set of errata as it always was. If
we now want ".do..ts."@example as is for 's', and ".do..ts."
as is for 'l', then SPF implementations have to deal with
%{l}.%{d} in DNS queries, because that is perfectly possible.

But it's tricky, in "qu\"ot"@example the \" is a quoted-pair
because that is the only way to get a quote in a local part.

The obscure "semantical content" is <label>qu"ot<\label>.
I don't know how a resolver can be convinced to make a query
containing label qu"ot. It could well be "qu\"ot" as it was.

For combinations like "\u\n\n\e\c\e\s\s\a\r\y"@example or
"okay"@example the SPF implementations have to know that
this is unnecessary and okay, resp. The DNS API can't know
it, just double or insert backslashes everywhere to see why.

And the SPF test suite better checks that implementations
get this right, it is no rocket science, but obviously also
not trivial.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com
Re: Quoted pairs in the test suite [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Julian Mehnle wrote:
> > There's a difference between the _value_ of the localpart in a RFC
> > 2821 e-mail address and its _representation_ in the e-mail address.
>
> Yep, I know the fine print, 2822upd has this as "semantical content"
> IIRC. But in practice these things tend to vary:
>
> [...] I just want to point out that "doing the right thing" isn't
> always the same thing in different contexts.
>
> IIRC you said here that SPF implementations strip the quotes from
> <quoted-string> in a discussion about "embedded dots".

Just out of curiosity: do you have a reference to me saying that?
I suspect that this was in a different context.

> ***BUT***
>
> > this is exactly what MTAs should pass to SPF implementations.
>
> I don't see why MTAs should suddenly start to be smart. They get an
> address with a <quoted-string> as local part, and they pass this on to
> SPF with a clear directive "now do your magic, I want PASS or FAIL,
> a.s.a.p.".
>
> MTAs have no reason to figure an odd "semantical content" out, that's
> hardcore standardista stuff, not practical programming.
>
> SPF has to do this - something above checkhost() is fine, but not so far
> above checkhost() that it is outside of SPF.

Well, let's just say that it is in the interface between SPF and the
outside world. I see no reason at all for the SPF spec to describe the
specific steps for determining the semantical value of its input
parameters. (Actually, doing that would rather be bad specification
design and a layer separation violation.) This is the duty of RFCs 2821
and 2822(bis/upd/whatever), as they define the on-the-wire / on-the-disk
representation.

At best I can see adding an erratum saying that input parameters have to
be fully parsed, per RFCs 2821/2822(bis/upd/whatever), into their
semantical values before being passed to the SPF implementation.

> There is no "duty" for MTAs or other SPF users to parse a MAIL FROM
> address into its "semantical content".

Perhaps not, but there certainly isn't one for the SPF specification to
define how to do it.

> I'd even fear that nothing removes the reverse route in cases where it's
> present. RFC 4408 clearly says "strip it". It's the job of SPF to
> strip a reverse route, therefore it's also the job of SPF to deal with
> <quoted-string> + <quoted-pair>.

There's a difference between saying "strip it!" and saying "strip it, and
this is how you must do it".

If anyone wants to know how to do it, they must begin interpreting RFCs
2821/2822, not the SPF spec.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkil/CoACgkQwL7PKlBZWjvXIQCgnx3HaH7VhYS4cdV1USinZoaD
L8sAn13PlrHjuKjDC9Gv+pjDEkHNruix
=njtV
-----END PGP SIGNATURE-----


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com