Mailing List Archive

On Wed, 24 Mar 2004 spf@anarres.org wrote:

> SRS is to be performed only when the address is actually rewritten by a
> forwarding mechanism. This does not happen on a secondary MX. Therefore
> this is not currently an issue. This problem only arises for people who
> have implemented SRS to do unconditional rewriting of all (or too much)
> mail.

Unconditional rewriting is *the* most effective anti-spam tool at our
site. It stops 2000 spams / day which are spoofed bounces. This
is compared with 500 / day for bayesian content filtering and
200 / day for domain and IP blacklists (and a miniscule number for
SPF fail so far).

The unconditional rewriting will not be as necessary once SPF is
wide spread enough for me to just ignore bounces from non-SPF
compliant mailers.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Very few of our customers are going to have a pure Unix
or pure Windows environment." - Dennis Oldroyd, Microsoft Corporation

> From: Neil Brown
> Sent: Tuesday, March 23, 2004 11:18 PM
>
>
> On Tuesday March 23, sethg@GoodmanAssociates.com wrote:
> > >
> > > When I send to a mailing list, it doesn't go through a forwarder. It
> > > goes straight to the list.
> >
> > That's the case for most of us, but not everyone. One case is a domain
> > on a commercial hosting service whose outgoing mail server is shared by
> > many domains. Each domain lists the common SMTP server as a permitted
> > sender in its SPF record. If the hosting service implements SRS, the
> > hosting service's SMTP server must rewrite the MAIL FROM: into an SRS0
> > address because the return-path domain name is different from the domain
> > name of the SMTP server, thus it is a forward. This is a common enough
> > situation that it has to be considered.
>
> I don't see this.
> You only *have* to rewrite the MAIL FROM: if you are sending out using
> an IP address that the owner of the domain in the MAIL FROM: has not
> authorised using SPF.
> But in your example, you say that each domain has authorised that SMTP
> server via SPF. So no rewriting is needed.
> It has nothing (directly) to do with the domain that the SMTP server
> is in.

Ooops (really big oops)! Rewriting is not necessary in this case, so the
example that I gave is bad. It admittedly would be pretty unusual to have
mail forwarded to a mailing list, though not impossible. My proposal would
indeed send SRS0-signed MAIL FROM: addresses to _all_ mailing lists and this
would break some of them.

It doesn't appear that many mailing lists use the MAIL FROM: address rather
than the From: header to qualify posts, so I don't think this is a major
source of breakage. I have no idea of the actual number of lists that would
be broken by this, but my personal experience suggests that it would be
relatively few. If anyone has any idea of the total number or percentage of
mailing lists that operate this way and how difficult it would be for them
to change to the From: header, that would be useful information.

Therefore you are correct, I cannot claim that the proposed alternate scheme
causes _no_ breakage, but only that it causes a miniscule amount of breakage
compared to the present SPF+SRS. Getting a few mailing lists to change
their logic is minor compared to convincing every forwarder to do SRS1
rewriting and dealing with forwarders who don't.


> I am not protected from joe-jobs just because I implement SPF. I am
> only protected if everyone else implements SPF (and the more that do,
> the more I am protected).

Joe-jobs are bad for two reasons. First, it harms your reputation and is
akin to libel. Secondly, it typically results in a flood of bounces to your
domain, which are a real nuisance. I agree that protection from joe-jobs
will increase along with SPF adoption. However, unless SPF adoption reaches
100%, there will continue to be successful joe-jobs that will result in lots
of bounces. My alternate proposal gives you the same amount of protection
against joe-jobs that SPF currently does, because it does an SPF check at
every hop, but it also protects you from the resulting bounces. The
protection from bounces is virtually air-tight and you get it today, without
requiring anyone else to adopt anything.


>
> Yes. DSN's are a problem that SPF doesn't help with.
> Universal SRS would help, but I believe the cost is too high.

SRS signing of all outgoing mail would do more than help, it rejects all
bogus DSN's. I'm not sure why you say the cost is too high. The cost is
certainly much lower than requiring all forwarders to do SRS1 rewriting and
dealing with the reality that they won't all comply. If you don't find
_that_ cost too high, isn't a scheme that has a lower cost and provides more
benefit preferable?


>
> There are two features that SPF helps provide I believe:
> 1/ The MAIL FROM: address can be safely used as an address for
> bounces.
> 2/ The MAIL FROM: address can be used to reliably identify the sender,
> and so can be tested against personal whitelists and blacklists.
>
> Universal SRS doesn't help a lot with (2) as it doesn't provide the
> recipient an easy way to determine if the address is a probable sender
> of the mail item.

Just SRS signing outgoing messages does not help with this, but including
the originating gateway MTA local-part in the SRS signature does. As the
destination gateway MTA, you can then do an SPF check on the original
sender's domain and validate the originating gateway MTA as a permitted
sender. Furthermore, you can interpret the SPF result according to _your_
SPF policy.

With the present SPF+SRS, if there is a forwarder, you have to trust that
the forwarder did the SPF check properly and you have to accept their local
SPF policy. Since you, in general, have no way of knowing either of these,
you can't know how much trust to put in a particular forwarder's assertion
of your point #2 above. That is a key limitation of the present SPF+SRS
scheme. The alternate scheme that I proposed does not suffer from this
since the destination gateway MTA has all the tools it needs to verify the
mail source itself.


>
> The problem of identifying bounces to faked email, while very real, is
> separate from SPF.

What you are saying is that by design, the present SPF+SRS does not prevent
this type of abuse. The alternate scheme that I proposed does solve this
problem as well as achieve all the other benefits of SPF.


>
> I think the idea of putting a crypto-cookie in the headers of all
> outgoing mail is worth pursuing. I think it has been mentioned on one
> of these lists before, but I'm not sure if there was much conclusion.
>
> While the absence of a crypto-cookie does not guarantee a bad bounce
> (any more than "SPF returns Fail" guarantees a bad message), it's
> presence does strongly indicate a good bounce (much like "SPF returns
> Pass") and so other heuristics can be more aggressive at minimal cost.

The presence of a crypto cookie only tells you something if you can verify
that it is valid. In addition, you really want to verify it before the DATA
stage of the SMTP session, so the mail headers are not the best place for
it. That's the motivation for proposing a verifiable crypto cookie in the
MAIL FROM: of all outgoing mail. There are many possible ways to generate
and validate such a cookie, but two general approaches seem most promising:
a one-way hash plus a CBV or a private key signature with public key
verification. The present proposal that I posted uses a SHA-1 hash as a
crypto cookie, so it requires a CBV for the original sender verify it.

If doing CBV's is a deal-breaker, it is feasible to use a private key
signature as a crypto cookie that can be verified through a public key
published in the DNS. This avoids CBV's, but puts considerable
computational overhead on an SMTP server. I am somewhat agnostic on this
issue. However, the anti-spam community has long advocated putting more
burden on mail senders than on mail recipients as a disincentive to
spamming. That is why I posted the CBV version of the modified scheme as
opposed to the private/public key version. It is conceivable for both
methods to coexist, though I don't advocate that. That is, if a site
doesn't want callbacks, it does private key signing and publishes a public
key for verification. We would simply need a way to distinguish which
method was used.


<...>

> > > Infact, I'm leaning toward the idea of leveraging this fact
> > > to overcome
> > > what I see as the biggest problem with SPF.
> > > That problem is that mail forwarded by a non-SRS aware forwarder might
> > > be assessed wrongly by SPF.
> > > The possibly solution would be to have our customers register any
> > > sites that they expect forwarded mail from, and to explicitly
> > > whitelist those.
> >
> > If you do that, you will lose the ability to reject bogus DSN's since
> > some of the outgoing mail that you forward for a particular domain does
> > not have the SRS0 hash for you to verify.
> >
>
> I think you mis-understood me.
> Let me be more specific.
>
> Mail to neil@brown.name gets forwarded to neilb@cse.unsw.edu.au.
> The managers of brown.name don't use SRS or support SPF.
> So the mailserver at cse.unsw.edu.au is likely to get an SMTP
> connection from a host associated with brown.name (typically
> mx*.nic.anme) with MAIL FROM: <anybody@any.where>
> I would register with the cse mail server that I am expecting mail
> to be forwarded from neil@brown.name, and if it is receiving mail from
> somewhere such that SPF is returning other than 'Pass', and it sees a
> "RCPT TO:<neilb@cse.unsw.edu.au>", it will retry the SPF lookup with a
> responsible sender of neil@brown.name. Though brown.name doesn't have
> an SPF record, the standard guess will let this though with a higher
> SPF rating.

I did misunderstand, so thanks for clearing that up. I didn't realize that
the whitelist was an alternate source of domains to do an SPF query on.
Your proposal should work as well as SPF-guess works. That is, it will
often get it right, but not always. That's a whole lot better than always
getting it wrong for non-SPF forwarding sites, so it sounds like a
reasonable way to configure things.

Of course, I should point out that the alternate proposal does not require
rewriting and your case above would be handled without any whitelisting.
According to the proposal, when an SMTP-client connects to your SMTP-server,
that triggers a rDNS lookup, a SPF query on the RHS of the result and a SPF
test on the SMTP-client. If there is no SPF record, SPF-guess would
automatically be invoked, so it would do exactly what you suggested without
a whitelist.

If the SMTP-client domain has a SPF record, you can now trust that it is a
designated sender for its domain. If there were any forwarders, that is
really all you can trust and you can't assert much about the original source
of the message. At this point, you are in exactly the same state as with
the existing SPF scheme. If all you care about is whether you can send a
DSN, you can probably stop here.

However, if you want some assurance that MAIL FROM: really represents where
the mail came from, unless the SMTP-client is the originating gateway MTA,
you need to do a few more tests. If MAIL FROM: contains a SRS signed
address with the originating gateway MTA local-part, you could then do a SPF
query on the sender's domain to validate the originating gateway MTA as a
designated sender. If it passes, you now trust that the _claimed_
originating gateway MTA is a designated sender for the _claimed_ sender
domain. Finally, you do a CBV to verify that the _claimed_ originating
gateway MTA was the actual source of the message. This allows you to trust
the return path, even in the presence of forwarding.

--

Seth Goodman

On Wed, 24 Mar 2004, Seth Goodman wrote:

> With the present SPF+SRS, if there is a forwarder, you have to trust that
> the forwarder did the SPF check properly and you have to accept their local
> SPF policy. Since you, in general, have no way of knowing either of these,
> you can't know how much trust to put in a particular forwarder's assertion
> of your point #2 above. That is a key limitation of the present SPF+SRS
> scheme. The alternate scheme that I proposed does not suffer from this
> since the destination gateway MTA has all the tools it needs to verify the
> mail source itself.

Classical game theory would argue that if such a forwarder was performing
SPF incorrectly, it would be treated exactly as if it were a source of
spam in its own right. Indeed, the two are always indistinguishable. It is
therefore to be assumed that any valid forwarder will attempt to act in
its own best interest and perform SPF correctly.

S.

--
Shevek http://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

> From: spf@anarres.org
> Sent: Wednesday, March 24, 2004 4:30 PM
>
>
> On Wed, 24 Mar 2004, Seth Goodman wrote:
>
> > With the present SPF+SRS, if there is a forwarder, you have
> > to trust that
> > the forwarder did the SPF check properly and you have to
> > accept their local
> > SPF policy. Since you, in general, have no way of knowing
> > either of these,
> > you can't know how much trust to put in a particular
> > forwarder's assertion
> > of your point #2 above. That is a key limitation of the
> > present SPF+SRS
> > scheme. The alternate scheme that I proposed does not
> > suffer from this
> > since the destination gateway MTA has all the tools it
> > needs to verify the
> > mail source itself.
>
> Classical game theory would argue that if such a forwarder
> was performing
> SPF incorrectly, it would be treated exactly as if it were a source of
> spam in its own right. Indeed, the two are always
> indistinguishable. It is
> therefore to be assumed that any valid forwarder will attempt
> to act in
> its own best interest and perform SPF correctly.

Thank you for your reply. I am unfamiliar with classical game theory,
but I trust your assertion. Being more comfortable with a statistical
approach, I consider that there will be a stochastic variation in the
quality of implementations and local policy across a large number of
sites. Over the very long term and taken as an ensemble average, I
agree with your statements. However, it is not applicable for a
particular site at any specific instant in time.

Unfortunately, not every SPF implementation will be correct and perfect.
Some SPF local policies will make more sense than others. The most
broken implementations will likely be abused and become listed as spam
sources. However, until a site is actually abused, they will continue
to operate despite any flawed SPF assertions that they pass on.

Trusting that another site has done SPF checks to _your_ satisfaction
simply because they are not blacklisted does not sound like a reasonable
assumption. Which blacklists are used and whether they are used at all
is a matter of local policy at each forwarder, so there is no assurance
that even a blacklisted site will be denied SMTP access to the next hop.
Until you receive all the message headers, you don't know how many or
which forwarders handled the message. Under these conditions, what is
the basis for trusting the return-path before the DATA stage? I would
assert that unless the SMTP-client is the originating gateway MTA,
trusting the return-path with the current SPF+SRS scheme requires trust
in parties whose identities are unknown to you at the time you have to
make the decision.

On the other hand, if you provide the final recipient with the
information they need to do the verification themselves before the DATA
stage, you don't have to depend on someone else's implementation to be
correct. The final recipient can verify the return path information
themselves or trust the forwarders as they see fit. Given the
environment, I suggest that "trust but verify" is a reasonable behavior
for a site. Right now, the return-path does not carry sufficient
information for the destination gateway MTA to verify it.

My suggestion to remedy this is to SRS0 sign all outgoing mail and
include the originating gateway MTA local-part as part of the address.

--

Seth Goodman

On Wednesday March 24, dwmw2@infradead.org wrote:
> On Wed, 2004-03-24 at 16:17 +1100, Neil Brown wrote:
> > I think the idea of putting a crypto-cookie in the headers of all
> > outgoing mail is worth pursuing. I think it has been mentioned on one
> > of these lists before, but I'm not sure if there was much conclusion.
>
> I suspect that too many mailers return bounces without enough of the
> original message included. I asked if anyone had numbers to confirm or
> refute that, but nobody obliged.

I just got one :-( Content below if you are interested.

It does seem to make universal SRS more attractive, and the arguments
against it are hard to sustain.

I would just really like a fairly reliable "sender" address, and some
indication of how reliable it was. MAIL FROM seemed to be that, and
SPF seemed to give a good measure of reliability. But if people start
re-writing the MAIL FROM header whole-sale, and quite possibly in
subtly different ways, then that seems to put a big hole in what I
thought SPF would give me.


NeilBrown




Here is the body of the message.... Maybe if the crypto cooke were put
as a comment on the end of the Date: line .... no, that's just silly.
-----------------------------


Attention: neilb@cse.unsw.edu.au


A problem was found in an Email message you sent.
This Email scanner intercepted it and stopped the entire message
reaching its destination.

The problem was reported to be:

the W32/Netsky.p@MM virus !!!


Please contact your IT support personnel with any queries regarding this
policy.


Your message was sent with the following envelope:

MAIL FROM: neilb@cse.unsw.edu.au
RCPT TO: dickson@vector.com.cn

... and with the following headers:

---
MAILFROM: neilb@cse.unsw.edu.au
Received: from unknown (HELO vector.com.cn) (218.18.130.204)
by ns1.vector.com.cn with SMTP; 25 Mar 2004 03:31:27 -0000
From: neilb@cse.unsw.edu.au
To: dickson@vector.com.cn
Subject: Mail Delivery (failure dickson@vector.com.cn)
Date: Thu, 25 Mar 2004 11:24:12 +0800
MIME-Version: 1.0
Content-Type: multipart/related;
type="multipart/alternative";
boundary="----=_NextPart_000_001B_01C0CA80.6B015D10"
X-Priority: 3
X-MSMail-Priority: Normal


---

> From: spf@anarres.org
> Sent: Wednesday, March 24, 2004 4:30 PM
>
>
> On Wed, 24 Mar 2004, Seth Goodman wrote:
>

<...>

>
> Classical game theory would argue that if such a forwarder was performing
> SPF incorrectly, it would be treated exactly as if it were a source of
> spam in its own right. Indeed, the two are always indistinguishable. ...

I had an additional comment on this that I forgot to mention. Though the
original spammer and the forwarder may be indistinguishable from a gaming
perspective, they are treated differently in terms of spam reporting and
blacklisting. When you report a spam to blacklist, they attempt to locate
the spam source. Though they send abuse reports to the spammer's ISP,
forwarders and in some cases upstream providers, the responsible lists try
to only list the IP of the machine that originated the spam. Though some
blacklists intentionally "widened the net" by listing an entire provider, an
upstream or a forwarder and some made listings for arbitrary or malicious
reasons, those lists fared poorly due to high false positive rates. This
used to be a serious problem, but it has largely corrected itself due to
people choosing the responsible lists.

If this is all a misconception on my part, hopefully the forwarders out
there will gently correct and educate me.

--

Seth Goodman

On Thu, 2004-03-25 at 14:56 +1100, Neil Brown wrote:
> On Wednesday March 24, dwmw2@infradead.org wrote:
> > I suspect that too many mailers return bounces without enough of the
> > original message included. I asked if anyone had numbers to confirm or
> > refute that, but nobody obliged.
>
> I just got one :-( Content below if you are interested.

Bear in mind that the mail it's 'returning' probably didn't have a
message-id in the first place. It's not _necessarily_ a case of a mailer
bouncing a mail and omitting its Message-ID.

Looking at the other headers which were included, I'd guess you _would_
have seen a Message-Id: there if the original message had had one.

TBH I wish Message-ID: had been made a 'MUST', but it's not. Still, I
elect not to accept any messages without it. It's largely crap -- if
it's not a virus it's often some broken MTA with broken References: and
top-posting.

> It does seem to make universal SRS more attractive, and the arguments
> against it are hard to sustain.
>
> I would just really like a fairly reliable "sender" address, and some
> indication of how reliable it was. MAIL FROM seemed to be that, and
> SPF seemed to give a good measure of reliability.

Sender verification callouts give you _some_ indication of how reliable
it is, and have done so for years. You check the sender address will
accept a bounce. The recent trend to refusing MAIL FROM:<> to addresses
which _never_ send mail makes that indication _more_ reliable, and doing
SRS on all outgoing mail so that even 'real' addresses can refuse MAIL
FROM:<> makes it even more reliable.

> But if people start re-writing the MAIL FROM header whole-sale, and
> quite possibly in subtly different ways, then that seems to put a big
> hole in what I thought SPF would give me.

Does it? SPF only looks at the domain, and nobody's really rewriting the
_domain_ part of their own outgoing mail, surely?

(OK, I am -- but that's just because I wanted more diagnostic
information in the address than the SRS scheme or the 64-char limit
permitted, and I was willing to overcome my distaste for wildcard MX
records to achieve that. And I don't rewrite the domain in ways that it
affects SPF.)

I think SPF is seriously misguided, and I'm trying to achieve its aim in
other ways -- but I'm not setting out to actively break it, and I don't
think I've done so. I think it's broken enough already.

--
dwmw2

> From: Neil Brown
> Sent: Wednesday, March 24, 2004 9:57 PM
>
>

<...>

> It does seem to make universal SRS more attractive, and the arguments
> against it are hard to sustain.

Thanks for saying so. I am advocating this as a way to gain extra benefits
while reducing the rather large breakage caused by the present SPF+SRS
scheme. I am completely in favor of the goals behind SPF, but the design
decision of doing an SPF query on the MAIL FROM: RHS domain set off an
unfortunate chain of events. This first caused the need for SRS0 to protect
the rewritten address, which caused the need for SRS1 to avoid the second
forwarder becoming an open relay. With all this extra baggage, the end
recipient still cannot have high confidence that an SPF pass means that the
return-path is not forged and there is no protection against forged DSN's at
the originating gateway MTA since it's MAIL FROM: is not cryptographically
protected.

Let's take a step back and examine what would happen if we made one small
change to that one particular design choice. If we instead choose to get
the domain for the SPF query from the rDNS of the IP of the SMTP-client,
here's what would happen:

1) The MAIL FROM: is no longer the source of the domain for the SPF query,
so we _never_ need to rewrite MAIL FROM:. This means that there is no
breakage of existing mail practice whether or not a site publishes SPF
records, does SPF checks, both or neither. This will greatly ease the task
of convincing the majority of major carriers to adopt SPF, since no changes
are required to their forwarding practices.

2) Non-SRS sites in the chain no longer have any effect on correct
end-to-end operation. The present scheme breaks down if any of the
forwarders do not implement SRS unless special measures are taken at
SPF-compliant sites. These measures will probably amount to custom
whitelists of sites to avoid doing SPF checks on, which will be a royal PITA
for site administrators to keep up with.

3) Since virtually all well-configured SMTP-servers today perform an rDNS on
the IP of the SMTP-client as a first heuristic test, we have introduced no
extra overhead. In fact, they often go further than this by requiring a
match to an A record. Some people consider this a best commercial practice.

4) Many SMTP-servers already reject connections from IP addresses that don't
return proper FQDN's. The SPF gurus would have to decide what SPF results
to return for each possible type of rDNS failure.

5) Since rDNS is just as reliable an indicator of domain as the RHS of MAIL
FROM:, the amount of confidence we have in the resulting SPF query and
result is unchanged from the present system.


This is a very large improvement for one very small change in usage. I
realize that the existing SPF specs and implementations are pretty far
along. However, if people think about this carefully, the benefits from
this change far outweigh the costs. In fact, with the challenge of getting
the present SPF+SRS adopted, I would argue that given the option of making
this change, it does not seem reasonable to continue on the current path.


>
> I would just really like a fairly reliable "sender" address, and some
> indication of how reliable it was. ...


That is why we are all here. This is a second place where a particular
design decision in SPF led to a less than optimal outcome with regard to
Neill's reasonable expectation. The particular architectural feature that I
refer to is that each SMTP-server would do an SPF query, test the
SMTP-client against the SPF policy returned by the query, and each
subsequent site trusts the SPF assertion made by the previous site. There
was no provision for the destination site to verify the SPF credentials of
the originating gateway MTA and thus whether or not MAIL FROM: is a forgery.
This is a problem for a couple of reasons.

First, if there even one non-SPF compliant forwarder in the chain who is
whitelisted at the next hop because they are trusted as non-spammers, that
breaks the chain of SPF assertions. Unfortunately, the destination site has
no way to know that this has occurred. All they know is that the
SMTP-client that delivered the message to them generated an SPF pass, so the
MAIL FROM: inappropriately "inherits" the trust that would result from an
unbroken chain of proper SPF checks. Unfortunately, this is very difficult
to work around, even after you have all the message headers. There is no
reliable way to know if a site implements SPF. Publishing an SPF record has
no relation to doing SPF checks.

The second problem created by the forcing the final SMTP-server to assign a
level of trust to the MAIL FROM: that it ascertained from the SPF check on
the SMTP-client is that the quality of the SPF implementations and the local
SPF policies used by intermediate sites are unknown to the final
SMTP-server. In an environment where all sites implemented SPF _and_ they
all met minimum implementation standards _and_ they all had reasonable local
SPF policies (would we all agree on what those are?), then there would be no
problem. Without that fictitious environment, you are left with an unknown
level of confidence that an SPF pass asserts whether or not the return-path
is forged.

Fortunately, it is possible to make another change to the SPF framework that
will allow the destination SMTP-client to verify the MAIL FROM: address
without accepting the assertions made by unknown third parties. That change
happens to dovetail very nicely with the change that I suggested above to
obviate the need for sender rewriting. The specific change is as follows:
all originating gateway MTA's must SRS0 sign the MAIL FROM: for all outgoing
mail that originates from their private networks, and the SRS0 signature
must additionally contain the local-part of the originating gateway MTA
address. One possible syntax for this would be:

MAIL FROM:<SRS0=HHHH=TT=sender-local-part=sender-domain=
MTA-local-part@MTA-domain>

where
"HHHH" is a SHA-1 hash computed over the remainder of the
MAIL FROM: address _including_ MTA-domain

"TT" is a timestamp

"sender-local-part" is the local part of the original
sender's address

"sender-domain" is the domain part of the original
sender's address

"MTA-local-part" is the local part of the originating
gateway MTA's address

"MTA-domain" is the domain part of the originating
gateway MTA's address


Now let's see what happens to SPF operation as a result of this change.

1) The entire return-path is protected by the hash function. If anyone
along the path attempts to modify the return-path, the hash won't verify.

2) It is for all practical purposes impossible to forge a completely new
return-path with a correct hash.

3) All forwarders along the message path can perform SPF checks by getting
the domain for SPF query from the rDNS of the IP of the SMTP-client. SPF
continues to operate exactly as it does now through the message chain.

4) At the end of the message chain, the SMTP-server can do an SPF query on
"sender-domain" to verify that "MTA-local-part@MTA-domain" is a designated
sender. Even if there are non-SPF compliant forwarders in the message path,
the final SMTP-server can still verify this key piece of information
independently.

5) At the end of the message chain, the SMTP-server can further verify that
the source of this message was "MTA-local_part@MTA-domain" by doing a CBV on
the MX for "MTA-domain". Since the MAIL FROM: contains an SRS0 hash, if the
CBV returns "recipient OK", you know that this message originated from that
site. This test also obviates the need to trust the assertions of unknown
third parties. It also guarantees that if you later have to generate a DSN,
it will be accepted.

6) This mechanism does not in any way hinder the use of SPF checks at each
hop in the message path, as long as they are done based on rDNS of the IP of
the SMTP-client. Neither does it depend on any of the forwarders actually
doing SPF checks, though that would be highly desirable. The final
SMTP-recipient can always verify MAIL FROM: independently, according to
_its_ local SPF policy and have a very high degree of confidence in the
result.

7) DSN's are handled exactly as they are now and go straight to the
originating gateway MTA, as opposed to going to the second forwarder, where
it is relayed back to the first forwarder where it is relayed back to the
originating gateway MTA. This does not break any existing mail practices.

8) Bogus DSN's cannot have correct hash values, so they will be rejected
immediately after MAIL FROM: with permanent errors. This makes you immune
to joe-job bounces, bounce spam and virus spew posing as a DSN. Even
Microsoft mail clients do not retain the MAIL FROM: SMTP information in
their headers, so it will not be possible for viruses to harvest valid SRS0
return-paths to attack.


This is a large set of benefits due to a relatively small change in
protocol. The main benefits are allowing the final SMTP-server to validate
the MAIL FROM: information regardless of who was in the message path, giving
senders immunity from bogus DSN's and bounces from joe-jobs plus not
breaking any existing mail practices. There is one caveat to that last
statement: as Neill pointed out, mailing lists that qualify their posts by
MAIL FROM: instead of the From: header will be broken and will have to
change their logic.

Given all of the above advantages, why would we _not_ be willing to
reconsider the current path we are on in favor of a more rational one that
causes less breakage, gives more benefits, is easier to adopt and works well
even if it is only partially adopted?

--

Seth Goodman

Hi,

I've been thinking a lot about the "universal-SRS" idea and am
definitely warming to it.

However I just noticed something else that it could break.

Suppose I have an address like neil@brown.name (which I do), and
suppose I forward mail from there to neilb@cse.unsw.edu.au (which I
also do).

Now suppose I chose to send mail so that it appeared to be "From"
neil@brown.name (which I do very occasionally), and suppose that I
sent this through an MTA which is authorised for "brown.name" in what
ever way is appropriate (e.g. SPF) (which I don't think I can for
brown.name, but I definitely can for some other forwarders),

And suppose that this mail I send results in a DSN that I really want
to see.
It will get to the authorised MTA address for neil@brown.name
(possible wrapped up in SRS if the MTA does that) and the MTA will
forward it to neilb@cse.unsw.edu.au. But obviously that MTA cannot
wrap the to address in an SRS that we will accept, so my (cse) MTA
will get Mail From:<>, Rcpt to:<neilb@cse.unsw.edu.au> and my MTA
might feel inclined to reject it. Which I would rather it didn't.

Now, for myself, I can just always send mail from my primary email
address. I don't have a problem with that. But I don't think I can
enforce that on all my customers.
And that means that even if I use SRS on all outgoing mail, I cannot
require it on all incoming DSN mail.

One way around this that I can see is to encourage forwarders to
rewrite <> addresses as (say) bounceto-neil@brown.name when forwarding.
Then the mail would be accepted.

And given that avoiding the requirement that forwarders rewrite the
return address was one of the motivations for universal SRS, this
doesn't seem helpful.

The only alternative that occurs to me is to require local addressees
to register any forwarders that forward to them, so that mail from
those forwarders gets white-listed. This alternative has the benefit
of being completely local (I don't depend on the behaviour of other
sites), and handles both sides of the forwarding problem (forwarded
originals and forwarded bounces). It is hardly low-impact though.

Comments?

NeilBrown

> From: Neil Brown
> Sent: Wednesday, March 31, 2004 7:07 PM
>
>
>
> Hi,
>
> I've been thinking a lot about the "universal-SRS" idea and am
> definitely warming to it.

Glad to hear it. You're definitely not alone. Meng previously suggested
that we call this scheme "Signed Envelope Sender" (SES) to distinguish it
from SRS, so I will refer to it that way from now on.

>
> However I just noticed something else that it could break.
>
> Suppose I have an address like neil@brown.name (which I do), and
> suppose I forward mail from there to neilb@cse.unsw.edu.au (which I
> also do).
>
> Now suppose I chose to send mail so that it appeared to be "From"
> neil@brown.name (which I do very occasionally), and suppose that I
> sent this through an MTA which is authorised for "brown.name" in what
> ever way is appropriate (e.g. SPF) (which I don't think I can for
> brown.name, but I definitely can for some other forwarders),
>
> And suppose that this mail I send results in a DSN that I really want
> to see.
> It will get to the authorised MTA address for neil@brown.name
> (possible wrapped up in SRS if the MTA does that) and the MTA will
> forward it to neilb@cse.unsw.edu.au. But obviously that MTA cannot
> wrap the to address in an SRS that we will accept, so my (cse) MTA
> will get Mail From:<>, Rcpt to:<neilb@cse.unsw.edu.au> and my MTA
> might feel inclined to reject it. Which I would rather it didn't.

Let's first assume that you sent the original message by logging into the
forwarder at which you have the neil@brown.name account. In this case, the
From: and Reply-To: will be neil@brown.name. Let's say that this forwarder
also uses SES. Since the domain brown.name has a valid MX record, the
forwarder's MTA will set the return-path on your outgoing message to:

SRS0=HHHH=TT=brown.name=neil@brown.name

The domain name appears twice in this address, which is wasteful, but this
is necessary if we want to be compatible with SRS. If the forwarder
chooses, they could alternatively set the return-path to:

SRS0=HHHH=TT=brown.name=neil@forwarder

If a DSN comes back to either of these addresses, the address will be
unwrapped and DSN delivered to the mailbox for neil@brown.name as the final
destination. The DSN is now a regular message in that mailbox and is no
different than any other message. Since this is a forwarding account, the
DSN will be forwarded as a regular message, not as a null-sender DSN, back
to your university account. Using SES in this case does not break anything.

Let's look at a much more elaborate case and see if things still hold
together. You send mail from your university account but make it look like
it came from your personal account. Let's further assume that all the MTA's
use SPF+SRS and your university outgoing MTA uses SES. To make things more
interesting, the recipient address that you send the message to is a
forwarder. Here is the message flow:

1) In your mail client at the university, you change the settings to:

From: neil@brown.name
Reply-To: neil@brown.name

and send the message through the account neilb@cse.unsw.edu.au

2) The university MTA uses SES and sets the return-path to:

SRS0=HHHH=TT=cse.unsw.edu.au=neilb@cse.unsw.edu.au

3) Your university MTA starts an SMTP session with "forwarder" and gives the
SRS0 address above as the MAIL FROM: address. "forwarder" does an SPF query
for "cse.unsw.edu.au" and verifies that the SMTP-client is a designated
sender for that domain. The SPF result is pass, so it accepts the message.
Being a forwarder, it changes the To: address to the final recipient
address. Since it sees an SRS0 address in the return-path and it is an
SRS-compliant forwarder, it rewrites the return-path to:

SRS1=HHHH=cse.unsw.edu.au==HHHH=TT=cse.unsw.edu.au=neilb@forwarder

4) The forwarder starts an SMTP session with the final recipient's MTA and
uses the SRS1 address above as the MAIL FROM: address. The SPF result is
pass, so it accepts the message. After a few days, the MTA cannot make
final delivery and gives up. The MTA starts an SMTP session with the MX for
"forwarder" to send a DSN.

5) Since the MX at "forwarder" sees MAIL FROM:<>, it does an SPF query on
the HELO name and the SPF result is pass. It responds "OK" and gets the
RCPT TO:<SRS1_address_from_step_3>. It verifies the SRS1 hash and then
accepts the DSN. It unwraps the return-path back to the SRS0 address in
step 2 above and starts an SMTP session with the MX for "cse.unsw.edu.au".

6) Since the MX at "cse.unsw.edu.au" see MAIL FROM:<>, it does an SPF query
on the HELO name and the SPF result is pass. It responds "OK" and gets the
RCPT TO:<SRS0_address_from_step_2>. It verifies the SRS0 hash and then
accepts the DSN. It unwraps the SRS0 address and delivers the DSN to the
"neilb" mailbox.

7) If the mail is successfully delivered to the final recipient in step 4
instead of failing, the recipient will see "From: neil@brown.name"
displayed. If the recipient replies, the mail will be addressed "To:
neil@brown.name", which will then be forwarded to your university account.
However, if they look at the top of the detailed headers, they will see:

Return-Path:<SRS1=HHHH=cse.unsw.edu.au==
HHHH=TT=cse.unsw.edu.au=neilb@forwarder>


As you can see, SES does not interfere with either SPF or SRS and can play
with or without them. Of course, this would all be a lot simpler and cause
less breakage if we didn't do sender rewriting. If we didn't have to worry
about compatibility with SRS, the return path would be shorter, particularly
the local part. For example, your university MTA could sign your
return-path as:

SES0=HHHH=TT=neilb@cse.unsw.edu.au

This is a real pity since SES gives the recipient more confidence in the
origin of the email than SPF+SRS without any of the breakage and doesn't
require anyone else to cooperate.

--

Seth Goodman

On Fri, 2004-04-02 at 00:23 -0600, Seth Goodman wrote:
> > And suppose that this mail I send results in a DSN that I really want
> > to see.
> > It will get to the authorised MTA address for neil@brown.name
> > (possible wrapped up in SRS if the MTA does that) and the MTA will
> > forward it to neilb@cse.unsw.edu.au. But obviously that MTA cannot
> > wrap the to address in an SRS that we will accept, so my (cse) MTA
> > will get Mail From:<>, Rcpt to:<neilb@cse.unsw.edu.au> and my MTA
> > might feel inclined to reject it. Which I would rather it didn't.

This is a real issue, yes.

Seth, I think you're misunderstanding. Imagine the case of a .forward
file redirecting mail to dwmw2@ftp.uk.linux.org -> dwmw2@infradead.org

Something sends mail from dwmw2@ftp.uk.linux.org which bounces. The
bounce is delivered to ftp.uk.linux.org which tries to forward it to
dwmw2@infradead.org. This is now a bounce being delivered to the 'raw'
address dwmw2@infradead.org, and it's rejected.

One solution for this is to use an address other than the 'raw' address
for forwarding to -- 'dwmw2+bounce@infradead.org', for example.

It's not perfect, but it's a lot less broken than the alternatives I've
found so far.

--
dwmw2

> From: Seth Goodman
> Sent: Friday, April 02, 2004 12:23 AM
>

Responding to and correcting my previous post.

>
> > From: Neil Brown
> > Sent: Wednesday, March 31, 2004 7:07 PM
> >
> >
> >
> > Hi,
> >
> > I've been thinking a lot about the "universal-SRS" idea and am
> > definitely warming to it.
>
> Glad to hear it. You're definitely not alone. Meng previously suggested
> that we call this scheme "Signed Envelope Sender" (SES) to distinguish it
> from SRS, so I will refer to it that way from now on.
>
> >
> > However I just noticed something else that it could break.
> >
> > Suppose I have an address like neil@brown.name (which I do), and
> > suppose I forward mail from there to neilb@cse.unsw.edu.au (which I
> > also do).
> >
> > Now suppose I chose to send mail so that it appeared to be "From"
> > neil@brown.name (which I do very occasionally), and suppose that I
> > sent this through an MTA which is authorised for "brown.name" in what
> > ever way is appropriate (e.g. SPF) (which I don't think I can for
> > brown.name, but I definitely can for some other forwarders),
> >
> > And suppose that this mail I send results in a DSN that I really want
> > to see.
> > It will get to the authorised MTA address for neil@brown.name
> > (possible wrapped up in SRS if the MTA does that) and the MTA will
> > forward it to neilb@cse.unsw.edu.au. But obviously that MTA cannot
> > wrap the to address in an SRS that we will accept, so my (cse) MTA
> > will get Mail From:<>, Rcpt to:<neilb@cse.unsw.edu.au> and my MTA
> > might feel inclined to reject it. Which I would rather it didn't.
>

<...>

> If a DSN comes back to either of these addresses, the address will be
> unwrapped and DSN delivered to the mailbox for neil@brown.name as
> the final
> destination. The DSN is now a regular message in that mailbox and is no
> different than any other message. Since this is a forwarding account, the
> DSN will be forwarded as a regular message, not as a null-sender DSN, back
> to your university account. Using SES in this case does not
> break anything.

Neil and David are right and the explanation I gave above is rather
brain-dead. I appreciate David pointing out my error. I must remind
myself: engage brain before typing. The .forward file prevents the message
from being delivered to the mailbox and causes it to be passed on intact.
The bounce will be forwarded with MAIL FROM:<>, and since it has an unsigned
RCPT TO: address, it will be rejected at the SES host, just as Neil pointed
out.

Neil and David each proposed a solution, and the two are nearly identical.
Neil proposed a special signed reverse address format that included a hash
to be used from forwarding accounts. This is an extremely clever way to
avoid any administration, once the mechanism is set up to generate the
addresses. David proposed a specific, secret address at the SES host at
which SES checking will not occur. They are functionally very similar.
Both act as whitelisted addresses for which SES checking will not be done,
but there is a slight difference.

In David's case, if the address were ever to be released or discovered, he
can just change it. In Neil's case, the address remains good until you
change the hash secret. Because of that, it would be best in Neil's
implementation to use a per user secret rather than a single hash secret for
the whole domain. The easiest way to implement a per user secret is to
simply use each user's password as their hash secret. That way, if a
whitelisted address does get discovered, all the user has to do is change
his password and the old whitelisted address will no longer work. It also
affects only that user. It remains a low administration solution.

SES works best, in general, if each user has their own hash secret. When a
recipient does a CBV to verify the return-path and the hash secret is
different for each user, a valid hash verifies the user as well as the
domain. That is something that SPF cannot currently do and it is valuable.

At first, it seems like a whitelisted address is a potential conduit for
spam. In fact, it just allows you to receive the mail in whatever form it
is from another account. It doesn't make anything better or worse. SES can
only protect a host when that host signs all outgoing messages. If the host
doesn't do this, it is vulnerable to bounce spam. Once the host accepts a
message, the game is essentially over, just like with SPF. Forwarding it to
another host cannot make it better nor does it make it worse. If you have
an account that is not protected by SES, SPF or other forgery prevention
methods and you are going to accept the mail in that mailbox, it doesn't
matter if you pick up the mail directly or forward it to a whitelisted
address on another host. The contents of the mailbox are the same either
way.

On the other hand, if the forwarding host does SES, it is protected from
bounce spam and forwarding from that account presents no problem, as the
account has the same properties as the host you are forwarding to. Again,
the act of forwarding does not change the forgery detection properties of a
host. It either does SES, SPF and/or other tests or it doesn't.

The longer example I gave is still correct, I believe, because the traffic
originated from a host that was SES protected. That example just showed
that SES can run with or without SPF+SRS and it does not interfere with
their operation.

--

Seth Goodman

On Friday April 2, sethg@GoodmanAssociates.com wrote:
>
> Neil and David each proposed a solution, and the two are nearly identical.
> Neil proposed a special signed reverse address format that included a hash
> to be used from forwarding accounts. This is an extremely clever way to
> avoid any administration, once the mechanism is set up to generate the
> addresses. David proposed a specific, secret address at the SES host at
> which SES checking will not occur. They are functionally very similar.
> Both act as whitelisted addresses for which SES checking will not be done,
> but there is a slight difference.

There is another difference.
My proposal embedded the address of that the mail was originally to, and
hence the address that the forward is (arguable) from in the
new recipient address.
The intention (I'm not sure if I made it clear) was that the receiving
MTA would perform an SPF check on *that* address to see if the
smtp-client was authorised.
i.e.
my MTA receives a message from the MX for brown.name with
MAIL FROM:<>
RCPT TO:<RRS+HHHH=brown.name=neil=neilb@cse.unsw.edu.au>
(or something like that).

As it is FROM:<> but not to an SES address, we could Fail, but instead
as it is an RRS address, SPF is tried for neil@brown.name,
extracted from the address. This returns Pass which Trumps the Fail.

Further, if my MTA receives a message from the MX for brown.name with
MAIL FROM:<someone@somewhere>
RCPT TO:<RRS+HHHH=brown.name=neil=neilb@cse.unsw.edu.au>

SPF against someone@somewhere returns Fail or at best Neutral.
Because it is an RRS address, SPF is tried for neil@brown.name. This
returns Pass which Trumps the Fail or Neutral.

So the fact that we are embedding the intermediate address in the new
recipient address means that the MTA can do SPF processing and get a
useful result for *any* forwarded mail.

Blending this with David's proposal, I realise that a crypto-signature
isn't needed. As long as your mail system allows you to create
arbitrary local address (which mine does and I believe qmail does and
there are probably others), all you needs is:

User creates a new local address which ends with (say)
RRS+localpart+domain
It will probably being with their username, but can contain anything
else they like. e.g.
neilb-secret+RRS+neil+brown.name
They then register this as the address that localpart@domain will
forward to.

Their MTA should do normal SPF checking, but if it sees a RCPT TO
address which ends "RRS+(.*)+(.*)", it does an SPF check on $1@$2 and
if that gives a "better" result, use that instead.

This, I think, is a neat way to overcome the "forwarding problem"
without requiring other people to use SRS. Once I convince all my
customers to modify any addresses that forward to them, I can start
rejecting SPF=Fail messages safely, all without any reference to, or
dependence on, other sites.

(Note that I'm not against forwarded implementing SRS if they want
to. It might make their customers lives easier. But requiring all
forwards, big and small, to implement SRS is a bit too much I think).

NeilBrown

> From: Neil Brown
> Sent: Monday, April 05, 2004 12:21 AM
>
>
> On Friday April 2, sethg@GoodmanAssociates.com wrote:
> >
> > Neil and David each proposed a solution, and the two are nearly
> >identical.

I do think Neil's proposal has lower administrative effort, though there are
a couple of points worth noting below. Both Neil's and David's workarounds
essentially create a secret back door address for a non-SRS forwarding
account to send mail to an SES (or SPF) host. In David's proposal, there is
one secret local forwarding address per customer he sets up manually, though
it could certainly be automated. In Neil's proposal, the customer uses an
automated mechanism to generate a unique forwarding address for each
forwarding account.

<...>

>
> There is another difference.
> My proposal embedded the address of that the mail was originally to, and
> hence the address that the forward is (arguable) from in the
> new recipient address.
> The intention (I'm not sure if I made it clear) was that the receiving
> MTA would perform an SPF check on *that* address to see if the
> smtp-client was authorised.

An incoming SPF check on a secret address won't tell you much, though you
are certainly free to do it. In both proposals, the RCPT TO: address is a
secret. As long as the address remains secret, mail arriving for that
recipient can only come from the place that you personally set up. The only
thing an SPF check would tell you is whether the secrecy of the address has
been compromised and the mail originated from somewhere other than the
forwarding host. If that does happen, it will become obvious very quickly
to your user without any SPF checks.


> So the fact that we are embedding the intermediate address in the new
> recipient address means that the MTA can do SPF processing and get a
> useful result for *any* forwarded mail.

You can do an SPF check, but as I pointed out above, it gives you virtually
no additional information.


<...>

> This, I think, is a neat way to overcome the "forwarding problem"
> without requiring other people to use SRS.

Yes, it really is. Anyone who wants to reject mail with SPF will need to
implement some kind of "back door" for non-SRS forwarders, which I believe
will be the majority of them. This solution is just as useful for them as
for anyone who wants to do SES.


> Once I convince all my
> customers to modify any addresses that forward to them, I can start
> rejecting SPF=Fail messages safely, all without any reference to, or
> dependence on, other sites.

I agree. Let me suggest a couple of successive refinements to your
proposal. I previously suggested that SES works best if each user has their
own hash secret, rather than a common domain secret. Since the RRS
addresses are static and have no timestamp that allows them to automatically
expire, they are vulnerable to being discovered and used in replay attacks.
If this does occur, the user would have to change their hash secret, then
create and configure new forwarding addresses for all of their forwarding
accounts. If your setup is good enough that this can't happen, you may not
care to worry about it.

However, there is a simple way to limit the possible damage to a single
forwarding account. That is to use a per forwarding domain salt value to
change the hash result in the RRS address. This permits you to change an
individual forwarding address without affecting any other forwarding
addresses for a particular user. The salt can be anything, so I propose to
use a sequential number at the time you generate the RRS address for a
particular forwarding account.

Once generated, you keep track of the salt value for each particular user
and particular account within each forwarding domain. If one of the user's
secret forwarding addresses is discovered, all they need to do is get a new
salt value to generate a new forwarding address for that one forwarding
account. All you do is assign the next unused salt value for the forwarding
domain for that user and calculate a new hash using the user's hash secret.
Since the salt value will force a unique hash value for each account at the
forwarding domain, you no longer need to include the local part of the
forwarding address. Besides, only the forwarding domain is needed for an
SPF query, should you feel the need to do one. The shortened format would
be:

RRS=HHHH=SS=forwarder_domain=local_part@domain

where SS = salt value consisting of two
base-64 digits

This gives 4K possible salt values per forwarding domain per user. It also
avoids exposing the full forwarding account address. After receiving a
message with a RCPT TO: in this format, do an SPF check, then lookup the
salt value under the user's account and forwarding domain for validity, and
finally lookup the user's hash secret and verify the hash value. This is a
lot of work, but fortunately we can reduce it.

Since the above RRS address is static, it can become a local alias on your
MTA to the user's account, as in David's proposal. In this case, the salt
value is not needed since an address can be verified by checking for its
existence. The "HHHH" value can be generated by successive hashing, so
there is no limit to the number of forwarding accounts. With this
implementation, the format would be:

RRS=HHHH=forwarder_domain=local_part@domain

where HHHH = a database key under the local user that represents
the local part of a forwarding account address; it
is a static value that is generated once for a given
forwarding account, or whenever a new RRS address
is desired for that forwarding account; it consists
of the first four base-64 digits of the SHA-1 hash
of the most recently generated RRS address for this
user prepended with the user's hash secret;

These addresses are aliases to the user's account on your MTA. Upon
receiving a message with this RCPT TO: format, do an SPF check, then check
for the existence of the RCPT TO: address and accept or reject the message
based on that.

If you agree that SPF checks on these special addresses don't buy you much,
the address can be further simplified since you don't need the forwarding
domain. In this case, we are left with:

RRS=HHHH=local_part@domain

where HHHH = a database key under the local user that represents
the FQDN of a forwarding account; it is a static
value that is generated once for a given forwarding
account, or whenever a new RRS address is desired
for that forwarding account; it consists of the
first four base-64 digits of the SHA-1 hash of the
most recently generated RRS address for this user
prepended with the user's hash secret;

These addresses are aliases to the user's account on your MTA. Upon
receiving a message with this RCPT TO: format, regardless of the MAIL FROM:
address, check for the existence of the RCPT TO: address and accept or
reject the message based on that. Now we have a very lightweight back door
for forwarding accounts.



>
> (Note that I'm not against forwarded implementing SRS if they want
> to. It might make their customers lives easier. But requiring all
> forwards, big and small, to implement SRS is a bit too much I think).

SRS doesn't break anything you've suggested, and I certainly agree that
expecting most forwarders to implement SRS is unrealistic at best.

--

Seth Goodman