Mailing List Archive

Michael Deutschmann wrote:
> Aside from being more fleshed out, one change is that I've removed the
> whitelist probing feature I included in my last TENBOX/E proposal. While
> it would be useful, I realized that the "whitelisting" needed by a
> forwarder is different from that wanted by a mailing list, so rather than
> add complexity to deal with the difference, I decided to leave that for a
> further, layered protocol.

As I'm not well acquainted with TENBOX, I don't understand why an alias
expansion should differ from a list expansion, as far as whitelisting is
concerned. Although lists should be characterized properly, they share
various operative similarities with aliases.

The draft contains various interesting ideas and concepts, some of which I
had already read on this list, but assembled them in my head in a somewhat
different way (see mine of Wed, 23 Jan 2008 16:13:51 +0100 "[spf-discuss]
Re: Relay".) I'll post a hopefully better description of my view later
today, so we may check for similarities and differences.

Now for some comments on the draft.

Ideally, whitelists would be keyed based on the From: address in the
RFC822 headers of a mail.

I'm not sure if "ideally" means "if addresses were trustworthy". IMHO,
whitelisting should be based on a <forwarding host, envelope recipient>
pair. BTW, why do you mention rfc822 rather than rfc2822?

It is not possible to whitelist a forwarder, since the MAIL FROM:
varies with the original return path.

If it varied (e.g. according to SRS) then it would be possible to apply an
SPF check successfully. To what variations of the envelope sender do you
allude?

However, to make this safer, this document proposes the
registration of a special virtual SASL mechanism, which always
succeeds but serves to seal an agreement between client and server
that the AUTH= argument is to be interpreted this way.

It probably should not have to "always succeed". This is probably where
some site may want to impose additional restrictions, such as the
forwarder having registered, its helo name to match an SPF "pass" test or
whatever. A failure message should tell the specific requirements
(possibly referring to a suitable web page.)

"Fred", an forwarded pseudonym of Ralph, at <fred@example.org>

maybe 's/an forwarded/a forwarding/'. I would also mention Fred before
Ralph, so that the order matches the message path.

Since they agree that such a shutdown would be a bad thing, they both
deploy SWK-SPF. On the hop to Ralph, the forwarded mail is sent with
null return path and using the forwarding input mailbox
(<fred@example.net>) as the SWK.

This apparently implies that in order to shutdown the forwarding agreement
one of the host would have to un-deploy SWK-SPF. The target host cannot do
that, if it also has other recipients who use the same mechanism. This is
another case where the initial authentication should fail.

C:MAIL FROM: <> AUTH=fred@example.org
[...]
C:MAIL FROM: <list-owner-ralph=example.net@fourth.example>
AUTH=list@fourth.example

It is not clear how does the forwarder decide whether to send an empty
envelope return address as in the first example, rather than keep it as in
the list example.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=92045013-fd8763
Powered by Listbox: http://www.listbox.com

On Thu, 31 Jan 2008, Alessandro Vesely wrote:
> As I'm not well acquainted with TENBOX, I don't understand why an alias
> expansion should differ from a list expansion, as far as whitelisting is
> concerned. Although lists should be characterized properly, they share
> various operative similarities with aliases.

The most courteous thing a forwarder could afford to do in a
backscatter-intolerant and SPF-using world, is to apply SRS when forwarding
mail that came in with an SPF-pass, and to rewrite the return-path of all
other mail to <>, indicating that it is non-bouncable since it might be
forged.

Such a forwarder won't mind if an email it SRSed gets rejected
in-transaction -- since it's obviously prepared to deal with a
post-transaction bounce, and internally generating a bounce is no harder. But
<> mails still must always go through to avoid Forwarding Problem B.

At the same time, exemptions from Forwarding Problem K are needed both for
the SRS mailstream and for the <> mailstream.

So we have a new color of list, one that implies that messages with nonnull
return paths may be treated normally except that IP-based karma is not
counted, and that messages with null return paths must always go through and
are also karma exempt.

> It is not possible to whitelist a forwarder, since the MAIL FROM:
> varies with the original return path.
>
> If it varied (e.g. according to SRS) then it would be possible to apply an
> SPF check successfully. To what variations of the envelope sender do you
> allude?

In traditional forwarding, the forwarder-to-recipient return-path obviously
varies with the sender-to-forwarder return path, since they are identical.
While you can easily whitelist any particular sender behind the forwarder,
you cannot whitelist the forwarding relationship as a whole.

In SRS forwarding, different sender-to-forwarder return paths also cause
different forwarder-to-recipient return paths, since information to recover
the former must be embedded in the latter.

> It probably should not have to "always succeed". This is probably where
> some site may want to impose additional restrictions, [...]

It's the initial AUTH command that always succeeds. However, SWK-SPF does
allow for a form of authentication failure, because AUTH= arguments that
claim SWKs to which the peer IP has no rights (according to SPF) will not be
honoured. They probably should cause 5xx on MAIL FROM:, although I've left
that undefined.

> maybe 's/an forwarded/a forwarding/'. I would also mention Fred before
> Ralph, so that the order matches the message path.

There are different orders in different examples.

1. Sarah -> Fred -> Ralph
2. Sarah -> Ralph !BOUNCE! -> Sarah
3. (unspecified) -> List -> Ralph
4. Sarah -> Ralph
5. Sarah -> List -> Fred -> Ralph !BOUNCE! -> Fred -> List

> This apparently implies that in order to shutdown the forwarding agreement
> one of the host would have to un-deploy SWK-SPF. The target host cannot do

No undeployment of SWK-SPF is necessary. What I mean by shutdown is that
after Example.org tries to send a MAIL FROM: <> to Ralph, and it 5xxes, for
any reason, then Example.org starts 4xxing every attempt to send a message to
"Fred". This means no more deadletters will pile up until Example.org
confirms that Ralph's misconfiguration is fixed.

Ralph won't like that because his forwarding will be unusable. That's his
incentive to deploy SWK-SPF.

> C:MAIL FROM: <> AUTH=fred@example.org
> [...]
> C:MAIL FROM: <list-owner-ralph=example.net@fourth.example>
> AUTH=list@fourth.example
>
> It is not clear how does the forwarder decide whether to send an empty
> envelope return address as in the first example, rather than keep it as in
> the list example.

The list example doesn't involve forwarding. SWK-SPF helps with the
forwarding problems but has other uses.

But anyways, the forwarder uses <> to indicate that the message is an
unbouncable hot potato. They want to wash their hands of it, and if they
can't they'll just shut down the forwarding.

The mailing list does use a bounce address, since if there is a delivery
problem, they'd want to log it.

---- Michael Deutschmann <michael@talamasca.ocis.net>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=92048446-536c9b
Powered by Listbox: http://www.listbox.com

On Thu, 31 Jan 2008, Michael Deutschmann wrote:

> The most courteous thing a forwarder could afford to do in a
> backscatter-intolerant and SPF-using world, is to apply SRS when forwarding
> mail that came in with an SPF-pass, and to rewrite the return-path of all
> other mail to <>, indicating that it is non-bouncable since it might be
> forged.

I hate to tell you this, but most MTAs are run by people who have never
heard of rfc2821, and merrily reply to a MAIL FROM of <>, "Damn
the mail loops and full speed ahead!"

That doesn't mean it isn't courteous to mark them anyway. But these
same MTAs will submit you to a blacklist if you send them too many DSNs,
(even for mail from their own IPs!)

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=92168750-a16ee1
Powered by Listbox: http://www.listbox.com

On Thu, 31 Jan 2008, "Stuart D. Gathman" wrote:
> I hate to tell you this, but most MTAs are run by people who have never
> heard of rfc2821, and merrily reply to a MAIL FROM of <>, "Damn
> the mail loops and full speed ahead!"

So to which IP address do they send their "RCPT TO: <>"?

If you mean that they'll bounce to RFC822 addresses, that's not the
forwarder's problem. It may make the original sender angry, but that
anger will be directed at the recipient's IP addresses, not the forwarder.
And this will be the same regardless of whether the forwarder is
traditional, SRS, or "Courteous".

> That doesn't mean it isn't courteous to mark them anyway. But these
> same MTAs will submit you to a blacklist if you send them too many DSNs,
> (even for mail from their own IPs!)

There's not much a forwarder can do if the recipient is going to be a
dip. After all, if there was a way a sender could unilaterally force a
recipient to 2xx everything, not do SPF, and not count karma, spammers
would be using it.

In practice, I would make one modification. In the unbouncable case, my
SMTP client would switch to Sham-SRS if it cannot negotiate SWK-SPF
authentication.

So, for <sarah@example.com> sending to <fred@example.org> which forwards
to <ralph@example.net>, example.org would do one of the following:

Case 1: Sarah's message got SPF pass, example.net offers SWK-SPF:
MAIL FROM: <sarah%example.com+HASH@example.org> AUTH=fred@example.org

Case 2: Sarah's message got SPF pass, example.net doesn't do SWK-SPF:
MAIL FROM: <sarah%example.com+HASH@example.org>

Case 3: Sarah's message got SPF neutral, example.net offers SWK-SPF:
MAIL FROM: <> AUTH=fred@example.org

Case 4: Sarah's message got SPF neutral, example.net doesn't do SWK-SPF:
MAIL FROM: <shamsrs-fred@example.org>
(where shamsrs-fred@example.org 5xxes everything. To circumvent dippy
admins who use callbacks, it might give 2xx at RCPT TO:, and save the 5xx
for DATA.)

---- Michael Deutschmann <michael@talamasca.ocis.net>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=92865077-5924b4
Powered by Listbox: http://www.listbox.com

Michael Deutschmann wrote:
> So, for <sarah@example.com> sending to <fred@example.org> which forwards
> to <ralph@example.net>, example.org would do one of the following:
>
> Case 1: Sarah's message got SPF pass, example.net offers SWK-SPF:
> MAIL FROM: <sarah%example.com+HASH@example.org> AUTH=fred@example.org

This is where the solution works, and the "forwarding-from" address is
made known to the recipient.

> Case 2: Sarah's message got SPF pass, example.net doesn't do SWK-SPF:
> MAIL FROM: <sarah%example.com+HASH@example.org>

That's ok, if the recipient won't cooperate there's not much more we can do.

> Case 3: Sarah's message got SPF neutral, example.net offers SWK-SPF:
> MAIL FROM: <> AUTH=fred@example.org
>
> Case 4: Sarah's message got SPF neutral, example.net doesn't do SWK-SPF:
> MAIL FROM: <shamsrs-fred@example.org>
> (where shamsrs-fred@example.org 5xxes everything. To circumvent dippy
> admins who use callbacks, it might give 2xx at RCPT TO:, and save the 5xx
> for DATA.)

In the latter two cases, a bounce may be needed to learn that a recipient
mailbox has been deleted. It is pretty useless to keep an obsolete recipe
which forwards to an inexistent address. For case 4, shamsrs-fred [at]
example.org should reach the postmaster there, and he should do a manual
check, deciding if the recipe should be deleted and if it is worth
forwarding back the DSN to the original sender. (The Return-Path for
manually configured recipes should reach the mailbox of the person who
manually configured them.)

In general, I think an acceptable degree of RFC compliance -not just
formal compatibility- is required for a protocol to became widely adopted.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=92987559-ab9edd
Powered by Listbox: http://www.listbox.com

On Sat, 2 Feb 2008, Alessandro Vesely wrote:
> Michael Deutschmann wrote:
> > Case 2: Sarah's message got SPF pass, example.net doesn't do SWK-SPF:
> > MAIL FROM: <sarah%example.com+HASH@example.org>
>
> That's ok, if the recipient won't cooperate there's not much more we can do.

And it won't even matter most of the time. The forwarder here is
confident that a bounce (whether internally generated after hitting a 5xx,
or relayed via SRS) will never be backscatter, so Forwarding Problem B is
not an issue. The MAIL FROM is SPF-correct, so no Problem S, either.

That does leave Problem K, but fortunately Karma-counting domains aren't
very common.

> > Case 3: Sarah's message got SPF neutral, example.net offers SWK-SPF:
> > MAIL FROM: <> AUTH=fred@example.org
> >
> > Case 4: Sarah's message got SPF neutral, example.net doesn't do SWK-SPF:
> > MAIL FROM: <shamsrs-fred@example.org>
>
> In the latter two cases, a bounce may be needed to learn that a recipient
> mailbox has been deleted. It is pretty useless to keep an obsolete recipe

A mail domain that does not in-transaction reject when the mailbox
doesn't exist deserves to lose. (They can wait until DATA if they want to
defeat stealth VRFYs.)

If there is an in-transaction rejection in Case 3/4, the forwarder will
have to shut down the input end of the forward anyway, to protect itself
from an accumulation of deadletters. Procedures for this will already
need to be in place, to deal with users who fail to whitelist. These
will have the effect of terminating the forward if the recipient cannot
be contacted.

---- Michael Deutschmann <michael@talamasca.ocis.net>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=93705016-b7e882
Powered by Listbox: http://www.listbox.com

Michael Deutschmann wrote:
>>> Case 3: Sarah's message got SPF neutral, example.net offers SWK-SPF:
>>> MAIL FROM: <> AUTH=fred@example.org
>>>
>>> Case 4: Sarah's message got SPF neutral, example.net doesn't do SWK-SPF:
>>> MAIL FROM: <shamsrs-fred@example.org>
>>
>> In the latter two cases, a bounce may be needed to learn that a recipient
>> mailbox has been deleted. It is pretty useless to keep an obsolete recipe
>
> A mail domain that does not in-transaction reject when the mailbox
> doesn't exist deserves to lose. (They can wait until DATA if they want to
> defeat stealth VRFYs.)

They may accept an RCPT if they have a forwarding recipe for it.

> If there is an in-transaction rejection in Case 3/4, the forwarder will
> have to shut down the input end of the forward anyway, to protect itself
> from an accumulation of deadletters.

Does that imply no mail queue? In that case, how can it handle the case of
multiple recipients requiring multiple connections, one of which fails
after DATA? (The raison d'être of LMTP.)

> Procedures for this will already
> need to be in place, to deal with users who fail to whitelist. These
> will have the effect of terminating the forward if the recipient cannot
> be contacted.

Except "no such user", 5xx replies may imply the recipient will still be
valid at some future date. It is always difficult to recognize those cases
properly.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=94124648-f18009
Powered by Listbox: http://www.listbox.com

On Wed, 6 Feb 2008, Alessandro Vesely wrote:
> Michael Deutschmann wrote:
> > If there is an in-transaction rejection in Case 3/4, the forwarder will
> > have to shut down the input end of the forward anyway, to protect itself
> > from an accumulation of deadletters.
>
> Does that imply no mail queue? In that case, how can it handle the case of
> multiple recipients requiring multiple connections, one of which fails
> after DATA? (The raison d'être of LMTP.)

No, there's still a queue. Which means that at least one deadletter
cannot be avoided.

The only way to avoid backscatter of non-SPF-pass messages is for them to
always be delivered forward. Within a single administrative domain, this can
be arranged, with inner mailservers told to suspend all judgement of mail
relayed in from a border MX.

It gets problematic when more than one administration is involved, such as
an externally administered backup MX, or of course forwarding. Here, the
border MTAs can fear being "betrayed" by the inner MTAs, via an
in-transaction 5xx.

To make betrayal a poor option for the inner MTA, I suggest that border MTAs
should apply the following brutal tactics:

1. Add a state to each message in the queue, which can be one of:
B - bouncable
U - unbouncable
H - unbouncable, held
R - unbouncable, re-queued

2. Every message begins in U state, unless it has a non-null MAIL FROM that
recieves an SPF Pass, in which case it begins in B state.

3. If a B state message fails to deliver (in-transaction 5xx or
in-transaction 4xx for "too long"), it is bounced normally.

4. If a U state or R state message fails to deliver, it is not converted to
a bounce, but instead transformed to H state and left frozen on the queue.
No automatic attempts are made to deliver H messages.

5. A border-MTA administrator can manually issue a command that all H-state
mail targetted at a specific address or domain be transformed to R-state,
which will re-start delivery attempts. This is an explicit, manual exception
to the general rule that 5xxes are final.

6. So long as a single R-state or H-state message is on the queue for an
endpoint, no new e-mails that would be U-state will be accepted for the
target. Instead, they will recieve in-transaction 4xx or 5xx.

7. The border-MTA administrater will be alerted when a message enters
H-state. Normally he will then contact the inner-MTA admin to let him know
he needs to fix his system. If the inner-MTA admin assures him the problem
is fixed, then the border-MTA gives the requeue order, and once the
resurrected deadletters exit the queue, normal relaying resumes.

---- Michael Deutschmann <michael@talamasca.ocis.net>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=94169575-7bbe09
Powered by Listbox: http://www.listbox.com

At 06:09 AM 2/6/2008 -0800, Michael Deutschmann wrote:

>The only way to avoid backscatter of non-SPF-pass messages is for them to
>always be delivered forward. Within a single administrative domain, this can
>be arranged, with inner mailservers told to suspend all judgement of mail
>relayed in from a border MX.
>
>It gets problematic when more than one administration is involved, such as
>an externally administered backup MX, or of course forwarding. Here, the
>border MTAs can fear being "betrayed" by the inner MTAs, via an
>in-transaction 5xx.
>
>To make betrayal a poor option for the inner MTA, I suggest that border MTAs
>should apply the following brutal tactics:
>
>1. Add a state to each message in the queue, which can be one of:
> B - bouncable
> U - unbouncable
> H - unbouncable, held
> R - unbouncable, re-queued
>
>2. Every message begins in U state, unless it has a non-null MAIL FROM that
>recieves an SPF Pass, in which case it begins in B state.
>
>3. If a B state message fails to deliver (in-transaction 5xx or
>in-transaction 4xx for "too long"), it is bounced normally.
>
>4. If a U state or R state message fails to deliver, it is not converted to
>a bounce, but instead transformed to H state and left frozen on the queue.
>No automatic attempts are made to deliver H messages.
>
>5. A border-MTA administrator can manually issue a command that all H-state
>mail targetted at a specific address or domain be transformed to R-state,
>which will re-start delivery attempts. This is an explicit, manual exception
>to the general rule that 5xxes are final.
>
>6. So long as a single R-state or H-state message is on the queue for an
>endpoint, no new e-mails that would be U-state will be accepted for the
>target. Instead, they will recieve in-transaction 4xx or 5xx.
>
>7. The border-MTA administrater will be alerted when a message enters
>H-state. Normally he will then contact the inner-MTA admin to let him know
>he needs to fix his system. If the inner-MTA admin assures him the problem
>is fixed, then the border-MTA gives the requeue order, and once the
>resurrected deadletters exit the queue, normal relaying resumes.

Excellent! This is good text for a BCP, if we decide to write one.

We might be able to reduce the burden of this procedure on Forwarders, if we "redistribute" some of the responsibilities to other Agents, in particular the Recipient. If the Recipient is required, when he sets up a Forwarder, to provide a reliable alternate path for communications (e.g. another email address, a postal address, or a phone number), then the Forwarder can contact him to let him know there was a problem at his MDA.

If the alternate path fails, or the Recipient simply abandons his responsibility, then discarding the dead message after 30 days is appropriate. It's no different than what his MDA would do. It's also a procedure that can be fully automated by the Forwarder.

Again, the situation I'm thinking of is:

|-------- Recipient's Network ---------|
/
--> / --> Receiver/Forwarder ~~> MDA ==> Recipient
/
Border

-- Dave


-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=94625558-8b4c32
Powered by Listbox: http://www.listbox.com

David MacQuigg wrote:
> At 06:09 AM 2/6/2008 -0800, Michael Deutschmann wrote:
>
>> The only way to avoid backscatter [...]
>
> Excellent! This is good text for a BCP, if we decide to write one.

Agreed. Except for SPF-specific parts, it strongly resembles the way MTAs
currently suppress backscatter. See e.g. the paragraph for *courier show
all* in http://www.courier-mta.org/courier.html

> We might be able to reduce the burden of this procedure on Forwarders,
> if we "redistribute" some of the responsibilities to other Agents, in
> particular the Recipient. If the Recipient is required, when he sets
> up a Forwarder, to provide a reliable alternate path for communications
> (e.g. another email address, a postal address, or a phone number), then
> the Forwarder can contact him to let him know there was a problem at
> his MDA.

I agree. However, a generalized statement should mention a "person in
charge", e.g. a user who can login at the Forwarder and write the relevant
forwarding recipe. That is possibly the same person as the Recipient, but
not necessarily. The person in charge is authorized by the admin, referred
to as the "data controller" in privacy parlance. The latter is yet another
alternative contact.

> If the alternate path fails, or the Recipient simply abandons his
> responsibility, then discarding the dead message after 30 days is
> appropriate.

However, in case the recipient mailbox has been deleted, after the message
has been discarded new deadletters will be accepted again. Compare that
to mailing list managers featuring automatic removal of dead addresses.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Archives: http://v2.listbox.com/member/archive/735/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/735/
Modify Your Subscription: http://v2.listbox.com/member/?member_id=1311532&id_secret=94780945-262228
Powered by Listbox: http://www.listbox.com