Mailing List Archive

On Sat, Feb 07, 2004 at 04:27:01PM +0100, Christophe Saout wrote:
|
| I was thinking about when a generic solution should do rewriting.
|

Last month Victor Duchovni suggested this algorithm. Now is a good time
for us all to revisit it.

http://archives.listbox.com/spf-discuss@v2.listbox.com/200401/0192.html

Also, Shevek's new Mail::SRS addresses many algorithm problems, I
suggest everybody review it to see how it handles the escaping problem
by clever use of prepending instead of appending.


-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

Meng Weng Wong <mengwong@dumbo.pobox.com> [2004-02-07/10:33]:
> Also, Shevek's new Mail::SRS addresses many algorithm problems, I
> suggest everybody review it to see how it handles the escaping problem
> by clever use of prepending instead of appending.

Ok, what I didn't immediately see was that the local part of the
forwarding alias has been dropped from the rewritten address, as Shevek
did not explicitly list it in his list of "removed" features, and the
interface still includes a parameter $alias (why's that?). As there is
no local alias in the rewritten envelope sender, the escaping delimiters
is not necessary anymore, and this also obsoletes my issue about locally
generated messages with foreign envelope senders -- the information
about the local alias which triggered the forwarding is not encoded
anyway, so the local case does not need any special handling.

Anyhow, the following assumption from the website seems still wrong, as
pointed out before: "no senduser can contain an @ sign". The senduser
may contain @ signs. For examples of valid addresses containing @-signs
in the local part, see my earlier message. However, this does not
introduce ambiguity into the scheme, but great care must be taken that
implementations get this right (ie. parse the domain from the right, not
from the left, and handle quoting properly).

When Mail::SRS 0.13 encounters an address like "xxx@xxx"@yyy, it will
die with "Sender username may not contain an @".

It should be able to handle an @ in a local part.

It should not just die on fatal errors, but instead return an error
(eg., return undef); the code calling Mail::SRS ought to be able to
handle errors gracefully.

Another issue is that it should be clearly defined how an MTA should
call Mail::SRS with addresses like "xxx\"xxx"@zzz or even those
containing weird stuff like control chars. Should it decode the quoted
form prior to calling Mail::SRS? Will the resulting address returned by
forward() be properly quoted? (it is not, currently)

Also, I think the database scheme should use some prefix too (SRSDB0+?);
if there is no such prefix, the addresses use the same namespace as
ordinary local mailboxes, which is something I would not want it to
do. It should be noted that there is actually no requirement for
using a MAC as key into the DB. A unique random string would suffice,
yield shorter addresses, and handling of key collisions is simpler.

Cheers,
Dan


--
Daniel Roethlisberger <daniel@roe.ch>
OpenPGP key id 0x804A06B1 (1024/4096 DSA/ElGamal)
144D 6A5E 0C88 E5D7 0775 FCFD 3974 0E98 804A 06B1
!->

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ï#ÄÏÉæGã!'Rzš´ˆ»£‡Æ~3com

On Sat, 2004-02-07 at 07:27, Christophe Saout wrote:
> Hi.

Christophe,

We need to stop thinking about any single MTA. While its great that you
have taken the time to run through this as respects Qmail, Qmail is only
ONE of many MTAs. We need a solution that is deployable into ALL MTAs.

Cheers,

James

--
James Couzens,
Programmer

Current projects:
http://libspf.org

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ï#ÄÏÉæGã!'Rzš´ˆ»£‡Æ~3com

So... on the topic or SRS or any other rewriting scheme.

I am writing this mail because I don't know much about it. I've read
the RFC proposals and all the information I could find on it, but I
still don't "get it".

My first beef: cutting out the middle man. It seems to me that cutting
out the middle-man introduces a weak point from the accountability
standpoint. The whole point of SPF is to increase accountability and
reduce spam. But.. there are remailers, thus the need for something
like SRS. If SRS allows you to cut out the middle-man, I think it does
something to damage the accountability added by SPF.

If I use a remailer, why would they not handle their own bounces. If I
send a mail from an MX (remailer or not), it should be responsible for
processing the bounce (not necessarily the same server, but the same
responsible entity on the business level -- e.g. same company). In
other words, anyone who sends a mail should be responsible for
accepting DSNs correlated with that mail -- that, to me, is one of the
many aspects of accountability.

Assuming we don't cut out the middleman, I don't see why we need a
"scheme" at all. Maybe a set of "best practices" guidelines. The
other end doesn't need to decode or inspect the envelope sender, they
just need to send bounces back to it. It doesn't matter at all what is
in it, as long as my inbound MXs can figure it out.

The database backend seems sketchy to me. The idea is that this mostly
applies to remailers right? The big ones send what 2 to 5 million
message per hour. It is very unlikely that they will want to maintain
a distributed database of keys across their globally distributed mail
servers just to implement something that doesn't really need a database
backing store. Distributing the key-values is a very difficult problem
-- and don't say "MySQL!" cause it won't solve your problem well. In
order to do this, you could back down to commutative replication
properties, but you still need synchronous, guaranteed commits before
using a token -- and that's damn expensive at 2 to 5 million an hour on
a large distributed system.

So, why build a solution that isn't scalable when it doesn't buy you
much? Email address can be very very long. A point was made earlier
about it being impossible to use reversible hashes as information
theory doesn't allow you to collapse N bytes to 48. Okay, that's
pretty obvious, but why collapse it at all? What is wrong with making
people's obnoxiously long email addresses stay obnoxiously long and
just encoding them for the "trip" to the next server.

What I am getting at is that I can accomplish a sender rewriting scheme
of sorts that makes me as a remailer responsible for both receiving the
bounces for re-mailed mails, and pushing received bounces back to the
correct accounts. What's more is that I can do it cleanly without
using the proposed scheme.

What would be wrong with me doing this:

I am acme-remailer.com
I have a secret MYSEC.
I have a message with an envelope sender OSENDER for which I need to
rewrite the RCPT TO and remail to example.com.
For the sake of simplicity I use base64 as my transform Te() to encode
and Td() to decode, but it could be anything for which the output of
Te() results in a RFC2822 compliant local part.
I have a secure message digest D(), perhaps SHA-1 -- but in the end,
whatever I want...

I remail with the envelope: "mysrs-" + time + "-" + T(OSENDER) + "-" +
T(D(MYSEC + time + OSENDER)) + "@acme-remailer.com"

I don't have to worry about wacky escaping, as no one will (or can)
take the middle-man out here. The bounce will come back to me and I
can both decode and validate it. time, of course, allowing me to
invalidate the return bounce path after my chosen expiry -- say 10
days.

A methodology like this give me the flexibility to implement thing as I
see fit (using encryption on top of everything if I like, choosing
different encodings, choosing better transforms) without disturbing
_anyone_ else. I can even roll keys (MYSEC can change every day --
just need to make sure I keep the secrets used over the last n days
where n is my expiry window on my return paths.

As I said in the beginning, I feel that I am missing something because
everyone here is smart. I am not looking to be lambasted, but rather
to be educated.

// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Postal Engine -- http://www.postalengine.com/
// Ecelerity: fastest MTA on earth

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

Theo Schlossnagle <jesus@omniti.com> [2004-02-07/22:45]:
> Assuming we don't cut out the middleman, I don't see why we need a
> "scheme" at all.

Valid points. I don't like cutting the middleman either, mainly because
an MTA should never attempt to interpret local parts for domains it is
not responsible for.

I think the best reason why somebody would want to cut the middleman is
to prevent overly long envelope sender addresses in a chain of multiple
forwarders. RFC 2821 does not seem to guarantee that envelope addresses
with local parts longer than 64 characters can be handled by every piece
of software implementing SMTP.

Whether there is need to cut the middleman depends on how gracefully
currently deployed MTA software can handle very long envelope addresses.

An alternative to cutting the middleman could be making sure that all
MTA can handle arbitrary long local parts in the envelope. Whether or
not that is feasible I am not sure.

Cheers,
Dan


--
Daniel Roethlisberger <daniel@roe.ch>
OpenPGP key id 0x804A06B1 (1024/4096 DSA/ElGamal)
144D 6A5E 0C88 E5D7 0775 FCFD 3974 0E98 804A 06B1
!->

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ï#ÄÏÉæGã!'Rzš´ˆ»£‡Æ~3com

On Sat, Feb 07, 2004 at 10:45:24PM -0500, Theo Schlossnagle wrote:

> The database backend seems sketchy to me. The idea is that this mostly
> applies to remailers right? The big ones send what 2 to 5 million
> message per hour. It is very unlikely that they will want to maintain
> a distributed database of keys across their globally distributed mail
> servers just to implement something that doesn't really need a database
> backing store. Distributing the key-values is a very difficult problem
> -- and don't say "MySQL!" cause it won't solve your problem well. In
> order to do this, you could back down to commutative replication
> properties, but you still need synchronous, guaranteed commits before
> using a token -- and that's damn expensive at 2 to 5 million an hour on
> a large distributed system.

I agree that doing this is difficult. The questions I have are :

a. is it strictly neccessary and
b. if it is, what are the minimum required stored data ?

>
> So, why build a solution that isn't scalable when it doesn't buy you
> much? Email address can be very very long. A point was made earlier
> about it being impossible to use reversible hashes as information
> theory doesn't allow you to collapse N bytes to 48. Okay, that's
> pretty obvious, but why collapse it at all? What is wrong with making
> people's obnoxiously long email addresses stay obnoxiously long and
> just encoding them for the "trip" to the next server.
>
> What I am getting at is that I can accomplish a sender rewriting scheme
> of sorts that makes me as a remailer responsible for both receiving the
> bounces for re-mailed mails, and pushing received bounces back to the
> correct accounts. What's more is that I can do it cleanly without
> using the proposed scheme.
>
> What would be wrong with me doing this:
>
> I am acme-remailer.com
> I have a secret MYSEC.
> I have a message with an envelope sender OSENDER for which I need to
> rewrite the RCPT TO and remail to example.com.
> For the sake of simplicity I use base64 as my transform Te() to encode
> and Td() to decode, but it could be anything for which the output of
> Te() results in a RFC2822 compliant local part.
> I have a secure message digest D(), perhaps SHA-1 -- but in the end,
> whatever I want...
>
> I remail with the envelope: "mysrs-" + time + "-" + T(OSENDER) + "-" +
> T(D(MYSEC + time + OSENDER)) + "@acme-remailer.com"
>
> I don't have to worry about wacky escaping, as no one will (or can)
> take the middle-man out here. The bounce will come back to me and I
> can both decode and validate it. time, of course, allowing me to
> invalidate the return bounce path after my chosen expiry -- say 10
> days.

I agree that it is possible to check that a time limited
returned token based on a temporary site secret, timestamp,
message ID and a secure
hash can indicate a bounce in connection with a message that
was forwarded without needing to save state in a complex
distributed database. As I understand it, however, without
the forwarder storing some information for each message sent,
at a minimum, _whether a bounce has already occurred_ , this
system could be vulnerable to replay attacks. What I understand
by such an attack is that more than one spam could be sent using
a token relating to a single forwarded message through the
bounce route connected with this token, within the
limited time window (e.g. 14 days) within which a bounce of
that particular message is valid.

Thinking about this further, the storage of state distributed
across a set of forwarders for a large domain to prevent replay attacks
of this kind does _not_ need information about every message forwarded.
It only really needs information about every message bounced,
for the rest of the duration of bounce validity if already bounced
messages were not known about, so that only the first bounce
is accepted and subsequent "bounces" supposedly in respect of the same
original unique message identity are not.

The distribution of this state information between a
dispersed set of forwarders also doesn't need to be transactionally
intact and complete in real time, so long as successful prevention of replay
spam masquerading as bounces is likely enough to make exploitation
of a possible replay window between non-real-time synchronised
forwarders unlikely to attract spammers towards this theoretical
method of distributing spam. I guess this database type defence
may further also only be needed in respect of as yet untrusted (e.g.
new) forwarding addresses, i.e. where there is not yet a high level
of confidence that spammers or their agents are unlikely to read the
forwarded messages, e.g. using automated network sniffers and
obtaining details of valid bounce tokens within their fortnight
expiry time.

If the percentage of messages bounced is low enough, these
considerations suggest to me that the database requirements,
assuming that one is in fact needed to prevent replays, are
much smaller and less stringent than I had previously thought. The
main reason for this is that messages which have not yet been
bounced don't need any information stored about them at all.

It also does not seem worth investing much time, money
and effort designing a solution to a theoretical attack that may
not in fact be practical. If such an attack vector is
ever used in practice, a better defence might be tighten up
on the standards required of an acceptable bounce message.
E.G. bounced messages might only be accepted from an SPF publishing domain
from a positively SPF indicated IP address for that domain,
if the bounce contains only information in a specified format, essentially
limited to telling the original sender that the original message
was not delivered and giving detail of the reason for non
delivery (e.g. an error number and a url within an ietf.org domain
explaining the error). If a bounce format and contents were
suitably standardised, how could such a message convey spam ?

Tightening up what is accepted as a bounce message combined
with use of secure tokens could be a much better overall approach.
As exploitation of bounce routes by spammers can be dealt
with by tightening up standards for bounce messages if and when
this occurs, this seems to avoid the need for a database
to prevent replays as a required aspect of initial SRS
implementation altogether.

Message bouncing should probably be regarded as an aspect of
the SMTP infrastructure which spammers have already badly
damaged. In a system where most bounces are misdirected and
seen by recipients as spam, mail administrators have stopped
sending these in marginal cases.
This suggests that standardisation of acceptable bounce messages
is probably needed , for this aspect of SMTP to be restorable.

Would it be worth including bounce standardisation proposals
as part of the new proposed SRS spec ?


> A methodology like this give me the flexibility to implement thing as I
> see fit (using encryption on top of everything if I like, choosing
> different encodings, choosing better transforms) without disturbing
> _anyone_ else. I can even roll keys (MYSEC can change every day --
> just need to make sure I keep the secrets used over the last n days
> where n is my expiry window on my return paths.
>
> As I said in the beginning, I feel that I am missing something because
> everyone here is smart. I am not looking to be lambasted, but rather
> to be educated.

I am also not yet convinced that I have the full picture on this,
as my view on it is still developing.

I agree with your analysis against requiring the local part of the
envelope sender address visibly to encode the full return path. I think
it better for a consensus to emerge as to where (e.g. in a new header
or, possibly using the existing Received: header chain)
to retain the delivery chain for bounce messages etc.,
except in situations where anonymous remailing is done for good
reasons.

Regards,
Richard Kay

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Feb 8, 2004, at 11:16 AM, Richard Kay wrote:

> I agree that it is possible to check that a time limited
> returned token based on a temporary site secret, timestamp,
> message ID and a secure
> hash can indicate a bounce in connection with a message that
> was forwarded without needing to save state in a complex
> distributed database. As I understand it, however, without
> the forwarder storing some information for each message sent,
> at a minimum, _whether a bounce has already occurred_ , this
> system could be vulnerable to replay attacks. What I understand
> by such an attack is that more than one spam could be sent using
> a token relating to a single forwarded message through the
> bounce route connected with this token, within the
> limited time window (e.g. 14 days) within which a bounce of
> that particular message is valid.

But a return path isn't used for "a" bounce. It is used for the many
many mails that can be returned along that path. DSNs also occur on
transient failure attempts as well. I think that DSNs are evil, but as
they are standard and not likely to go away, these bounce paths have to
be reusable.

> Thinking about this further, the storage of state distributed
> across a set of forwarders for a large domain to prevent replay attacks
> of this kind does _not_ need information about every message forwarded.
> It only really needs information about every message bounced,
> for the rest of the duration of bounce validity if already bounced
> messages were not known about, so that only the first bounce
> is accepted and subsequent "bounces" supposedly in respect of the same
> original unique message identity are not.

It does need to be on every message that is forwarded because at the
time of forwarding you don't know if a message will be produced along
the return path at a later time by a subsequent mail exchange.

> The distribution of this state information between a
> dispersed set of forwarders also doesn't need to be transactionally
> intact and complete in real time, so long as successful prevention of
> replay

It does need to be repllicated consistently. If I have 5 clusers of
MXs providing remailing services located around the globe, then I can
have:

SENDER -> MX-Japan -> Destination
Destination -> MX-California -> SENDER

If the second path relies on know "information" stored in a database
during the original mail's transit through MX-Japan, then it _MUST_
exist on MX-california before it is sent to the destination. Otherwise
a bad race occurs. What I was saying earlier is that enforcing
consistency would not require a 2-phase commit system -- it could be
done with commutative application. The fact that it would require
global commitment would mean that there would be a horrible latency
between MX-Japan creation of the "token" and the time when it can
actually use it safely. The latency introduced here would require most
MTAs (that can't support vast concurrency) to use store-and-forward
instead of in-session forwarding.

If you _just_ used the database to tally the number of message sent
across the return path, then the scale of the problem would be
substantially reduced, but I fear the implications on MTAs that don't
scale well. It could take anywhere from 1 second to 30 seconds to
perform an atomic counter increment in a globally distributed system.
That would mean that a bounce session (SMTP session receiving a bounce)
would have to put that session on hold while it checks-and-increments
the use counter. If you run a high traffic system that could easily
mean that your MX will need to support 100,000 or more concurrent SMTP
connections. While this isn't a problem for some MTAs, it is certainly
a problem for others.

> spam masquerading as bounces is likely enough to make exploitation
> of a possible replay window between non-real-time synchronised
> forwarders unlikely to attract spammers towards this theoretical
> method of distributing spam. I guess this database type defence
> may further also only be needed in respect of as yet untrusted (e.g.
> new) forwarding addresses, i.e. where there is not yet a high level
> of confidence that spammers or their agents are unlikely to read the
> forwarded messages, e.g. using automated network sniffers and
> obtaining details of valid bounce tokens within their fortnight
> expiry time.

So, if the message's original envelope sender isn't exposed, the
spammer would have no idea who they are sending to. While some may
argue that spammers don't care about that, that isn't true. Not _all_
spammers care, but many do. So, it reduces the potential abuses is
reduced by some (albeit known) non-zero percentage.

Also, you can be reasonably stringent on the From: and enveloper
senders you accept along the bounce path -- as it should only be used
by MTA software. This makes it even more unusable for spammers.

Basically, there are many ways to deal with spam now. One of which is
SPF (which is nice as it is about accountability and not content.
Yeah!!!). So, if SPF and all your other spam prevention mechanisms are
applied to messages send across the return path, the it makes the
address almost worthless to spammers: (1) they don't know where its
really going and (2) it isn't likely to get there if they use an open
proxy or "suspicious" methods.

> If the percentage of messages bounced is low enough, these
> considerations suggest to me that the database requirements,
> assuming that one is in fact needed to prevent replays, are
> much smaller and less stringent than I had previously thought. The
> main reason for this is that messages which have not yet been
> bounced don't need any information stored about them at all.

I am not seeing this. It seems to me that the whole point of storing
things in the database is to prevent long local parts. Which means
that EVERY forwarded message would need to a "short key" -> "long key"
mapping in the database before it could be forwarded.

> Message bouncing should probably be regarded as an aspect of
> the SMTP infrastructure which spammers have already badly
> damaged. In a system where most bounces are misdirected and
> seen by recipients as spam, mail administrators have stopped
> sending these in marginal cases.
> This suggests that standardisation of acceptable bounce messages
> is probably needed , for this aspect of SMTP to be restorable.

Agreed. I think that all bounces should happen _inline_ in SMTP. I
think the out-of-bound bounce messages should be done away with
entirely -- they were only useful on the Internet of 10 years ago.
But, that is very unlikely to happen for a variety of reasons.
Ideally, a remailer would hold the initial connection open while it
attempted to remail and hand back any delivery failures it would
receive to the client. As the chain can be a couple of steps long and
latency can be high, the concurrency requirements for MTAs would be
higher than what can be accomplished with X (name you favorite OS MTA).

> Would it be worth including bounce standardisation proposals
> as part of the new proposed SRS spec ?

Maybe... but I wouldn't know where to begin.

// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Postal Engine -- http://www.postalengine.com/
// Ecelerity: fastest MTA on Earth

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sat, 7 Feb 2004, Daniel Roethlisberger wrote:

> Meng Weng Wong <mengwong@dumbo.pobox.com> [2004-02-07/10:33]:
> > Also, Shevek's new Mail::SRS addresses many algorithm problems, I
> > suggest everybody review it to see how it handles the escaping problem
> > by clever use of prepending instead of appending.
>
> Ok, what I didn't immediately see was that the local part of the
> forwarding alias has been dropped from the rewritten address, as Shevek
> did not explicitly list it in his list of "removed" features, and the

Sorry! I thought that was removed from the pobox spec rather than the SRS
concept. That's a documentation error on my part.

> interface still includes a parameter $alias (why's that?). As there is

It uses the domain part of that address as the domain for the new outgoing
sender. This is to deal with systems which host multiple virtual mail
servers and want SRS addresses to appear from the appropriate virtual
server.

> no local alias in the rewritten envelope sender, the escaping delimiters
> is not necessary anymore, and this also obsoletes my issue about locally

> Anyhow, the following assumption from the website seems still wrong, as
> pointed out before: "no senduser can contain an @ sign". The senduser
[SNIP]
> When Mail::SRS 0.13 encounters an address like "xxx@xxx"@yyy, it will
> die with "Sender username may not contain an @".

This is a bug, and I will fix it. I should have documented it. I asked in
a previous post for examples of quoted addresses since I didn't want to
parse the BNF in a hurry at 1am. Has anyone got a digested version of that
BNF, or some decent human readable copy of that RFC? I suspect that just
introducing a dependency on some email address parsing module will solve
this.

There is also an outstanding issue regarding comments in email addresses,
which (I seem to remember) are allowed, nested to an arbitrary depth?

> It should not just die on fatal errors, but instead return an error
> (eg., return undef); the code calling Mail::SRS ought to be able to
> handle errors gracefully.

There is a general argument about the relative merits of using exceptions
to report errors against the merits of using discriminated return values.
Needless to say, there are major players on both sides, and I happened to
use exceptions in this code. I don't think the point is critical to SRS.

> Another issue is that it should be clearly defined how an MTA should
> call Mail::SRS with addresses like "xxx\"xxx"@zzz or even those
> containing weird stuff like control chars. Should it decode the quoted
> form prior to calling Mail::SRS? Will the resulting address returned by
> forward() be properly quoted? (it is not, currently)
>
> Also, I think the database scheme should use some prefix too (SRSDB0+?);
> if there is no such prefix, the addresses use the same namespace as
> ordinary local mailboxes, which is something I would not want it to

Computationally, there's no point. It _IS_ in the same namespace. There's
no time at which discriminating it would be at all useful. See my point
below about forcing people with databases to do things in a particular
way.

> do. It should be noted that there is actually no requirement for
> using a MAC as key into the DB. A unique random string would suffice,
> yield shorter addresses, and handling of key collisions is simpler.

This is true, but since that particular form of address isn't in the SRS
namespace, it doesn't matter. I see SRS as comprising two factors:

a) The convention that sender addresses are rewritten.
b) A common format for rewriting such addresses in the absence of store.

When someone has store, you can't force them to do anything. They can and
WILL use anything they like as a key. Since you don't have to parse this
key, there's nothing you can do about it. If they want to be compliant
with both SRS shortcutting and some database system, then they can use an
entire SRS local-part as a key.

The database implementation I provided was an example, no more. It was
just intended to lower the bar for people who wanted extras in their SRS
system.

S.

--
Shevek http://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sat, 7 Feb 2004, Theo Schlossnagle wrote:

> My first beef: cutting out the middle man. It seems to me that cutting
> out the middle-man introduces a weak point from the accountability
> standpoint. The whole point of SPF is to increase accountability and

This is true. It's a stack inspection system. The trick is either to
maintain the entire stack, or to make the system such that while it can be
cheated, it is impossible to obtain advantage from doing so. For reasons
detailed below, the second looks more promising.

> If I use a remailer, why would they not handle their own bounces. If I

No reason at all, except for the point-of-failure. This isn't the reason
the shortcutting is done.

> What would be wrong with me doing this:
>
> I am acme-remailer.com
> I have a secret MYSEC.
> I have a message with an envelope sender OSENDER for which I need to
> rewrite the RCPT TO and remail to example.com.
> For the sake of simplicity I use base64 as my transform Te() to encode
> and Td() to decode, but it could be anything for which the output of
> Te() results in a RFC2822 compliant local part.
> I have a secure message digest D(), perhaps SHA-1 -- but in the end,
> whatever I want...
>
> I remail with the envelope: "mysrs-" + time + "-" + T(OSENDER) + "-" +
> T(D(MYSEC + time + OSENDER)) + "@acme-remailer.com"

I think you have missed a couple of extremely points.

Firsrly, you have described almost exactly what Mail::SRS::Reversable
does, saving the fact that Mail::SRS is slightly more space efficient
since it doesn't need to use Base64.

Secondly, OSENDER can be arbitrarily long, and your scheme just made it
much longer. You're allowed maximum 64 bytes. So after 2 remailings, the
reversable system breaks this limit.

> I don't have to worry about wacky escaping, as no one will (or can)
> take the middle-man out here. The bounce will come back to me and I
> can both decode and validate it. time, of course, allowing me to
> invalidate the return bounce path after my chosen expiry -- say 10
> days.

This is already implemented.

> A methodology like this give me the flexibility to implement thing as I
> see fit (using encryption on top of everything if I like, choosing
> different encodings, choosing better transforms) without disturbing
> _anyone_ else. I can even roll keys (MYSEC can change every day --
> just need to make sure I keep the secrets used over the last n days
> where n is my expiry window on my return paths.

Even this is already implemented in Mail::SRS.

> As I said in the beginning, I feel that I am missing something because
> everyone here is smart. I am not looking to be lambasted, but rather
> to be educated.

I strongly suggest you download Mail::SRS and run "make teach". It
presents you with an interactive tutorial which expands upon many of these
points with real examples. If anything in that tutorial is unclear, please
tell me so that I can add further documentation and examples.

Thanks for the comments.

S.

--
Shevek http://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

It is my impression that the main problem with our discussion is that we
talk about a scheme and one particular implementation of that scheme at
the same time. They should ideally be designed and discussed seperately.

Or put in other words, SRS should not be defined by the implementation,
instead there should be a scheme definition, which is authorative over
the actual implementation(s).

Cheers,
Dan


--
Daniel Roethlisberger <daniel@roe.ch>
OpenPGP key id 0x804A06B1 (1024/4096 DSA/ElGamal)
144D 6A5E 0C88 E5D7 0775 FCFD 3974 0E98 804A 06B1
!->

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ï#ÄÏÉæGã!'Rzš´ˆ»£‡Æ~3com

On Feb 8, 2004, at 2:22 PM, Shevek wrote:

> On Sat, 7 Feb 2004, Theo Schlossnagle wrote:
>
>> My first beef: cutting out the middle man. It seems to me that cutting
>> out the middle-man introduces a weak point from the accountability
>> standpoint. The whole point of SPF is to increase accountability and
>
> This is true. It's a stack inspection system. The trick is either to
> maintain the entire stack, or to make the system such that while it
> can be
> cheated, it is impossible to obtain advantage from doing so. For
> reasons
> detailed below, the second looks more promising.
>> If I use a remailer, why would they not handle their own bounces. If
>> I
>
> No reason at all, except for the point-of-failure. This isn't the
> reason
> the shortcutting is done.

I don't agree. I think that remailers should be _made_ to accept the
bounces generated from the mail they send. Point-of-failure is
irrelevant. If the remailer failed, we would not have the problem of
it being unavailable for bounce processing. It is each domain's
responsibility to keep it's mail exchange architecture running
smoothly. If you believe that a remailer will have failure issues, I
would think carefully about using that remailer.

I think having the return path executed with no short-cuts increases
responsibility and accountability on all parts -- regardless if they
_could_ be side-stepped securely.

>> What would be wrong with me doing this:
>>
>> I am acme-remailer.com
>> I have a secret MYSEC.
>> I have a message with an envelope sender OSENDER for which I need to
>> rewrite the RCPT TO and remail to example.com.
>> For the sake of simplicity I use base64 as my transform Te() to encode
>> and Td() to decode, but it could be anything for which the output of
>> Te() results in a RFC2822 compliant local part.
>> I have a secure message digest D(), perhaps SHA-1 -- but in the end,
>> whatever I want...
>>
>> I remail with the envelope: "mysrs-" + time + "-" + T(OSENDER) + "-" +
>> T(D(MYSEC + time + OSENDER)) + "@acme-remailer.com"
>
> I think you have missed a couple of extremely points.
>
> Firsrly, you have described almost exactly what Mail::SRS::Reversable
> does, saving the fact that Mail::SRS is slightly more space efficient
> since it doesn't need to use Base64.
>
> Secondly, OSENDER can be arbitrarily long, and your scheme just made it
> much longer. You're allowed maximum 64 bytes. So after 2 remailings,
> the
> reversable system breaks this limit.

Base64 was just an example, my quick test implementation used
base64(gzip(lp)). But that is beside the point, the argument was about
methodology, not implementation details.

I don't see how one system is better than the other since both break
during remailing a not-uncommonly-long email address.

theo.schlossnagle@mail.lethargy.org remailed to
theo.schlossnagle@omniti.com would result in a >64 character localpart
using Mail::SRS.

They both break _during_ the first remailing. It seems painfully
limited either way and under that argument they are both unusable and
broken.

>> I don't have to worry about wacky escaping, as no one will (or can)
>> take the middle-man out here. The bounce will come back to me and I
>> can both decode and validate it. time, of course, allowing me to
>> invalidate the return bounce path after my chosen expiry -- say 10
>> days.
>
> This is already implemented.

I wasn't saying it wasn't implemented. In fact, I'd prefer to not talk
about implementations at all. I'm a professional implementor too, but
I am not going to start on this until I understand why we are trying to
solve the problem this way. I don't do things while concerned or
confused -- it's a policy :-)

>> A methodology like this give me the flexibility to implement thing as
>> I
>> see fit (using encryption on top of everything if I like, choosing
>> different encodings, choosing better transforms) without disturbing
>> _anyone_ else. I can even roll keys (MYSEC can change every day --
>> just need to make sure I keep the secrets used over the last n days
>> where n is my expiry window on my return paths.
>
> Even this is already implemented in Mail::SRS.

>> As I said in the beginning, I feel that I am missing something because
>> everyone here is smart. I am not looking to be lambasted, but rather
>> to be educated.
>
> I strongly suggest you download Mail::SRS and run "make teach". It
> presents you with an interactive tutorial which expands upon many of
> these
> points with real examples. If anything in that tutorial is unclear,
> please
> tell me so that I can add further documentation and examples.

I looked at this. I wasn't suggesting something "new". In fact, my
scheme above looks almost identical to Dan's:
http://roe.ch/spam/return-path-rewriting.xml. I was bringing two
points up that I still think are important:

(1) I think building a system for wide adoption with a DB backend is a
bad idea for this context. It is _very_ hard to scale specifically for
the people/sites that need most to implement it.

(2) I don't understand why the details of the encoding scheme should be
dictated in such detail. Why is it "this is how it is done" as opposed
to, "this is the purpose and this is how it _could_ be done." As the
encoding scheme itself is only important within a single organizational
unit.

What I'm getting from the conversation is that "reversible encoding
won't work, so let's build something that won't scale." To me, that
sort statement means that everyone should drop their pencils and we
should all go back to the chalkboard.

// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Postal Engine -- http://www.postalengine.com/
// Ecelerity: fastest MTA on Earth

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sun, 8 Feb 2004, Theo Schlossnagle wrote:

> I think having the return path executed with no short-cuts increases
> responsibility and accountability on all parts -- regardless if they
> _could_ be side-stepped securely.

[SNIP]

> (1) I think building a system for wide adoption with a DB backend is a
> bad idea for this context. It is _very_ hard to scale specifically for
> the people/sites that need most to implement it.

So if we don't use a database, and we also maintain the entire state of
the chain, how can we avoid the spiralling of large local-parts? If the
information isn't embedded into the local-part, and it isn't in a
database, where is it?

I'm afraid that at this point, I have no suggestions to offer to solve
these two problems simultaneously, so I have to leave this to the floor.

> (2) I don't understand why the details of the encoding scheme should be
> dictated in such detail. Why is it "this is how it is done" as opposed
> to, "this is the purpose and this is how it _could_ be done." As the
> encoding scheme itself is only important within a single organizational
> unit.

This is true for any system that doesn't use shortcutting. The reasons for
shortcutting are to do with local-part length. Therefore the problems of
shortcutting need to be solved before we can even decide that we are going
to fix on one concrete format.

S.

--
Shevek http://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Feb 8, 2004, at 3:47 PM, Shevek wrote:

> On Sun, 8 Feb 2004, Theo Schlossnagle wrote:
>> (1) I think building a system for wide adoption with a DB backend is a
>> bad idea for this context. It is _very_ hard to scale specifically
>> for
>> the people/sites that need most to implement it.
>
> So if we don't use a database, and we also maintain the entire state of
> the chain, how can we avoid the spiralling of large local-parts? If the
> information isn't embedded into the local-part, and it isn't in a
> database, where is it?
>
> I'm afraid that at this point, I have no suggestions to offer to solve
> these two problems simultaneously, so I have to leave this to the
> floor.

I think leaving it on the floor is dangerous. I, for one, want a
solution that valid and doesn't require a huge infrastructure
investment. SPF is low cost to implement and almost zero cost to
publish -- which make it trivial to adopt. RPR (and SRS) seem to have
issues that will make them very hard to widely adopt. This really
needs to be solved. We shouldn't walk away from a problem just because
it is hard -- we have too much to lose.

>> (2) I don't understand why the details of the encoding scheme should
>> be
>> dictated in such detail. Why is it "this is how it is done" as
>> opposed
>> to, "this is the purpose and this is how it _could_ be done." As the
>> encoding scheme itself is only important within a single
>> organizational
>> unit.
>
> This is true for any system that doesn't use shortcutting. The reasons
> for
> shortcutting are to do with local-part length. Therefore the problems
> of
> shortcutting need to be solved before we can even decide that we are
> going
> to fix on one concrete format.

Okay, I need more explanation. If you use an encoding scheme, then by
the time the message reaches its final destination the return path
envelope sender will have reaches it largest size. Short cutting only
effects the actual execution of that return path, right? I don't
understand how short-cutting helps reduce local-part length at all,
what am I missing?

Maybe I am misinterpreting you. Do you mean that the reasons for _not_
shortcutting are to do with local-part length?

I think we should all just throw away short-cutting and require the
email to actually traverse the explicit return path. I fail to see
what is gained by short-cutting.

// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Postal Engine -- http://www.postalengine.com/
// Ecelerity: fastest MTA on earth

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sun, 8 Feb 2004, Theo Schlossnagle wrote:
> On Feb 8, 2004, at 3:47 PM, Shevek wrote:
> > On Sun, 8 Feb 2004, Theo Schlossnagle wrote:
> >> (1) I think building a system for wide adoption with a DB backend is
> >> a bad idea for this context. It is _very_ hard to scale specifically
> >> for the people/sites that need most to implement it.
> >
> > So if we don't use a database, and we also maintain the entire state of
> > the chain, how can we avoid the spiralling of large local-parts? If the
> > information isn't embedded into the local-part, and it isn't in a
> > database, where is it?
> >
> > I'm afraid that at this point, I have no suggestions to offer to solve
> > these two problems simultaneously, so I have to leave this to the
> > floor.
>
> I think leaving it on the floor is dangerous. I, for one, want a
> solution that valid and doesn't require a huge infrastructure
> investment. SPF is low cost to implement and almost zero cost to

I'm sorry, I was meaning to say that I can see no other solution for SRS,
and I don't believe one exists. There seems to be opposition to both
databasing and shortcutting. However, there is a forced choice between:

a) Shortcutting some hops (Mail::SRS does this)
b) Having an unbounded return path (Mail::SRS::Reversable does this)
c) Using a database (Mail::SRS::DB does this)

There is (as far as I can see) no other solution.

> publish -- which make it trivial to adopt. RPR (and SRS) seem to have
> issues that will make them very hard to widely adopt. This really needs
> to be solved. We shouldn't walk away from a problem just because it is
> hard -- we have too much to lose.

I don't see how SRS is made particularly "hard" to implement, especially
in the shortcut system. In fact, the way the current modules have been
written allows them to coexist on the same internet without clashing. They
will all interoperate correctly. This means that an individual
implementing SRS can choose which module to run, and thus choose which
tradeoff they wish to make.

In practice, if you have any sort of published standard, there will be no
way to prevent people from abusing it as they see fit. There is no way to
stop people doing (c) if they want to, and if the data format for the
local part is public (as it will be), there is no way to stop people from
doing (a) or (b).

> Okay, I need more explanation. If you use an encoding scheme, then by
> the time the message reaches its final destination the return path
> envelope sender will have reaches it largest size. Short cutting only
> effects the actual execution of that return path, right? I don't
> understand how short-cutting helps reduce local-part length at all,
> what am I missing?
[SNIP]
> I think we should all just throw away short-cutting and require the
> email to actually traverse the explicit return path. I fail to see
> what is gained by short-cutting.

Considerations for the local part must be:

a) Keep it small / Keep the size bounded. These are separate and related
points.
- It will fit into static buffers (boundedness).
- Short cutting makes the overhead introduced by Mail::SRS bounded
above by 32 bytes. This is quite small.
b) Keep it human readable.
- A system which is transparent and understandable to humans is more
easily
* implemented
* debugged
* understood
c) Keep it machine readable.
- Escaping is to be avoided. People grepping for email addresses in
things like spamassassin don't want to have to do an arbitrary number
of unescapings before they get to something they can match on.

If we store the entire return path in the local part (as implemented in
Mail::SRS::Reversable), the local part will very rapidly grow beyond
reasonable size, and no fixed upper limit exists for this size. This is
not good.

The question I have to ask is "Do we really need to keep the full path?"

I think that this question is important enough to merit a separate thread,
which I will post.

S.

--
Shevek http://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sun, Feb 08, 2004 at 08:47:52PM +0000, Shevek wrote:
> So if we don't use a database, and we also maintain the entire state of
> the chain, how can we avoid the spiralling of large local-parts? If the
> information isn't embedded into the local-part, and it isn't in a
> database, where is it?
>
> I'm afraid that at this point, I have no suggestions to offer to solve
> these two problems simultaneously, so I have to leave this to the floor.

Is there really no other way of putting the information into
a forwarded message, such that a multi-hop bounce path is preserved
within the bounce when a forwarded message is bounced back ?

Perhaps not with all MTAs in current use ? But will this work
with MTAs which handle _enough_ current mail, such that if for example,
another header is introduced for this purpose, or this information is
encoded into the Received: chain, sufficient information about the chain
will come back with the bounce to extract this and send it back to
the last MTA on the message delivery chain in the case of most messages ?

Who suffers the loss of reputation by, from the point of
view of the originator, deleting mis-addressed mail silently ? Not
the operators of the system which returns an unusable bounce ?

Or is the difficulty the cycles used in grepping the required
data out of the bounced message ? I suspect this is more scalable
than it initially seems, if a mail server which handles 1million
forwards an hour has 2% of these bounced, this means that
20,000 bounce messages might have to be parsed per hour for the
relevant data, which doesn't seem such a massive task as to be
that expensive, compared to what such a server is doing already.

What we might lose is the ability to decide whether the bounce is
handleable at the envelope stage. Is this a price worth paying ?

Are we willing to consider breaking the ability to return bounce messages for
MTAs which don't return this information in a usable format if
this proportion of messages is acceptably low, and this price is
considered worth paying to tackle this dilemma, and the reputation
cost is mainly borne by system operators which don't upgrade MTAs
sufficiently to return usable bounce information within an
SPF upgrading SMTP world ?

Richard Kay

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sun, Feb 08, 2004 at 08:47:52PM +0000, Shevek wrote:
|
| So if we don't use a database, and we also maintain the entire state of
| the chain, how can we avoid the spiralling of large local-parts? If the
| information isn't embedded into the local-part, and it isn't in a
| database, where is it?
|

I should clarify the requirement that led to the shortcutting solution:
i understand email addresses may not exceed 256 chars, and localparts
should not exceed 64 chars. we're going to break the 64 limit some of
the time but i'd like to do it in a way that has O(1) nature rather than
O(N) nature.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sun, Feb 08, 2004 at 07:47:59PM -0500, Meng Weng Wong wrote:
|
| I should clarify the requirement that led to the shortcutting solution:
| i understand email addresses may not exceed 256 chars, and localparts
| should not exceed 64 chars. we're going to break the 64 limit some of
| the time but i'd like to do it in a way that has O(1) nature rather than
| O(N) nature.
|

I meant O(k), not O(1). I think.

Anyway, the important thing is that it would be bad to have

a-b-c-d-e-f-...@z

Shortcutting lets you say "hop N can remove hop N-1"

So if you start with a-b-c@d instead of doing a-b-c-d@e you do the
shortcut: a-b-d@e

I think it's ok for d to remove c, and for e to remove d, but I think we
want to always leave b in there.

a has to stay

b protects a

c cannot remove b because c doesn't know b well enough to vouch for it

d *can* remove c because b is still in the mix and wraps a

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h

On Sun, 8 Feb 2004, Meng Weng Wong wrote:

> On Sun, Feb 08, 2004 at 07:47:59PM -0500, Meng Weng Wong wrote:
> |
> | I should clarify the requirement that led to the shortcutting solution:
> | i understand email addresses may not exceed 256 chars, and localparts
> | should not exceed 64 chars. we're going to break the 64 limit some of
> | the time but i'd like to do it in a way that has O(1) nature rather than
> | O(N) nature.
> |
>
> I meant O(k), not O(1). I think.

Constant factors are ignored, therefore either is correct and usually one
says O(1).

> Anyway, the important thing is that it would be bad to have
>
> a-b-c-d-e-f-...@z
>
> Shortcutting lets you say "hop N can remove hop N-1"
>
> So if you start with a-b-c@d instead of doing a-b-c-d@e you do the
> shortcut: a-b-d@e
>
> I think it's ok for d to remove c, and for e to remove d, but I think we
> want to always leave b in there.
>
> a has to stay
>
> b protects a
>
> c cannot remove b because c doesn't know b well enough to vouch for it
>
> d *can* remove c because b is still in the mix and wraps a

This is correct, but we can do much better than just a second srs0 step:

If b is a spammer then he will spam a. Therefore b is not a spammer.
Therefore d can rely on b to do cryptographic and timestamp checks.
Therefore d does not need to do them. Therefore the first forwarding (b)
uses the existing srs0 format, the second format need only use:

srs0+<yaddayadda>@forwarder.com
becomes
srs1+forwarder.com+<yaddayadda>@d.com
which forwards a second (etc) time to
srs1+forwarder.com+<yaddayadda>@e.com
and always reverses in one step to
srs0+<yaddayadda>@forwarder.com

It doesn't matter at all if d can be caused to bounce an invalid mail to
b, because noone ever receives it.

This costs two bytes and one domain name in the local-part. We're closer
to the 64 byte limit, but we've increased by 1 domain name and 1 byte (We
are allowed to remove the "srs0").

This scheme satisfies all the requirements of SRS, and provides a
cryptographic guard around a, which prevents gaming. Furthermore, this
strategy can coexist on the "same internet" with SRS::Shortcut,
SRS::Reversable and SRS::DB! Furthermore the API is (still) identical to
that of the previous versions.

I have implemented this as an extension of SRS::Shortcut called
SRS::Guarded and I will be releasing later tonight UK time, so expect it
by about 3pm or 4pm Vancouver time. It isn't tested yet, and I won't
release code without tests.

S.

--
Shevek http://www.anarres.org/
I am the Borg. http://www.gothnicity.org/

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname@Ë`Ì{5¤¨wâÇSÓ°)h