Mailing List Archive

On Tue, 2004-06-08 at 08:36 +0930, Stephen Carr wrote:
> I was hoping you could point me to a document that has the procedures
> to undertake a full test of SPF and SRS before I make the system live.
> End users are very touchy about loosing emails.

To test SRS, do this:

First, list every external address which any one of your users uses to
forward to their University email account, or which they may use like
that in the future.

For every such address, you need to check that SRS is already working.

Those servers are outside your control but you can test them -- send a
mail to the address which forwards to the University, and then watch
your own logs to see the sender address of the mail when it comes to the
final destination.

If the sender address is rewritten and looks like an SRS address, and
passes SPF, then SRS is working.

On the other hand, if _any_ of the mails come back _without_ the sender
address rewritten, then the world has not finished upgrading to EmailV2
and SRS yet, so you should wait longer before implementing a strict SPF
checking policy.

To assist you in this, I've temporarily set up the address sgcarr at
infradead.org which forwards to the address from which you posted your
mail. You can send a mail to that address, then see whether it comes
back rewritten or not. Let me know when you're done testing and I'll
remove the alias.

--
dwmw2

On Mon, 2004-06-07 at 16:06, Stephen Carr wrote:
> Dear List
>
> I am about to implement SPF - so far short periods of testing when the
> mail traffic is low seem to be OK .
>
> I was hoping you could point me to a document that has the procedures
> to undertake a full test of SPF and SRS before I make the system live.
> End users are very touchy about loosing emails.
>
> Thanks in advance
> Stephen Carr

As far as implementing SPF, step one is to publish records making sure
to end your SPF records with ?all. If you are very paranoid, you can
end them with +all. This ensures that even if the mail is invalid
(fails to pass the network space or conditions that you described with
your query) it will still make it through. This is a good way to test
when you have SPF applying headers to each email.

Once you have confidence that its doing that, and you are seeing PASS
where you should, and FAIL where you should and so on and so forth, you
might wish to change the +all to ?all. It is not recommended to change
to -all yet as we need to reach critical mass first or else you will
likely loose mails for those who have not yet heard about or implemented
SPF.

For SRS, you could have a look at libsrs wich is an ANSI C
implementation of SRS which you can compile and then inside of the tools
directory (providing all compiles well) you will find a tool called srs
(short for SRSQuery). You can feed this binary arbitrary values to help
understand what happens to an email address as it works its way through
the SRS process.

If you wish to code with libsrs, you can use the source code of SRSQuery
to see how the API works.

code3 tools $ ./srs -z
SRS Query v0.3 - James Couzens <jcouzens@obscurity.org>

From: (james@hosta.org)
to: (james@hostb.org)
which forwards to: (james@hostc.org)

HOSTA Send: envelope-from: (james@hosta.org)
HOSTB Rewrite: envelope-from: (SRS0=eM7Z=JE=hosta.org=james@hostb.org)
HOSTC Rewrite: envelope-from:
(SRS1=hostb.org=eM7Z=JE=hosta.org=james@hostc.org)
HOSTD Rewrite: envelope-from:
(SRS1=hostb.org=eM7Z=JE=hosta.org=james@hostd.org)
HOSTB Bounce: envelope-from: (SRS0=eM7Z=JE=hosta.org=james@hostb.org)
HOSTB HMAC VERIFIED : enevelope-from: (james@hosta.org)

Rewrite executed in 0.193 seconds

I would also recommend reading all of the available information linked
from http://spf.pobox.com under the heading "Sender Rewriting Sceheme".

In addition to this you will find invaluable the information available
elsewhere on the spf site. Any questions you have that don't involve
placing XML in DNS records by all means, ask away!

Cheers,

James

--
James Couzens,
Programmer
-----------------------------------------------------------------
XML is WRONG, and here it doesn't BELONG.
Neither in SPF, nor inside of DNS,
its fat and its bloated and so I express:
JSON - "The FAT FREE alternative to XML"
http://www.crockford.com/JSON/xml.html
-----------------------------------------------------------------
http://libspf.org -- ANSI C Sender Policy Framework library
http://libsrs.org -- ANSI C Sender Rewriting Scheme library
-----------------------------------------------------------------
PGP: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBD3BF855

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=srs-discuss@v2.listbox.com

James Couzens wrote:

> It is not recommended to change to -all yet as we
> need to reach critical mass first or else you will
> likely loose mails for those who have not yet
> heard about or implemented SPF.

Well, of 2843 SPF records of different domains that my mail server has
examined in the last month 2550 have '-all'. That's 90%.
94 have '~all', 136 have '?all', and 1 has 'all' (simplet.irislink.com).

Roger

On Sun, 2004-06-13 at 07:17, Roger Moser wrote:
> James Couzens wrote:
>
> > It is not recommended to change to -all yet as we
> > need to reach critical mass first or else you will
> > likely loose mails for those who have not yet
> > heard about or implemented SPF.
>
> Well, of 2843 SPF records of different domains that my mail server has
> examined in the last month 2550 have '-all'. That's 90%.
> 94 have '~all', 136 have '?all', and 1 has 'all' (simplet.irislink.com).

This is there choice, but as we saw last week, Verizon didn't have a
great time of it. There just aren't enough publishing domains for the
big mail servers to be doing this sort of thing imo.

Cheers,

James

--
James Couzens,
Programmer
-----------------------------------------------------------------
XML is WRONG, and here it doesn't BELONG.
Neither in SPF, nor inside of DNS,
its fat and its bloated and so I express:
JSON - "The FAT FREE alternative to XML"
http://www.crockford.com/JSON/xml.html
-----------------------------------------------------------------
http://libspf.org -- ANSI C Sender Policy Framework library
http://libsrs.org -- ANSI C Sender Rewriting Scheme library
-----------------------------------------------------------------
PGP: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBD3BF855

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=srs-discuss@v2.listbox.com

On Sun, 2004-06-13 at 07:22, James Couzens wrote:

> This is there choice, but as we saw last week, Verizon didn't have a
> great time of it. There just aren't enough publishing domains for the
> big mail servers to be doing this sort of thing imo.

s/there/THEIR :)

Cheers,

James

--
James Couzens,
Programmer
-----------------------------------------------------------------
XML is WRONG, and here it doesn't BELONG.
Neither in SPF, nor inside of DNS,
its fat and its bloated and so I express:
JSON - "The FAT FREE alternative to XML"
http://www.crockford.com/JSON/xml.html
-----------------------------------------------------------------
http://libspf.org -- ANSI C Sender Policy Framework library
http://libsrs.org -- ANSI C Sender Rewriting Scheme library
-----------------------------------------------------------------
PGP: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBD3BF855

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=srs-discuss@v2.listbox.com

On Sun, 2004-06-13 at 16:17 +0200, Roger Moser wrote:
> Well, of 2843 SPF records of different domains that my mail server has
> examined in the last month 2550 have '-all'. That's 90%.
> 94 have '~all', 136 have '?all', and 1 has 'all' (simplet.irislink.com).

Roger, how did you accumulate this information? I would be very
interested in your methodologies. And as a side comment, i personally
use the -all in my published SPF records, though i realize its NOT
recommended. Are most list readers here doing the same?

Michael Weiner
Systems Administrator
The UserFriendly Network (UFN)

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=srs-discuss@v2.listbox.com

Michael Weiner wrote:

> Roger, how did you accumulate this information?
> I would be very interested in your methodologies.

I dumped the SPF cache of my SPF implementation to a text file and counted
the 'v=spf1' and '-all'.

(The cache keeps all entries until I manually erase them. Of course the
program ignores the expired ones.)

Roger

Michael Weiner wrote:

>Roger, how did you accumulate this information? I would be very
>interested in your methodologies. And as a side comment, i personally
>use the -all in my published SPF records, though i realize its NOT
>recommended. Are most list readers here doing the same?
>
>
>
I use the -all for 5 domains, including a major Australian company, with
no ill effects. Surely if the SPF is not being checked, then there is no
risk. The only risk I see is for servers that don't check the SPF correctly.

Am I correct, or is there another issue I'm missing here?

Cheers
Bill

On Sun, 2004-06-13 at 07:53, Bill Maidment wrote:

> I use the -all for 5 domains, including a major Australian company, with
> no ill effects. Surely if the SPF is not being checked, then there is no
> risk. The only risk I see is for servers that don't check the SPF correctly.
>
> Am I correct, or is there another issue I'm missing here?

Unless you are doing NO forwarding you would be ok, since all e-mail
leaving your domains would all be valid when SPF checked, but if any of
your users are forwarding their addresses to other accounts, and your
record is evaluated by another MTA parsing, those forwards will be
denied.

Cheers,

James

--
James Couzens,
Programmer
-----------------------------------------------------------------
XML is WRONG, and here it doesn't BELONG.
Neither in SPF, nor inside of DNS,
its fat and its bloated and so I express:
JSON - "The FAT FREE alternative to XML"
http://www.crockford.com/JSON/xml.html
-----------------------------------------------------------------
http://libspf.org -- ANSI C Sender Policy Framework library
http://libsrs.org -- ANSI C Sender Rewriting Scheme library
-----------------------------------------------------------------
PGP: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBD3BF855

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=srs-discuss@v2.listbox.com

On Mon, 2004-06-14 at 17:51 +1000, Bill Maidment wrote:
> James Couzens wrote:
>
> >Unless you are doing NO forwarding you would be ok, since all e-mail
> >leaving your domains would all be valid when SPF checked, but if any of
> >your users are forwarding their addresses to other accounts, and your
> >record is evaluated by another MTA parsing, those forwards will be
> >denied.

Wrong. It's not about whether _you_ do any forwarding. It's about
whether any of the millions of addresses to which your users may send
email are doing forwarding.

> I am using virtusertable redirects, but isn't SRS supposed to deal with
> this?

If SRS is implemented at every forwarding site to which you ever send
email from the domain for which you publish a '-all' record, then yes.

--
dwmw2

David Woodhouse wrote:

>If SRS is implemented at every forwarding site to which you ever send
>email from the domain for which you publish a '-all' record, then yes.
>
>
>
OK I get the picture. Changing to ~all for now.

Thanks
Bill

On Mon, Jun 14, 2004 at 05:51:47PM +1000, Bill Maidment wrote:
> James Couzens wrote:
> >Unless you are doing NO forwarding you would be ok, since all e-mail
> >leaving your domains would all be valid when SPF checked, but if any of
> >your users are forwarding their addresses to other accounts, and your
> >record is evaluated by another MTA parsing, those forwards will be
> >denied.
> >
> I am using virtusertable redirects, but isn't SRS supposed to deal with
> this?

I take it you are using sendmail then.. Actually, srs+virtusertable is somewhat tricky if you're using libsrs or libsrs2 (as opposed to calling perl programs, a somewhat slow way of implementing srs imho), i recently wrote a sendmail patch that does virtusertable srs. It will be available for use with libsrs2/libsrs soon I think (sorry if i'm offending someone by putting libsrs and libsrs2 in the same sentence).

Just out of curiosity: what srs implementation are you using? And have you checked that it actually works on virtusertable forwarding?

Koen

--
K.F.J. Martens, Sonologic, http://www.sonologic.nl/
Networking, embedded systems, unix expertise, artificial intelligence.
Public PGP key: http://www.metro.cx/pubkey-gmc.asc
Wondering about the funny attachment your mail program
can't read? Visit http://www.openpgp.org/

Koen Martens wrote:

>I take it you are using sendmail then.. Actually, srs+virtusertable is somewhat tricky if you're using libsrs or libsrs2 (as opposed to calling perl programs, a somewhat slow way of implementing srs imho), i recently wrote a sendmail patch that does virtusertable srs. It will be available for use with libsrs2/libsrs soon I think (sorry if i'm offending someone by putting libsrs and libsrs2 in the same sentence).
>
>Just out of curiosity: what srs implementation are you using? And have you checked that it actually works on virtusertable forwarding?
>
>
>
I'm using the Mail::SRS perl programs with
sendmail/mimedefang/spamassassin/clamav patched with perlsrs.m4 Is there
another patch I should be using? Virtusertable seems to be working at
least on my test addresses.

Bill

On Mon, Jun 14, 2004 at 07:00:10PM +1000, Bill Maidment wrote:
> Koen Martens wrote:
> >Just out of curiosity: what srs implementation are you using? And have you
> >checked that it actually works on virtusertable forwarding?
> >
> I'm using the Mail::SRS perl programs with
> sendmail/mimedefang/spamassassin/clamav patched with perlsrs.m4 Is there
> another patch I should be using? Virtusertable seems to be working at
> least on my test addresses.

Well, if it works for you don't change it. Personally, i wouldn't use perl programs to do this, since it means invoking an external program on each incoming / forwarded mail.

An alternative to external program invocation is to patch sendmail and link it with libsrs or libsrs2, which eliminates the external program invocations (srs becomes internal to sendmail). If speed is of high concern (eg. high volume site), that is the preferred way to go I believe.

Koen

--
K.F.J. Martens, Sonologic, http://www.sonologic.nl/
Networking, embedded systems, unix expertise, artificial intelligence.
Public PGP key: http://www.metro.cx/pubkey-gmc.asc
Wondering about the funny attachment your mail program
can't read? Visit http://www.openpgp.org/

Koen Martens wrote:

>Well, if it works for you don't change it. Personally, i wouldn't use perl programs to do this, since it means invoking an external program on each incoming / forwarded mail.
>
>An alternative to external program invocation is to patch sendmail and link it with libsrs or libsrs2, which eliminates the external program invocations (srs becomes internal to sendmail). If speed is of high concern (eg. high volume site), that is the preferred way to go I believe.
>
>
>
I'm only running Received-SPF and SRS on my test domain until I find the
best implementation for a low-medium volume site. I don't mind trying
different methods for evaluation purposes, so I'll give libsrs2 a go.

Thanks for all your help.
Bill

On Mon, 14 Jun 2004, Koen Martens wrote:

> Well, if it works for you don't change it. Personally, i wouldn't use perl
> programs to do this, since it means invoking an external program on each
> incoming / forwarded mail.

If your sendmail supports socket maps, then a Perl or Python version
will work quite efficiently.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

On Mon, 14 Jun 2004, David Woodhouse wrote:

> Wrong. It's not about whether _you_ do any forwarding. It's about
> whether any of the millions of addresses to which your users may send
> email are doing forwarding.

This is not right. If the recipients have never heard of SPF, then
there is no problem. If they are checking SPF, and have a clue,
then they will whitelist the forwarders (which they have selected)
that don't do SRS or RSR. If they are checking SPF, and don't have
a clue, then they need to be educated.

Blindly rejecting mail from a non-SRS forwarder is quite simply an
incorrect configuration on the part of the recipient. This is not
unique to SPF. Quite a few mail servers are configured stupidly
for all sorts of reasons.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Stuart D. Gathman wrote:

> If they are checking SPF, and don't have
> a clue, then they need to be educated.
>

Yes, but gently. Remember this is still new to a lot of people.

> Blindly rejecting mail from a non-SRS forwarder is quite simply an
> incorrect configuration on the part of the recipient. This is not
> unique to SPF. Quite a few mail servers are configured stupidly
> for all sorts of reasons.
>

Again, not too many admins know about SRS. We don't want to put them off.

Nevertheless, your point about education is key to the whole issue and
hopefully softfails will trigger off a learning process.

Cheers
Bill

On Mon, 14 Jun 2004, Bill Maidment wrote:

> Stuart D. Gathman wrote:
>
> > If they are checking SPF, and don't have
> > a clue, then they need to be educated.
> >
>
> Yes, but gently. Remember this is still new to a lot of people.

This is a case where a little knowledge is worse than no knowledge at all.
The SPF web page and implementations should have a big warning:

WARNING - If any mailboxes in your domain use forwarders, do NOT
reject mail based solely on SPF unless you thoroughly understand
SRS, Reverse Source Route, and the effect of SPF on forwarders.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

[David Woodhouse]
>Wrong. It's not about whether _you_ do any forwarding. It's about
>whether any of the millions of addresses to which your users may send
>email are doing forwarding.
>
>>I am using virtusertable redirects, but isn't SRS supposed
>>to deal with this?
>
>If SRS is implemented at every forwarding site to which you ever send
>email from the domain for which you publish a '-all' record, then yes.

The SPF Publishing wizard (http://spf.pobox.com/wizard.html) uses -all
in most situations, because most sysadmins using the wizard answer "yes"
to the question "Do the above lines describe all the hosts that send
mail from yourdomain.com?"

I used this wizard when I first published SPF records, and answered yes
to that question. I thought only of mail ORIGINATING from my domain, as
most people would.

If publishing '-all' is as dangerous as you say, shouldn't someone
change the SPF Publishing Wizard to highlight the forwarding problem and
recommend '~all' instead, at least for the time being?

As many others have noted, I have not seen any bounced mail as a result
of publishing '-all', but this is probably because most SPF-aware
recipients are using SPF checks only as part of a larger spam scoring
system, and are not rejecting based on SPF checks yet.

Regards,
Ryan

On Mon, 2004-06-14 at 16:30 -0500, Ryan Malayter wrote:
> [David Woodhouse]
> >Wrong. It's not about whether _you_ do any forwarding. It's about
> >whether any of the millions of addresses to which your users may send
> >email are doing forwarding.
> >
> >>I am using virtusertable redirects, but isn't SRS supposed
> >>to deal with this?
> >
> >If SRS is implemented at every forwarding site to which you ever send
> >email from the domain for which you publish a '-all' record, then yes.
>
> The SPF Publishing wizard (http://spf.pobox.com/wizard.html) uses -all
> in most situations, because most sysadmins using the wizard answer "yes"
> to the question "Do the above lines describe all the hosts that send
> mail from yourdomain.com?"
>
> I used this wizard when I first published SPF records, and answered yes
> to that question. I thought only of mail ORIGINATING from my domain, as
> most people would.

Yes. The wizard is dangerously misleading. It tricks naïve admins into
setting up a situation where they ask third parties to throw away valid
email.

> If publishing '-all' is as dangerous as you say, shouldn't someone
> change the SPF Publishing Wizard to highlight the forwarding problem and
> recommend '~all' instead, at least for the time being?

Yes. But admitting the problem right there in the wizard would detract
from the evangelism effort, don't you think? :)

It seems that the authors of the wizard in question would prefer to
encourage others to publish broken ('-all') records, and bank on the
fact that few people are actually dim enough to bounce mail just because
it gets an SPF 'fail', as you yourself observed.

> As many others have noted, I have not seen any bounced mail as a result
> of publishing '-all', but this is probably because most SPF-aware
> recipients are using SPF checks only as part of a larger spam scoring
> system, and are not rejecting based on SPF checks yet.

Indeed. To actually reject mail solely on the basis of an SPF 'fail'
result would be completely irresponsible.

If your behaviour carries any influence with anyone else, I also
consider it irresponsible even to publish SPF records.

--
dwmw2

[David Woodhouse]
>It seems that the authors of the wizard in question would prefer to
>encourage others to publish broken ('-all') records, and bank on the
>fact that few people are actually dim enough to bounce mail
>just because it gets an SPF 'fail', as you yourself observed.

The problem is not naïve administrators, it is poor documentation. The distinction between 'fail' and 'softfail' is not explained in detail anywhere on the main pages of the main SPF site. The only portion that describes what 'softfail' truly means is the draft RFC, which is dated February 2004, well after I used the wizard to publish my SPF records.

In fact, many of the spf.pobox.com pages encourage sysadmins to publish '-all' after they have their all their users using SASL. No mention of the forwarding issue as it relates to '-all' can be found:
(http://spf.pobox.com/execsumm.html)
(http://spf.pobox.com/forsysadmins.html)
(http://spf.pobox.com/adoption.html)

>If your behaviour carries any influence with anyone else, I also
>consider it irresponsible even to publish SPF records.

I don't understand... why exactly is it irresponsible?

Regards,
Ryan

On Mon, 14 Jun 2004, Ryan Malayter wrote:

> [David Woodhouse]
> >It seems that the authors of the wizard in question would prefer to
> >encourage others to publish broken ('-all') records, and bank on the
> >fact that few people are actually dim enough to bounce mail
> >just because it gets an SPF 'fail', as you yourself observed.

> The problem is not naïve administrators, it is poor documentation. The
> distinction between 'fail' and 'softfail' is not explained in detail anywhere
> on the main pages of the main SPF site. The only portion that describes what
> 'softfail' truly means is the draft RFC, which is dated February 2004, well
> after I used the wizard to publish my SPF records.
>
> In fact, many of the spf.pobox.com pages encourage sysadmins to publish
> '-all' after they have their all their users using SASL. No mention of the
> forwarding issue as it relates to '-all' can be found:

That is because the forwarding issue is the responsibility of the
mail recipient - they are the one that chooses any forwarders.
The naieve administrator is completely correct in assuming that if
no mail for that domain should ever originate except from the listed machines
(in particular if he promises never to use greeting card sites), then
he should use -all when *publishing* SPF records.

However, configuring an MTA to *check* SPF is another matter. If users in the
receiving domain use forwarders, then the administrator cannot
reject mail based solely on SPF without making some provision for
the forwarders. These provisions might include:

o whitelisting trusted forwarders
o not checking SPF for forwarders which don't implement SRS, RSR, or SUBMITTER
o if the originating address is available (via SRS, RSR, or SUBMITTER),
and is SRS or SES signed, do CBV or the yet-to-be-defined DNS equivalent to
verify the originating address.

Note that just because a forwarder has an SPF record, doesn't mean
you should trust them. In todays email world, you should only
accept forwards from MTAs which you (or your users) have actually designated
as forwarders. Otherwise, regardless of SPF, you are essentially providing
an open relay for spam via DSNs, auto-replies, etc.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

On Mon, Jun 14, 2004 at 09:25:16AM -0400, Stuart D. Gathman wrote:
> On Mon, 14 Jun 2004, Koen Martens wrote:
>
> > Well, if it works for you don't change it. Personally, i wouldn't use perl
> > programs to do this, since it means invoking an external program on each
> > incoming / forwarded mail.
>
> If your sendmail supports socket maps, then a Perl or Python version
> will work quite efficiently.

That's right, is there such an implementation of srs already??

Koen

--
K.F.J. Martens, Sonologic, http://www.sonologic.nl/
Networking, embedded systems, unix expertise, artificial intelligence.
Public PGP key: http://www.metro.cx/pubkey-gmc.asc
Wondering about the funny attachment your mail program
can't read? Visit http://www.openpgp.org/