Mailing List Archive

Tom Lahti wrote:

> In the page http://asarian-host.net/srs/sendmailsrs.htm an SRS
> integration with sendmail is presented. It is implied in the opening
> paragraphs that after implementation something like the following
> would happen:
>
> >>> MAIL from: <>
> <<< 250 2.1.0 <>... Sender ok
> >>> RCPT To:<admin@asarian-host.net>
> <<< 250 2.1.5 <admin@asarian-host.net>... Recipient ok
> >>> DATA
> <<< 550 5.7.1 Bounce address not SRS signed!
>
> However, this is NOT the case. There are no modifications to the
> sendmail config file presented on the web page, which would generate
> a 550 error after the data phase if the bounce recipient is not SRS
> signed.

My page also reads:

"You could certainly put a front-end Milter at the gate. Especially so, if
you wish to implement the above outlined "empty envelope-from + no SRS
signed recipient + DATA phase" anti-spammer policy. But that falls outside
the scope of this document, which is to integrate SRS functionality into
sendmail."

> How can this presentation be amended to achieve the goal of rejecting
> bounces that are NOT signed AT ALL? This would need to be invoked:
>
> 1) Only if the envelope is <>, and
> 2) Only if the recipient is unsigned (either valid or invalid), and
> 3) only after we enter the DATA phase.
>
> Since ParseLocal is called on the recipient before the possible DATA
> phase, we can't reject the mail there. I am at a loss as to how to
> do anything when we enter the DATA phase.

To actually reject right after the DATA command is tricky. In practice, you
would have to wait out the DATA phase, and reject at its completion.
Officially, per RFC 2821, sending a 550 response code after DATA is no
longer appropriate, and this should be 554 ("Transaction failed"). I should
change my documentation to wit.

A Milter is really a good place for implementing such a policy, as it gives
you the nice "overiew" of all conditions which need to be met before you
reject.

> It might be sufficient to add the envelope sender to the ARGV list
> for the program invoked by is_srs and rewrite the recipient to
> "/dev/null" ... If its a valid probe, it won't matter, and if
> there's DATA later then it'll get delivered to the bit bucket. The
> only thing is then we allow the DATA to be transmitted even if we're
> going to drop it anyway.

I will see about adding a script that actually does implement the outlined
anti-spammer policy. Though I doubt the DATA phase can actually be
circumvented from sendmail.cf.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

>My page also reads:
>
>"You could certainly put a front-end Milter at the gate. Especially so, if
>you wish to implement the above outlined "empty envelope-from + no SRS
>signed recipient + DATA phase" anti-spammer policy. But that falls outside
>the scope of this document, which is to integrate SRS functionality into
>sendmail."

My apologies. I did not read that part, I was only scanning for the
"meat". When I saw that example SMTP transaction bit at the top, I assumed
that was included in the scope.

>I will see about adding a script that actually does implement the outlined
>anti-spammer policy. Though I doubt the DATA phase can actually be
>circumvented from sendmail.cf.

I think it could be as easy as eliminating the is_srs macro, _always_ call
reverse_srs program, and include in its arg list both the recipient address
and the sender envelope. What I don't see is the purpose of

} elsif ($use_address =~ /^\|/) {
print "\"$old_address\"\n";

in that script. Also, if we send $f to srs2envtol.pl does it include the
brackets <> ? I could experiment but if someone already has a test
environment set up, it would save me the trouble for one little test.. :)


sendmail.cf:
---------------------------------------------------------
SParseLocal=98

# always call reverse_srs
R$* $@ $(reverse_srs $1 $f $)
---------------------------------------------------------


patch srs2envtol.pl:

--- srs2envtol.pl 2004-05-24 14:28:24.000000000 -0700
+++ srs2envto2.pl 2004-05-25 10:24:23.000000000 -0700
@@ -29,5 +29,6 @@

my $old_address = $ARGV[0];
+my $envelope_sender = $ARGV[1];
my $use_address;
my $srs = new Mail::SRS (Secret => $secret, HashLength => 8,
AlwaysRewrite => 1);
@@ -58,4 +59,7 @@
print "\"$old_address\"\n";
} else {
+ if (length($envelope_sender)==0) {
+ $old_address = "/dev/null";
+ }
print "$old_address\n";
}





--
-- =========================
Tom Lahti
Tx3 Online Services

(888)4-TX3-SVC (489-3782)
http://www.tx3.net/
-- =========================

Tom Lahti wrote:

> What I don't see is the purpose of
>
> } elsif ($use_address =~ /^\|/) {
> print "\"$old_address\"\n";
>
> in that script.

When you have defined a "piped" alias, like so:

test: "| /usr/local/bin/perl /scripts/whatever.pl"

The space-laced address needs to be requoted on return. That is what the
extra check + output does.

> Also, if we send $f to srs2envtol.pl does it include
> the brackets <> ?

No.

> sendmail.cf:
> ---------------------------------------------------------
> SParseLocal=98
>
> # always call reverse_srs
> R$* $@ $(reverse_srs $1 $f $)
> ---------------------------------------------------------

I am pretty sure you did not test your code; $f needs to be dereferenced
here, and should be written as $&f instead.

> patch srs2envtol.pl:
>
> my $old_address = $ARGV[0];
> +my $envelope_sender = $ARGV[1];

This will not work either. If you parse both variables to the reverse_srs
macro, you will need to split the one $ARGV[0], as follows:

my ($old_address, $envelope_from) = split (/ /, $ARGV[0], 2);

(this, btw, may look strange, as you would expect two args; but if you look
at the above piped alias example, sendmail's rationale becomes clear: what
we would generally regard as multiple, space-separated args, in reality is
likely just one arg: a recipient with spaces).

The principle of parsing $f is sound, of course. Indeed you could make the
determination from that whether we have an empty envelope-from + no SRS
signature. By then setting the recipient to "/dev/null" you could indeed
blackhole the email.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

>The space-laced address needs to be requoted on return. That is what the
>extra check + output does.

Got it, I thought it must be something like that.

> > sendmail.cf:
> > ---------------------------------------------------------
> > SParseLocal=98
> >
> > # always call reverse_srs
> > R$* $@ $(reverse_srs $1 $f $)
> > ---------------------------------------------------------
>
>I am pretty sure you did not test your code; $f needs to be dereferenced
>here, and should be written as $&f instead.

I'm pretty sure I didn't either :) $&f does work quite a bit
better. However, it seems that this gets called twice for each email; once
for the SMTP envelope sender with $f empty, and a second time with the
recipient and envelope sender in the correct place. Apparently, the
envelope sender is passed through ParseLocal with $f unset. I do not know
why, or how to get around this, since having $f empty is precisely what I'm
looking for, so I can't use that as an indicator that the the $old_address
is in fact the sender and not the recipient.

Too bad I'm not a sendmail expert or I'd have this done already and a web
page written up on it.


--
-- =========================
Tom Lahti
Tx3 Online Services

(888)4-TX3-SVC (489-3782)
http://www.tx3.net/
-- =========================

Tom Lahti wrote:

>>> ---------------------------------------------------------
>>> SParseLocal=98
>>>
>>> # always call reverse_srs
>>> R$* $@ $(reverse_srs $1 $f $)
>>> ---------------------------------------------------------
>>
>> I am pretty sure you did not test your code; $f needs to be
>> dereferenced here, and should be written as $&f instead.
>
> I'm pretty sure I didn't either :) $&f does work quite a bit
> better. However, it seems that this gets called twice for each
> email; once for the SMTP envelope sender with $f empty, and a second
> time with the recipient and envelope sender in the correct place.
> Apparently, the envelope sender is passed through ParseLocal with $f
> unset. I do not know why, or how to get around this, since having $f
> empty is precisely what I'm looking for, so I can't use that as an
> indicator that the the $old_address is in fact the sender and not the
> recipient.

I cannot corroborate this. I had srs2envtol log ptr, client_ip, env-from,
and recipient; and, in both instances, $f remains defined. A mail sent over
the internal network:

SRS2ENVTOL: my-xp-machine.org, 192.168.5.3, admin@asarian-host.net,
SRS0=JAJJYUm5=IT=asarian-host.net=admin<@asarian-host.net.>
SRS2ENVTOL: my-xp-machine.org, 192.168.5.3, admin@asarian-host.net,
SRS0=JAJJYUm5=IT=asarian-host.net=admin<@asarian-host.net.>

And one coming from a remote location:

SRS2ENVTOL: xs1.xs4all.nl, 194.109.21.2, admin@asarian-host.net,
SRS0=JAJJYUm5=IT=asarian-host.net=admin<@asarian-host.net.>

No problem with $f at all. I am curious how you tested your setup.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

On Mon, 24 May 2004, Tom Lahti wrote:

> In the page http://asarian-host.net/srs/sendmailsrs.htm an SRS integration
> with sendmail is presented. It is implied in the opening paragraphs that
...
> However, this is NOT the case. There are no modifications to the sendmail
> config file presented on the web page, which would generate a 550 error
> after the data phase if the bounce recipient is not SRS signed. What this
> implementation WILL do is reject a bounce with an _invalid_ SRS signed
> recipient, but if the recipient is not SRS signed and is nonetheless a
> valid address, the bounce will be delivered anyway.

I know you have been working on the Perl implementation, but you might
want to look at my Python implementation at

http://www.bmsi.com/pysrs.html

I includes a Redhat RPM that installs the m4 package (with bug fixes)
and will generate the 550 error for unsigned bounces if you also use
Python milter.

http://www.bmsi.com/milter.html

I refer to running the supplied production milter, not the sample.py
tutorial milter. The RPM installs the production milter with Redhat
sysvinit scripts.

I am running this system on on 6 production mail servers with 6 - 70
users each.

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Stuart D. Gathman wrote:

> On Mon, 24 May 2004, Tom Lahti wrote:
>
>> In the page http://asarian-host.net/srs/sendmailsrs.htm an SRS
>> integration with sendmail is presented. It is implied in the
>> opening paragraphs that .... However, this is NOT the case. There
>> are no modifications to the sendmail config file presented on the
>> web page, which would generate a 550 error after the data phase if
>> the bounce recipient is not SRS signed. What this implementation
>> WILL do is reject a bounce with an _invalid_ SRS signed recipient,
>> but if the recipient is not SRS signed and is nonetheless a valid
>> address, the bounce will be delivered anyway.
>
> I know you have been working on the Perl implementation, but you might
> want to look at my Python implementation ...
>
> and will generate the 550 error for unsigned bounces if you also use
> Python milter.

Perhaps in abundance, but spf-milter, as of 1.40, at:

http://spf.pobox.com/sendmail-milter-spf-1.41.pl

also has full support for "fake DSN detection" (-S), using SRS. A Milter,
imho, is really a good place to implement such a policy.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

>[snip]
>No problem with $f at all. I am curious how you tested your setup.

I had my modified srs2envtol.pl log its input/what it was doing. I get two
log entries per email sent. I think it might be because I'm using
MailScanner, so I have two sendmail queue runners. I think I need a
separate config file for each queue runner and just use SRS on the second,
final delivering queue runner.

>http://spf.pobox.com/sendmail-milter-spf-1.41.pl
>
>also has full support for "fake DSN detection" (-S), using SRS. A Milter,
>imho, is really a good place to implement such a policy.

Agreed. However, I'd like to be able to do fake DSN rejection without
implementing SPF. Upon a cursory look at the perl milter code it seems
that SPF is required for this milter to function. How difficult would it
be to modify it so that it can do the SRS+fake DSN rejection without
implementing SPF? I might be able to do it eventually (I'm much better at
perl coding than sendmail config'ing), but if you know this code you might
wrap a couple of blocks of code in "if" statements and introduce and new
command line switch in a couple of minutes....


--
-- =========================
Tom Lahti
Tx3 Online Services

(888)4-TX3-SVC (489-3782)
http://www.tx3.net/
-- =========================

>I know you have been working on the Perl implementation, but you might
>want to look at my Python implementation at
>
>http://www.bmsi.com/pysrs.html
>
>I includes a Redhat RPM that installs the m4 package (with bug fixes)
>and will generate the 550 error for unsigned bounces if you also use
>Python milter.
>
>http://www.bmsi.com/milter.html
>
>I refer to running the supplied production milter, not the sample.py
>tutorial milter. The RPM installs the production milter with Redhat
>sysvinit scripts.
>
>I am running this system on on 6 production mail servers with 6 - 70
>users each.

Appreciate your input, however both of your links are bad (404 error).


--
-- =========================
Tom Lahti
Tx3 Online Services

(888)4-TX3-SVC (489-3782)
http://www.tx3.net/
-- =========================

On Wed, 26 May 2004, Tom Lahti wrote:


> Appreciate your input, however both of your links are bad (404 error).

Ooops. Sorry.

http://www.bmsi.com/python/pysrs.html
http://www.bmsi.com/python/milter.html

--
Stuart D. Gathman <stuart@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

>Perhaps in abundance, but spf-milter, as of 1.40, at:
>
>http://spf.pobox.com/sendmail-milter-spf-1.41.pl
>
>also has full support for "fake DSN detection" (-S), using SRS. A Milter,
>imho, is really a good place to implement such a policy.
>
>- Mark

I have implement sendmail-milter-spf-1.41.pl and found that the 550 error
occurs immediately upon the RCPT command and NOT after DATA command. Is
this a limitation of milter capability? It seems this breaks valid SMTP
probes.


--
-- =========================
Tom Lahti
Tx3 Online Services

(888)4-TX3-SVC (489-3782)
http://www.tx3.net/
-- =========================

[snip]
>Perhaps in abundance, but spf-milter, as of 1.40, at:
>
>http://spf.pobox.com/sendmail-milter-spf-1.41.pl
>
>also has full support for "fake DSN detection" (-S), using SRS. A Milter,
>imho, is really a good place to implement such a policy.

After a week of using sendmail-milter-spf-1.41.pl, I have come to the
conclusion that it is far from rock solid stable. It may work for long
periods on low volume mail servers, but its limitations become apparent
when you throw some real load at it.

To indicate my volume, an average day consists of 20,000+ connections
rejected do to RBL+ lookups, and another 85,000+ due to unsigned SRS
addresses in the SMTP envelope. Then we drop 45,000+ emails by
SpamAssassin, and MailScanner catches 1,000 viruses in the remainder. And
after all that, we still have 100,000+ emails being delivered per day.

The sendmail-milter-spf does not handle this kind of volume. The first
thing I noticed is the memory leaks. The RSS of the process slowly
climbs. I wrote a script that runs from cron and checks on it and when it
gets to 80MB it restarts. That is currently happening about once every 21
hours.

The second thing is it dies periodically. Last night I was paged out of
bed twice because the sendmail-milter-spf process had died and I had to
restart sendmail. When it dies, the sendmail processes stack up quickly,
run my machine out of RAM (1GB) and by the time I get paged (less than 5
minutes) its got 600MB+ in the swap file, all full of sendmail processes.

This all is not unusual for milters though, I have previously experienced
similiar instability with all sorts of milters, including MIMEDefang and a
myriad of other milters that try to perform the same function (which is why
I now using MailScanner for that purpose). The only milter I ever used
that was truly stable was rays-filter, because it tried to do very little,
and the less you do the less that goes wrong. In any event, what is really
needed is a way to do this without using the milter interface, i.e. a
direct patch to sendmail, perhaps.


--
-- =========================
Tom Lahti
Tx3 Online Services

(888)4-TX3-SVC (489-3782)
http://www.tx3.net/
-- =========================

Tom Lahti wrote:

> [snip]
>
>> Perhaps in abundance, but spf-milter, as of 1.40, at:
>>
>> http://spf.pobox.com/sendmail-milter-spf-1.41.pl
>>
>> also has full support for "fake DSN detection" (-S), using SRS. A
>> Milter, imho, is really a good place to implement such a policy.
>
> After a week of using sendmail-milter-spf-1.41.pl, I have come to the
> conclusion that it is far from rock solid stable. It may work for
> long periods on low volume mail servers, but its limitations become
> apparent when you throw some real load at it.
>
> The sendmail-milter-spf does not handle this kind of volume. The
> first thing I noticed is the memory leaks. The RSS of the process
> slowly climbs. I wrote a script that runs from cron and checks on it
> and when it gets to 80MB it restarts. That is currently happening
> about once every 21 hours.

I think this is actually a Perl problem itself. I have a similar problem on
a news server in Perl. It seems memory occupied by strings, especially
hashes, gets poorly freed, even if you specifically delete or undef the
content.

spf-milter will always grow a bit since startup, btw. It could grow with max
64K segments (for body chunks), times the amount of pending connections.
Those spurts would be temporary, though. Memory, as in my news server,
really seems to get lost over time as deleted hash-space is not freed up. I
have yet to find a solution for that.

When perl 5.8.0 was released, it leaked memory like crazy, especially using
ithreads. It is better in 5.8.4, but remains an issue. There are, as far as
I can tell, no identifiable, obvious places in spf-milter where memory is
not freed where it should have been. All private per-connection data gets
destroyed at close_callback: $ctx -> setpriv (undef); it may very well be
that Sendmail::Milter itself leaks the memory.


> The second thing is it dies periodically.

This has not yet happened to me; you are the first to report this. So I
cannot comment on why this would occur.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

>When perl 5.8.0 was released, it leaked memory like crazy, especially using
>ithreads. It is better in 5.8.4, but remains an issue. There are, as far as
>I can tell, no identifiable, obvious places in spf-milter where memory is
>not freed where it should have been. All private per-connection data gets
>destroyed at close_callback: $ctx -> setpriv (undef); it may very well be
>that Sendmail::Milter itself leaks the memory.

I was thinking possibly a package that you used in the code, i.e.
Sendmail::Milter or Mail::SPF::Query, but I suppose its more likely perl
itself.

> > The second thing is it dies periodically.
>
>This has not yet happened to me; you are the first to report this. So I
>cannot comment on why this would occur.

What kind of volume do you process? I suppose it might be that I'm
currently running this on Fedora Core 1, which has funky Redhat gcc stuff
(which was used to compile sendmail and milter libs). I have been thinking
of switching the mail server to Debian. What do you run it on?


--
-- =========================
Tom Lahti
Tx3 Online Services

(888)4-TX3-SVC (489-3782)
http://www.tx3.net/
-- =========================

Tom Lahti wrote:

>> When perl 5.8.0 was released, it leaked memory like crazy,
>> especially using ithreads. It is better in 5.8.4, but remains an
>> issue. There are, as far as I can tell, no identifiable, obvious
>> places in spf-milter where memory is not freed where it should have
>> been. All private per-connection data gets destroyed at
>> close_callback: $ctx -> setpriv (undef); it may very well be that
>> Sendmail::Milter itself leaks the memory.
>
> I was thinking possibly a package that you used in the code, i.e.
> Sendmail::Milter or Mail::SPF::Query, but I suppose its more likely
> perl itself.

I am pretty sure Mail::SPF::Query is not the culprit. With Sendmail::Milter
it is really hard to tell whether it leaks the memory itself, or whether
Perl running ithreads causes it.

Perhaps I should start looking into Sendmail::Pmilter. It is a
Sendmail::Milter API emulation without requiring that threads be used. It
offers fork-on-request ("postfork") and child-pool ("prefork") means of
dispatching requests to the protocol engine. The author emailed me about it,
relatively recently; yet I have not gotten around to checking it out. The
longer I think about it, the more I like it, though. :) A nice pre-forked
PMilter model would be much better than having to use the leaky ithreads
(plus, that if anything were to go wrong, in terms of internal errors, only
the child would die).

>>> The second thing is it dies periodically.
>>
>> This has not yet happened to me; you are the first to report this.
>> So I cannot comment on why this would occur.
>
> What kind of volume do you process?

Less than you. Around 40,000 emails a day; 21,000 of which comes from
dynamic IP space (or RBL early-out); another 13,000 is marked as spam by SA;
and around 1,200 are fake DSN messages.

> I suppose it might be that I'm
> currently running this on Fedora Core 1, which has funky Redhat gcc
> stuff (which was used to compile sendmail and milter libs). I have
> been thinking of switching the mail server to Debian. What do you
> run it on?

I run it on FreeBSD 4.9R, gcc version 2.95.4. That is a known, stable
platform (no gcc issues). Apart from slowly leaking memory, spf-milter has
never died on me yet.

- Mark

System Administrator Asarian-host.org

---
"If you were supposed to understand it,
we wouldn't call it code." - FedEx

> [snip]
> The second thing is it dies periodically.

This has not yet happened to me; you are the first to report this. So I
cannot comment on why this would occur.

I concur, I have been running 1.41 since it as publically available and
have yet to restart it on a medium volume mail server running on a
Fedora Core 1 server.

Michael Weiner