Mailing List Archive

RBL checks and milter
Hi all,

I've been using spamass-milter in production for quite some time now, on
a single FreeBSD/sendmail server that sustains 2,500 messages per hour
during peak times. I'm currently using Mail::SpamAssassin 2.61.

With RBL checks enabled, spamc takes several seconds (7.8, on average)
to process each message. I expect it to. However, the problem is that
when run as a milter, the sending MTA is forced to wait, and, in some
cases (i.e., when spamc needs somewhat longer than 7.8 seconds), this is
causing timeouts with delivery, and consuming needless resources
(sockets and RAM) to filter spam in "real time", when I'll never have
any real-time status to report to the sender. With RBL enabled, the
system becomes extremely sluggish (load > 10.00) and starts rejecting
connections (451 4.7.1 Try again later), timing out, or just ignoring
the filter (thanks to "F="). Without RBL, it's very snappy, and load
averages hover < 1.00.

I'm also using clamav-milter, so here's the relevant M4 (mind the line
breaks). Nothing special:

INPUT_MAIL_FILTER(`clamav',`S=local:/var/run/clmilter.sock, F=,
T=S:4m;R:4m')dnl
INPUT_MAIL_FILTER(`spamassassin', `S=local:/var/run/spamass-milter.sock,
F=,T=C:15m;S:4m;R:4m;E:10m')

For now, I've disabled RBL checks with spamd -L, and the combination of
clamav and spamassassin takes much less than a second for the same
messages tested with RBL enabled, which is definitely acceptable. I'd
really like to continue to reap the benefits of RBL, though!

Where should I look next?

Thanks,
- Ryan
RE: RBL checks and milter [ In reply to ]
I received a good off-list reply that I'm responding to, here.
I've anonymized it to protect the innocent.

> Ryan Thompson wrote:
>
> > With RBL checks enabled, spamc takes several seconds (7.8, on
> > average) to process each message. I expect it to. However, the
> > problem is that when run as a milter, the sending MTA is forced to
> > wait, and, in some cases (i.e., when spamc needs somewhat longer
> > than 7.8 seconds), this is causing timeouts with delivery, and
> > consuming needless resources (sockets and RAM) to filter spam in
> > "real time", when I'll never have any real-time status to report to
> > the sender. [...]
>
> Ryan,
>
> I've offloaded the RBL into sendmail using it's built-in blacklist
> features using the following in the sendmail.mc file.
>
> [...]

Hello,

Thanks for the tip. I'd like to benefit from SA's genetic algorithm,
though, with respect to the RBL tests. Having sendmail reject anything
from blacklisted hosts could lead to more false positives than I'd like
to see (i.e., > 0 :-)

> Takes the load off spamd significantly.

I bet it would.

Hmm. What'd be really useful is to have a switch to only use the RBL
checks when necessary to classify a message. If the local tests already
come up with a really high (or low) score (definable threshold in
config, like the auto whitelist), don't bother with the RBL tests. The
only time the RBL tests would be used is if the message score is already
within a certain score interval. Something along those lines would
certainly cut down on the amount of RBL queries generated, at what seems
like a very small risk of incorrectly classifying a message.

That still doesn't solve the slow MTA response problem I described in my
original message, but it *would* improve overall performance. Then
again, I'm not an SA developer, so I don't know exactly how feasable
this would be.

Thoughts?

Thanks,
- Ryan