Mailing List Archive

I think you should start by telling us what hardware you are running, disk
subsystem, amount of RAM, processor(s) and speed, and what is running on
that box aside from spamassin (sendmail/postfix/exim/?, DNS, etc), and
whether you're using RBL checks and additional rulesets. Any virus scanning
done on the box?

And last, but not least, what are your mail volumes (quantity and size)?

Cheers,

Phil

---------------------------------------------
Phil Randal
Network Engineer
Herefordshire Council
Hereford, UK

> -----Original Message-----
> From: Frank DeChellis DSL [mailto:frankd@iaw.on.ca]
> Sent: 13 February 2004 13:11
> To: spamassassin-users@incubator.apache.org
> Subject: Hardware and software setup questions
>
>
> Hi.
>
> I was wondering if I could take a straw poll of sorts. We're
> having some
> trouble with SA in terms of loading down the system it's on.
>
> May I respectfully ask some of you what kind of hardware you run
> (Processor and RAM) compared to how many concurrent SA processes you
> allow?
>
> We run a seaprate server for SA but it still seems to use
> more CPU than it
> should. I have 2.63 installed and running.
>
> Thanks
> Frank
>
> --------------------------------------------------------------
> ---------
> Frank DeChellis
> Internet Access Worldwide
> 3 East Main Street Welland, Ontario, Canada L3B 3W4
> 905-714-1400 fax 905-732-0524
> www.iaw.com
> ----------------------------------------------------------------------
>

Frank DeChellis DSL wrote:

> Hi.
>
> I was wondering if I could take a straw poll of sorts. We're having some
> trouble with SA in terms of loading down the system it's on.
>
> May I respectfully ask some of you what kind of hardware you run
> (Processor and RAM) compared to how many concurrent SA processes you
> allow?

2 x Intel(R) Xeon(TM) CPU 2.40GHz, 2 GB RAM, 50 concurrent procs.

The CPU might be overkill, but the RAM isn't; We tried 500 MB and 900 MB,
both of which caused the box to thrash at peak levels.

- Ole.

Frank DeChellis DSL wrote:
> Hi.
>
> I was wondering if I could take a straw poll of sorts. We're having some
> trouble with SA in terms of loading down the system it's on.
>
> May I respectfully ask some of you what kind of hardware you run
> (Processor and RAM) compared to how many concurrent SA processes you
> allow?
>
> We run a seaprate server for SA but it still seems to use more CPU than it
> should. I have 2.63 installed and running.
>
> Thanks
> Frank
>

I'm running a dual P-II 450 box with 512MB RAM, 9GB SCSI HD. Bayes DB is
shared, unfortunately. I've got network tests on, Razor2, DCC, some
extra RBLs, plus extra rulesets (all of Jennifer's rules, BigEvil,
evilnumbers and antidrug). The box only runs spamd, the main SMTP server
connects over the network. No unnecessary processes are running on this box.

Currently I've got --max-children set to 27. The server processes around
23K messages per day.

I'm going to be setting up a redundant server with a similar config
soon. This server handles the overall load alright, but sudden floods of
mail (like when one of my amavis-running MX boxes has trophie fail to
restart... argh!) can kill the box. Plus then I'll have some redundancy,
and room to grow.

I'm *just* starting to see the machine get processor-bound, and only on
heavy mail periods. Otherwise, it's just a matter of loading up with
enough RAM to have parallel spamd processes running.

--Rich

On Fri, Feb 13, 2004 at 10:41:23AM -0600, Rich Puhek wrote:

> I'm running a dual P-II 450 box with 512MB RAM, 9GB SCSI HD. Bayes DB is
> shared, unfortunately. I've got network tests on, Razor2, DCC, some
> extra RBLs, plus extra rulesets (all of Jennifer's rules, BigEvil,
> evilnumbers and antidrug). The box only runs spamd, the main SMTP server
> connects over the network. No unnecessary processes are running on this box.
>
> Currently I've got --max-children set to 27. The server processes around
> 23K messages per day.

We have a dedicated server with a single AMD Duron 600 with 384M RAM,
10G IDE HD. 384M swap, which does not seem to come into play.
Linux-2.4.24 kernel.

Razor2 is disabled but network tests are on, some extra RBLs, plus the same
extra rule sets. SA 2.63. --max-children 30. Common Bayes Db located on hard
drive local to the dedicated SA server. From local.cf:

bayes_learn_to_journal 1
bayes_path /var/lib/spamassassin/bayes
bayes_file_mode 777

and sa-learn run from crontab once an hour.

Previously we'd used individual Bayes DBs but found the NFS load
imposed for the user directories was sometimes excessive and in
addition maintaining the individual Bayes DBs through various
upgrade-related breakages was taking too much sysadmin effort.

Razor server slowness or outages had previously appeared to have
been associated with process load spiraling out of control. We
have not observed that since we turned off Razor2.

Screening about 12,000 messages/24h, roughly 2600 detected as ham,
the remainder spam. Some of the worst spamsites are prescreened
using an RBL at the MTA level.

The box never seems to break into a sweat. I'm running a monitor
script which looks for rapid increase of swapspace used and restarts
spamd if it finds this. So far the script has not triggered even
once in several months. "sar -q" run daily never sees load average
in excess of 2.0 on either the one-minute or the five-minute periods.

--
Dan Wilder

Dan Wilder wrote:


> We have a dedicated server with a single AMD Duron 600 with 384M RAM,
> 10G IDE HD. 384M swap, which does not seem to come into play.
> Linux-2.4.24 kernel.
>
> Razor2 is disabled but network tests are on, some extra RBLs, plus the same
> extra rule sets. SA 2.63. --max-children 30. Common Bayes Db located on hard
> drive local to the dedicated SA server. From local.cf:
>

Ouch! your --max-children could be a problem. In my experience, once
spamd starts swapping, you're in big trouble. You might have been able
to survive the Razor problems you mention below if -m was lower.

In my case, guessed at an initial number of children based on the memory
usage of one spamd process, then I fine tuned that value after observing
the server under high load. I started with 30, which was occasionally
swapping at high loads, and ended up at 27.

> bayes_learn_to_journal 1
> bayes_path /var/lib/spamassassin/bayes
> bayes_file_mode 777
>
> and sa-learn run from crontab once an hour.
>
> Previously we'd used individual Bayes DBs but found the NFS load
> imposed for the user directories was sometimes excessive and in
> addition maintaining the individual Bayes DBs through various
> upgrade-related breakages was taking too much sysadmin effort.
>
Ugh, yes, keep the bayes (plus any razor-related stuff) on local drives.
I'm even pondering a small partition mounted with noatime (any one with
thoughts on bayes and a noatime mount?)

> Razor server slowness or outages had previously appeared to have
> been associated with process load spiraling out of control. We
> have not observed that since we turned off Razor2.
>
> Screening about 12,000 messages/24h, roughly 2600 detected as ham,
> the remainder spam. Some of the worst spamsites are prescreened
> using an RBL at the MTA level.
>
Same here, RBLs at the MTA helps greatly!

> The box never seems to break into a sweat. I'm running a monitor
> script which looks for rapid increase of swapspace used and restarts
> spamd if it finds this. So far the script has not triggered even
> once in several months. "sar -q" run daily never sees load average
> in excess of 2.0 on either the one-minute or the five-minute periods.

Why not just set the max-children limit to prevent swapping? That would
seem gentler, since the 20+ messages currently being processed will get
through, and if you get lucky, other messages will come through
eventually as well.

Ideally, spamd would hold off forking if not enough free memory was
available. I can't find any portable way to determine free RAM, though.

--Rich

Dan Wilder wrote:
> The box never seems to break into a sweat. I'm running a monitor
> script which looks for rapid increase of swapspace used and restarts

Unreliable approach. Use your shell's ulimit or a program like
'softlimit' to limit the server's memory usage. Use 'runit' or
'daemontools' to restart the service if it exits.

--
Anybody practicing the fine art of composing music, no matter how
cynical or greedy or scared, still cant help serving all humanity.
Music makes practically everybody fonder of life than he or she
would be without it.
--Kurt Vonnegut

On Fri, Feb 13, 2004 at 01:30:27PM -0500, Dan Melomedman wrote:
> Dan Wilder wrote:
> > The box never seems to break into a sweat. I'm running a monitor
> > script which looks for rapid increase of swapspace used and restarts
>
> Unreliable approach. Use your shell's ulimit or a program like
> 'softlimit' to limit the server's memory usage. Use 'runit' or
> 'daemontools' to restart the service if it exits.

In what respect is ulimit superior to well-tuned --max-child-processes?
Recall we're looking at a runaway aggregate load, not a single-process
problem.

softlimit appears from the documentation to be mostly a DJB extension
to the ulimit concept. Again, what does this have to do with
excessive aggregate usage?

At the present time the monitor is just a canary in the coalmine.
A very healthy canary, I'd like to add. One that consumes zero
sysadmin time.

--
Dan Wilder

Dan Wilder wrote:
> In what respect is ulimit superior to well-tuned --max-child-processes?
> Recall we're looking at a runaway aggregate load, not a single-process
> problem.

A memory leak in perl for is a good way to eat all your memory. You
should use OS memory limits for your services.

> softlimit appears from the documentation to be mostly a DJB extension
> to the ulimit concept. Again, what does this have to do with
> excessive aggregate usage?


Wrong. softlimit is a program that sets rlimit()s, then runs your
program. rlimit() is the same system call your shell calls when you tell
it to use its ulimit.