Mailing List Archive: [OT] Specing out a new box

[OT] Specing out a new box

Feb 10, 2004, 7:56 AM

Post #1 of 8 (1758 views)

Hello all,

I've been lurking here for a little while, and haven't seen this kind of
thing come up, so I was curious if anyone has any suggestions about it.
One of the companies we do backend work for has a very busy mail server
- averages probably 50 messages/minute, with peak burst periods of about
100-110 messages/minute. Running spamc/spamd on that box resulted in
huge load averages (30-40 or more) at peak time, and loads of 15 or 16
at normal time.

We are in the process of revamping things, and will probably move to a
distributed spamd, but we are also thinking about replacing the mail
server itself. We need a little more juice to do other things like
running clamd and whatnot. I was just wondering what hardware other
people are using for this volume of email - ~75000 emails/day, some days
more (especially when one of these worms pops up it's head). Right now
it's running on an smp AMD K7 or so, but we're looking for suggestions
for upgrade.

Thanks for any advice on this,
--
--------------------------------------------------------------------------
| Stephen Gran | Q: What looks like a cat, flies like a |
| steve@lobefin.net | bat, brays like a donkey, and plays |
| http://www.lobefin.net/~steve | like a monkey? A: Nothing. |
--------------------------------------------------------------------------

RE: [OT] Specing out a new box [ In reply to ]

prandal at herefordshire

Feb 10, 2004, 8:06 AM

Post #2 of 8 (1741 views)

Permalink

What sort of mail volume (MB) are you talking about here?

We handle about 8000 emails a day totalling around 600MB (those figures are
fairly constant) on a single processor 2.4GHz P4 Xeon-based 1 GB RAM Dell
2650 mail gateway box running Fedora Core 1, Sendmail, MailScanner,
MailWatch, mailscanner-mrtg, and virus scanning with both ClamAv and McAfee
uvscan with a load average hovering around 0.2 (max in the last 24 hours was
0.7).

I'd hazard a guess that dual Xeon 3.2GHz processors would do the job for you
at a much lower load average than you're experiencing now.

Cheers,

Phil

---------------------------------------------
Phil Randal
Network Engineer
Herefordshire Council
Hereford, UK

> -----Original Message-----
> From: Stephen Gran [mailto:steve@lobefin.net]
> Sent: 10 February 2004 14:56
> To: Spamassassin-Users
> Subject: [OT] Specing out a new box
>
>
> Hello all,
>
> I've been lurking here for a little while, and haven't seen
> this kind of
> thing come up, so I was curious if anyone has any suggestions
> about it.
> One of the companies we do backend work for has a very busy
> mail server
> - averages probably 50 messages/minute, with peak burst
> periods of about
> 100-110 messages/minute. Running spamc/spamd on that box resulted in
> huge load averages (30-40 or more) at peak time, and loads of 15 or 16
> at normal time.
>
> We are in the process of revamping things, and will probably move to a
> distributed spamd, but we are also thinking about replacing the mail
> server itself. We need a little more juice to do other things like
> running clamd and whatnot. I was just wondering what hardware other
> people are using for this volume of email - ~75000
> emails/day, some days
> more (especially when one of these worms pops up it's head).
> Right now
> it's running on an smp AMD K7 or so, but we're looking for suggestions
> for upgrade.
>
> Thanks for any advice on this,
> --
>
> --------------------------------------------------------------
> ------------
> | Stephen Gran | Q: What looks like a cat,
> flies like a |
> | steve@lobefin.net | bat, brays like a donkey,
> and plays |
> | http://www.lobefin.net/~steve | like a monkey? A: Nothing.
> |
>
> --------------------------------------------------------------
> ------------
>

Re: [OT] Specing out a new box [ In reply to ]

martinh at solid-state-logic

Feb 10, 2004, 8:08 AM

Post #3 of 8 (1736 views)

Permalink

Stephen Gran wrote:
> Hello all,
>
> I've been lurking here for a little while, and haven't seen this kind of
> thing come up, so I was curious if anyone has any suggestions about it.
> One of the companies we do backend work for has a very busy mail server
> - averages probably 50 messages/minute, with peak burst periods of about
> 100-110 messages/minute. Running spamc/spamd on that box resulted in
> huge load averages (30-40 or more) at peak time, and loads of 15 or 16
> at normal time.
>
> We are in the process of revamping things, and will probably move to a
> distributed spamd, but we are also thinking about replacing the mail
> server itself. We need a little more juice to do other things like
> running clamd and whatnot. I was just wondering what hardware other
> people are using for this volume of email - ~75000 emails/day, some days
> more (especially when one of these worms pops up it's head). Right now
> it's running on an smp AMD K7 or so, but we're looking for suggestions
> for upgrade.
>
> Thanks for any advice on this,
Stephen

I know of serveral sites that use Dual 2ghz Xeon's and handle this of
load regularly. BUT they use MailScanner to front end SA, which I think
can be tuned for high loads better than using spamd on it's own....

--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300

**********************************************************************

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.

**********************************************************************

Re: [OT] Specing out a new box [ In reply to ]

rickm at nougen

Feb 10, 2004, 8:09 AM

Post #4 of 8 (1735 views)

Permalink

Stephen Gran wrote:

> Hello all,
>
> I've been lurking here for a little while, and haven't seen this kind of
> thing come up, so I was curious if anyone has any suggestions about it.
> One of the companies we do backend work for has a very busy mail server
> - averages probably 50 messages/minute, with peak burst periods of about
> 100-110 messages/minute. Running spamc/spamd on that box resulted in
> huge load averages (30-40 or more) at peak time, and loads of 15 or 16
> at normal time.
>
> We are in the process of revamping things, and will probably move to a
> distributed spamd, but we are also thinking about replacing the mail
> server itself. We need a little more juice to do other things like
> running clamd and whatnot. I was just wondering what hardware other
> people are using for this volume of email - ~75000 emails/day, some days
> more (especially when one of these worms pops up it's head). Right now
> it's running on an smp AMD K7 or so, but we're looking for suggestions
> for upgrade.

Hi,

We are running a 2.8Ghz P4 with 1 gig of ram as our inbound
pre-filtering mail server. It runs qmail with qmail-scanner and clamd
and uses spamc to process them through SpamAssassin running on a second
machine.

It's average email processing (not smtp connections as a lot are blocked
via a few rbl's, badmailfrom entries and the like) is 75 messages per
minute with bursts to over 175 per minute. Load averages stay around
0.8 and I have seen it go as high as 2.x or 3.x for small periods of time.

The SpamAssassin machine is actually just going into production now and
it is a 3 Ghz P4 with 1.5 gigs of memory. We ran it last night during
our peak user period and it processed everything we threw at it (the
above mail server plus another mail server) with load averages hovering
around 0.45.

All of our machines run FreeBSD or Linux Slackware. These two happen to
be running FreeBSD 4.8.

HTH,

Rick

Re: [OT] Specing out a new box [ In reply to ]

davidb at pins

Feb 10, 2004, 8:12 AM

Post #5 of 8 (1734 views)

Permalink

We have an old Sun E3500 with 4GB and 8 360MHz processors here. It does
full SpamAssassin (RBLs and all) and Sophos-based antivirus. It's doing
about 120-150k messages a day, and can average 1-2 mails per second
without too much trouble. The biggest delays seems to be RBL-type
queries. The load has never gone above 4-5, running up to 12 spamassassin
processes in a daemon mode.

David.

-----

On Tue, 10 Feb 2004, Rick Macdougall wrote:

>
>
> Stephen Gran wrote:
>
> > Hello all,
> >
> > I've been lurking here for a little while, and haven't seen this kind of
> > thing come up, so I was curious if anyone has any suggestions about it.
> > One of the companies we do backend work for has a very busy mail server
> > - averages probably 50 messages/minute, with peak burst periods of about
> > 100-110 messages/minute. Running spamc/spamd on that box resulted in
> > huge load averages (30-40 or more) at peak time, and loads of 15 or 16
> > at normal time.
> >
> > We are in the process of revamping things, and will probably move to a
> > distributed spamd, but we are also thinking about replacing the mail
> > server itself. We need a little more juice to do other things like
> > running clamd and whatnot. I was just wondering what hardware other
> > people are using for this volume of email - ~75000 emails/day, some days
> > more (especially when one of these worms pops up it's head). Right now
> > it's running on an smp AMD K7 or so, but we're looking for suggestions
> > for upgrade.
>
> Hi,
>
> We are running a 2.8Ghz P4 with 1 gig of ram as our inbound
> pre-filtering mail server. It runs qmail with qmail-scanner and clamd
> and uses spamc to process them through SpamAssassin running on a second
> machine.
>
> It's average email processing (not smtp connections as a lot are blocked
> via a few rbl's, badmailfrom entries and the like) is 75 messages per
> minute with bursts to over 175 per minute. Load averages stay around
> 0.8 and I have seen it go as high as 2.x or 3.x for small periods of time.
>
> The SpamAssassin machine is actually just going into production now and
> it is a 3 Ghz P4 with 1.5 gigs of memory. We ran it last night during
> our peak user period and it processed everything we threw at it (the
> above mail server plus another mail server) with load averages hovering
> around 0.45.
>
> All of our machines run FreeBSD or Linux Slackware. These two happen to
> be running FreeBSD 4.8.
>
> HTH,
>
> Rick
>
>
>

RE: [OT] Specing out a new box [ In reply to ]

csanterre at MerchantsOverseas

Feb 10, 2004, 8:41 AM

Post #6 of 8 (1745 views)

Permalink

Good grief there is some big iron being used :) *server envy*

The only thing I have to offer is get lots of memory.

Now a slight hijack......

Do any of you guys with big iron and high traffic use any custom rulesets?
The group is still curious about the effects they might have on high volume.
We are trying to write fast regex. But without having a high traffic server
I have no idea.

--Chris

> -----Original Message-----
> From: David Birnbaum [mailto:davidb@pins.net]
> Sent: Tuesday, February 10, 2004 10:12 AM
> To: Rick Macdougall
> Cc: Stephen Gran; Spamassassin-Users
> Subject: Re: [OT] Specing out a new box
>
>
> We have an old Sun E3500 with 4GB and 8 360MHz processors
> here. It does
> full SpamAssassin (RBLs and all) and Sophos-based antivirus.
> It's doing
> about 120-150k messages a day, and can average 1-2 mails per second
> without too much trouble. The biggest delays seems to be RBL-type
> queries. The load has never gone above 4-5, running up to 12
> spamassassin
> processes in a daemon mode.
>
> David.
>
> -----
>
> On Tue, 10 Feb 2004, Rick Macdougall wrote:
>
> >
> >
> > Stephen Gran wrote:
> >
> > > Hello all,
> > >
> > > I've been lurking here for a little while, and haven't
> seen this kind of
> > > thing come up, so I was curious if anyone has any
> suggestions about it.
> > > One of the companies we do backend work for has a very
> busy mail server
> > > - averages probably 50 messages/minute, with peak burst
> periods of about
> > > 100-110 messages/minute. Running spamc/spamd on that box
> resulted in
> > > huge load averages (30-40 or more) at peak time, and
> loads of 15 or 16
> > > at normal time.
> > >
> > > We are in the process of revamping things, and will
> probably move to a
> > > distributed spamd, but we are also thinking about
> replacing the mail
> > > server itself. We need a little more juice to do other
> things like
> > > running clamd and whatnot. I was just wondering what
> hardware other
> > > people are using for this volume of email - ~75000
> emails/day, some days
> > > more (especially when one of these worms pops up it's
> head). Right now
> > > it's running on an smp AMD K7 or so, but we're looking
> for suggestions
> > > for upgrade.
> >
> > Hi,
> >
> > We are running a 2.8Ghz P4 with 1 gig of ram as our inbound
> > pre-filtering mail server. It runs qmail with
> qmail-scanner and clamd
> > and uses spamc to process them through SpamAssassin running
> on a second
> > machine.
> >
> > It's average email processing (not smtp connections as a
> lot are blocked
> > via a few rbl's, badmailfrom entries and the like) is 75
> messages per
> > minute with bursts to over 175 per minute. Load averages
> stay around
> > 0.8 and I have seen it go as high as 2.x or 3.x for small
> periods of time.
> >
> > The SpamAssassin machine is actually just going into
> production now and
> > it is a 3 Ghz P4 with 1.5 gigs of memory. We ran it last
> night during
> > our peak user period and it processed everything we threw at it (the
> > above mail server plus another mail server) with load
> averages hovering
> > around 0.45.
> >
> > All of our machines run FreeBSD or Linux Slackware. These
> two happen to
> > be running FreeBSD 4.8.
> >
> > HTH,
> >
> > Rick
> >
> >
> >
>

Re: [OT] Specing out a new box [ In reply to ]

rickm at nougen

Feb 10, 2004, 8:43 AM

Post #7 of 8 (1735 views)

Permalink

Chris Santerre wrote:

> Good grief there is some big iron being used :) *server envy*
>
> The only thing I have to offer is get lots of memory.
>
> Now a slight hijack......
>
> Do any of you guys with big iron and high traffic use any custom rulesets?
> The group is still curious about the effects they might have on high volume.
> We are trying to write fast regex. But without having a high traffic server
> I have no idea.

Hi Chris,

We run big evil and tripwire.

I might add Jennifer's rules as well once the new SA box is fully in
production (hopefully later today).

Regards,

Rick

RE: [OT] Specing out a new box [ In reply to ]

davidb at pins

Feb 10, 2004, 8:53 AM

Post #8 of 8 (1743 views)

Permalink

Big Iron...well, Ebay is your friend. ;)

That server does a lot of other things (like running an entire telco
billing system), so it's hard to abstract out, but I'm running these
extra rulesets right now (thanks to the authors):

99_FVGT_Tripwire.cf
bigevil.cf
antidrug.cf
chickenpox.cf
backhair.cf
evilnumbers.cf
weeds_2.cf

There's really no measurable delay as compared to without them. The bulk
of the time is in razor2, DCC, and RBL. We could easily run 50 processes
on that box before we really ran into CPU problems as opposed to I/O wait
of various flavors. We did see a bit of wait on AWL/Bayes sometimes, but
once I put in my code handle remove_entry quickly, that pretty much went
away.

I think we handle the average email in 2-4 seconds, with the net queries.
Big'uns take more like 5-6, which is the added regex time mostly. I'm
pretty sure this box could do 4-6 emails/second at peak if it was tuned
for it and I took the other junk off.

David.

-----

On Tue, 10 Feb 2004, Chris Santerre wrote:

>
> Good grief there is some big iron being used :) *server envy*
>
> The only thing I have to offer is get lots of memory.
>
> Now a slight hijack......
>
> Do any of you guys with big iron and high traffic use any custom rulesets?
> The group is still curious about the effects they might have on high volume.
> We are trying to write fast regex. But without having a high traffic server
> I have no idea.
>
> --Chris
>
> > -----Original Message-----
> > From: David Birnbaum [mailto:davidb@pins.net]
> > Sent: Tuesday, February 10, 2004 10:12 AM
> > To: Rick Macdougall
> > Cc: Stephen Gran; Spamassassin-Users
> > Subject: Re: [OT] Specing out a new box
> >
> >
> > We have an old Sun E3500 with 4GB and 8 360MHz processors
> > here. It does
> > full SpamAssassin (RBLs and all) and Sophos-based antivirus.
> > It's doing
> > about 120-150k messages a day, and can average 1-2 mails per second
> > without too much trouble. The biggest delays seems to be RBL-type
> > queries. The load has never gone above 4-5, running up to 12
> > spamassassin
> > processes in a daemon mode.
> >
> > David.
> >
> > -----
> >
> > On Tue, 10 Feb 2004, Rick Macdougall wrote:
> >
> > >
> > >
> > > Stephen Gran wrote:
> > >
> > > > Hello all,
> > > >
> > > > I've been lurking here for a little while, and haven't
> > seen this kind of
> > > > thing come up, so I was curious if anyone has any
> > suggestions about it.
> > > > One of the companies we do backend work for has a very
> > busy mail server
> > > > - averages probably 50 messages/minute, with peak burst
> > periods of about
> > > > 100-110 messages/minute. Running spamc/spamd on that box
> > resulted in
> > > > huge load averages (30-40 or more) at peak time, and
> > loads of 15 or 16
> > > > at normal time.
> > > >
> > > > We are in the process of revamping things, and will
> > probably move to a
> > > > distributed spamd, but we are also thinking about
> > replacing the mail
> > > > server itself. We need a little more juice to do other
> > things like
> > > > running clamd and whatnot. I was just wondering what
> > hardware other
> > > > people are using for this volume of email - ~75000
> > emails/day, some days
> > > > more (especially when one of these worms pops up it's
> > head). Right now
> > > > it's running on an smp AMD K7 or so, but we're looking
> > for suggestions
> > > > for upgrade.
> > >
> > > Hi,
> > >
> > > We are running a 2.8Ghz P4 with 1 gig of ram as our inbound
> > > pre-filtering mail server. It runs qmail with
> > qmail-scanner and clamd
> > > and uses spamc to process them through SpamAssassin running
> > on a second
> > > machine.
> > >
> > > It's average email processing (not smtp connections as a
> > lot are blocked
> > > via a few rbl's, badmailfrom entries and the like) is 75
> > messages per
> > > minute with bursts to over 175 per minute. Load averages
> > stay around
> > > 0.8 and I have seen it go as high as 2.x or 3.x for small
> > periods of time.
> > >
> > > The SpamAssassin machine is actually just going into
> > production now and
> > > it is a 3 Ghz P4 with 1.5 gigs of memory. We ran it last
> > night during
> > > our peak user period and it processed everything we threw at it (the
> > > above mail server plus another mail server) with load
> > averages hovering
> > > around 0.45.
> > >
> > > All of our machines run FreeBSD or Linux Slackware. These
> > two happen to
> > > be running FreeBSD 4.8.
> > >
> > > HTH,
> > >
> > > Rick
> > >
> > >
> > >
> >
>
>