Mailing List Archive: Spamd out of memory error on 2.63

Spamd out of memory error on 2.63

Mar 22, 2004, 11:45 AM

Post #1 of 6 (136 views)

Any ideas what causes memory usage to exceed the softlimit I have set...
This is a stock spamassassin 2.63 installed from the RPM downloads.
Not using Bayes, not using AWL, not using Razor, pyzor, or dcc.

Here is a debug log of one that happened at 11:20 today...

[root@spamd1 spamd]# grep 16443 /var/log/maillog
Mar 22 11:20:00 spamd1 spamd[16443]: Still running as root: user not
specified with -u, not found, or set to root. Fall back to nobody.
Mar 22 11:20:00 spamd1 spamd[16443]: processing message
<BAY1-F139GGmNN79xav0000343f@hotmail.com> for rpopp@media-net.net:99.

As you can see, it never completed the 3rd line of output in maillog...
Looking at the debug, you see why...

2004-03-22 11:20:00.720490500 logmsg: connection from p35n0.ruraltel.net
[24.225.0.35] at port 60146
2004-03-22 11:20:00.735873500 debug: retrieving prefs for
rpopp@media-net.net from SQL server
2004-03-22 11:20:00.736508500 logmsg: Still running as root: user not
specified with -u, not found, or set to root. Fall back to nobody.
2004-03-22 11:20:00.775573500 logmsg: processing message
<BAY1-F139GGmNN79xav0000343f@hotmail.com> for rpopp@media-net.net:99.
2004-03-22 11:20:00.781011500 debug: is Net::DNS::Resolver available?
yes
2004-03-22 11:20:00.798152500 debug: looking up PTR record for
'65.54.245.139'
2004-03-22 11:20:00.813954500 debug: PTR for '65.54.245.139':
'bay1-f139.bay1.hotmail.com'
2004-03-22 11:20:00.814153500 debug: received-header: parsed as [
ip=65.54.245.139 rdns=bay1-f139.bay1.hotmail.com helo=hotmail.com by=0
ident= ]
2004-03-22 11:20:00.818235500 debug: looking up PTR record for
'63.245.177.144'
2004-03-22 11:20:00.862131500 debug: PTR for '63.245.177.144':
'63-245-177-144.kitusa.com'
2004-03-22 11:20:00.862208500 debug: received-header: parsed as [
ip=63.245.177.144 rdns=63-245-177-144.kitusa.com helo=
by=by1fd.bay1.hotmail.msn.com ident= ]
2004-03-22 11:20:00.862557500 debug: received-header: 'by' 0 has
reserved IP 0.0.0.0
2004-03-22 11:20:00.862592500 debug: received-header: 'by' 0 has no
public IPs
2004-03-22 11:20:00.862620500 debug: received-header: relay
65.54.245.139 trusted? yes
2004-03-22 11:20:00.884012500 debug: received-header: 'by'
by1fd.bay1.hotmail.msn.com has public IP 65.54.244.250
2004-03-22 11:20:00.884072500 debug: received-header: relay
63.245.177.144 trusted? no
2004-03-22 11:20:03.847334500 Out of memory!

30 more have died in the last few minutes...

[root@spamd1 spamd]# grep "Out of me" spamdlog | tail -30
2004-03-22 11:23:51.492275500 Out of memory!
2004-03-22 11:24:52.796228500 Out of memory!
2004-03-22 11:24:16.447001500 Out of memory!
2004-03-22 11:24:32.954441500 Out of memory!
2004-03-22 11:27:19.036917500 Out of memory!
2004-03-22 11:26:52.732102500 Mar 22 11:26:52.731869 check[28992]: [ 6]
comOut of memory!
2004-03-22 11:27:10.636831500 Out of memory!
2004-03-22 11:27:37.992590500 Out of memory!
2004-03-22 11:27:51.613404500 Out of memory!
2004-03-22 11:28:42.213287500 Out of memory!
2004-03-22 11:28:17.089936500 Out of memory!
2004-03-22 11:28:28.380509500 Out of memory!
2004-03-22 11:28:36.223548500 Mar 22Out of memory!
2004-03-22 11:28:44.234585500 Out of memory!
2004-03-22 11:29:34.610361500 Out of memory!
2004-03-22 11:29:57.089946500 Out of memory!
2004-03-22 11:29:15.976731500 Out of memory!
2004-03-22 11:29:30.547763500 Mar 22 11Out of memory!
2004-03-22 11:29:23.540859500 Out of memory!
2004-03-22 11:30:32.746352500 Out of memory!
2004-03-22 11:29:43.760651500 Mar 22 11:29:43.760427 checOut of memory!
2004-03-22 11:30:10.260573500 Out of memory!
2004-03-22 11:30:26.840217500 Out of memory!
2004-03-22 11:30:27.996426500 Out of memory!
2004-03-22 11:30:36.304536500 Out of memory!
2004-03-22 11:30:53.279664500 Out of memory!
2004-03-22 11:31:21.348780500 Out of memory!
2004-03-22 11:31:31.815492500 Mar 22 11:31:31.815255 check[29620]: [ 6]
Out of memory!
2004-03-22 11:32:45.813325500 Out of memory!
2004-03-22 11:32:49.086297500 Out of memory!
[root@spamd1 spamd]#

Running spamd with just -L, I still get an ass load of them going out of
memory.. Here is a ps of one that went out of memory, so you can see
the memory footprint it was at, and how I call spamd..

nobody 17661 94.1 11.3 93600 86700 ? R 11:44 0:06
/usr/bin/perl -T -w /usr/bin/spamd -L -A 127.0.0.0/8,24.225.0.0/24 -i
0.0.0.0 -D -x -m 15

I just installed 3.0-cvs and don't see this problem occuring... Any
ideas whats up with 2.63?

D

RE: Spamd out of memory error on 2.63 [ In reply to ]

dallase at nmgi

Mar 23, 2004, 12:20 PM

Post #2 of 6 (136 views)

Permalink

>
> I just installed 3.0-cvs and don't see this problem
> occuring... Any ideas whats up with 2.63?
>

Ok, I take that back. I'm seeing "Out of memory" errors in the debug
output with 3.0-cvs also, just not as often. Here were a bunch that
came in in the last few minutes..

[root@spamd4 spamd]# tail -50000 spamdlog | grep "Out of memory"
2004-03-23 12:48:48.695128500 Out of memory!
2004-03-23 12:50:50.735185500 Out of memory!
2004-03-23 12:51:03.151317500 Out of memory!
2004-03-23 12:51:36.312812500 Out of memory!
2004-03-23 12:52:16.608292500 Out of memory!
2004-03-23 12:56:32.809453500 Out of memory!

My softlimit on spamd is 96mb... What could cause this kind of memory
usage per process?

I enabled local tests only with -L and turned off awl and bayes in the
configs, and I still see this occur...

[root@spamd4 spamd]# tail -5000 spamdlog | grep "Out of memory"
2004-03-23 13:15:25.451648500 Out of memory!
2004-03-23 13:16:37.426329500 Out of memory!

Normal RSS values for spamd children are normally 24-25MB, how can 96MB
not be enough when bayes and awl are not running??

Dallas

Re: Spamd out of memory error on 2.63 [ In reply to ]

jm at jmason

Mar 23, 2004, 12:30 PM

Post #3 of 6 (136 views)

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Dallas L. Engelken" writes:
>>=20
>> I just installed 3.0-cvs and don't see this problem=20
>> occuring... Any ideas whats up with 2.63?
>>=20
>
>Ok, I take that back. I'm seeing "Out of memory" errors in the debug
>output with 3.0-cvs also, just not as often. Here were a bunch that
>came in in the last few minutes..
>
>[root@spamd4 spamd]# tail -50000 spamdlog | grep "Out of memory"
>2004-03-23 12:48:48.695128500 Out of memory!
>2004-03-23 12:50:50.735185500 Out of memory!
>2004-03-23 12:51:03.151317500 Out of memory!
>2004-03-23 12:51:36.312812500 Out of memory!
>2004-03-23 12:52:16.608292500 Out of memory!
>2004-03-23 12:56:32.809453500 Out of memory!
>
>My softlimit on spamd is 96mb... What could cause this kind of memory
>usage per process? =20
>
>I enabled local tests only with -L and turned off awl and bayes in the
>configs, and I still see this occur...
>
>[root@spamd4 spamd]# tail -5000 spamdlog | grep "Out of memory"
>2004-03-23 13:15:25.451648500 Out of memory!
>2004-03-23 13:16:37.426329500 Out of memory!
>
>Normal RSS values for spamd children are normally 24-25MB, how can 96MB
>not be enough when bayes and awl are not running??

Are you limiting the size of messages scanned? What happens if a
20MB message is sent through the scanner?

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFAYJBGQTcbUG5Y7woRAsbVAJ9a3Lrt8d/L0yeM0El1TL7N7XTlYgCeJw6x
ymegwQ7kEuX8Vzl85ExGAqQ=
=zPIY
-----END PGP SIGNATURE-----

RE: Spamd out of memory error on 2.63 [ In reply to ]

dallase at nmgi

Mar 23, 2004, 12:47 PM

Post #4 of 6 (136 views)

Permalink

y!
> >
> >Normal RSS values for spamd children are normally 24-25MB,
> how can 96MB
> >not be enough when bayes and awl are not running??
>
> Are you limiting the size of messages scanned? What happens if a
> 20MB message is sent through the scanner?
>

Yes, I do not override the default... So -s 256000 or whatever is
assumed. Messages that come through larger than that are not scored...
So they show 0/0.

I guess I can recordio and try and figure out what messages cause it to
occur, but its going to be hard to match the qmail-smtpd PID with the
spamd PID with as much mail as goes through here.

I guess I can also take my softlimit higher and see if memory
consumption keeps rising much above the 96MB limit. Any other ideas are
welcomed.

Does anyone else see any 80+MB RSS usage ever on spamd children?

Dallas

RE: Spamd out of memory error on 2.63 [ In reply to ]

dallase at nmgi

Mar 23, 2004, 1:08 PM

Post #5 of 6 (136 views)

Permalink

> -----Original Message-----
> From: Dallas L. Engelken [mailto:dallase@nmgi.com]
> Sent: Tuesday, March 23, 2004 1:47 PM
> To: spamassassin-dev@incubator.apache.org
> Subject: RE: Spamd out of memory error on 2.63
>
>
> y!
> > >
> > >Normal RSS values for spamd children are normally 24-25MB,
> > how can 96MB
> > >not be enough when bayes and awl are not running??
> >
> > Are you limiting the size of messages scanned? What happens if a
> > 20MB message is sent through the scanner?
> >
>
> Yes, I do not override the default... So -s 256000 or whatever is

I mean "No, I do not"... LOL

d

RE: Spamd out of memory error on 2.63 [ In reply to ]

shalligan at 333tech

Mar 23, 2004, 3:07 PM

Post #6 of 6 (136 views)

Permalink

> Does anyone else see any 80+MB RSS usage ever on spamd children?
>
> Dallas
>

Kinda. I am seeing Amavisd-new children spin out of control and get
upwards of 90MB.
If I look at the amavisd log file, they are hanging up on calling
NoMailAudit::check.

Summary of my post to the amavis list:

Here are the hung procs from top:
22219 filter 62 0 86M 13M run - 102:32 34.96% perl
6859 filter 63 0 86M 13M run - 113:15 36.43% perl

Here is the last message for each in the amavis log:

Mar 19 11:00:11 filter amavis[22219]: (22219-05) CALLING
NoMailAudit::check

Mar 19 10:46:18 filter amavis[6859]: (06859-04) CALLING
NoMailAudit::check

So, it looks like we are getting hung up calling SpamAssassin.

This is what we should see:
Mar 19 11:06:27 filter amavis[30386]: (30386-05) CALLING
NoMailAudit::check
Mar 19 11:06:28 filter amavis[30386]: (30386-05) RETURNED FROM
NoMailAudit::check, time left: 0 s

Ideas?

-steve