Mailing List Archive: [clamav-users] clamd RAM issue?

[clamav-users] clamd RAM issue?

Oct 31, 2021, 10:05 AM

Post #1 of 6 (589 views)

Hi,

I'm running sendmail+mimedefang+clamav on a bunch of MX servers.

This morning over a period of several hours each of my instances
appear to have caused clamd to consume all RAM and swap. Normally
swap is empty and 10GB of the 16GB per host is free. This happened
immediately following db updates, but hours apart, and all the
systems have matching db updates centrally distributed here, so I
suspect some e-mail message payload was the commonality.

Has anyone else had similar experiences recently?

All the clamd Limits settings are as default.

CentOS Linux release 7.9.2009 (Core)
clamav-0.104.0 (current stable release)

Oct 31 09:01:39 imx1 clamd: Database correctly reloaded (12623346 signatures)
Oct 31 09:01:39 imx1 clamd: Activating the newly loaded database...
Oct 31 09:02:51 imx1 kernel: mimedefang.pl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Oct 31 09:02:51 imx1 kernel: mimedefang.pl cpuset=/ mems_allowed=0
Oct 31 09:02:51 imx1 kernel: CPU: 2 PID: 30341 Comm: mimedefang.pl Kdump: loaded Not tainted 3.10.0-1160.42.2.el7.x86_64 #1
Oct 31 09:02:51 imx1 kernel: Hardware name: ...
Oct 31 09:02:51 imx1 kernel: Call Trace:
Oct 31 09:02:51 imx1 kernel: [<ffffffffb9583539>] dump_stack+0x19/0x1b
Oct 31 09:02:51 imx1 kernel: [<ffffffffb957e5d8>] dump_header+0x90/0x229
Oct 31 09:02:51 imx1 kernel: [<ffffffffb8f06992>] ? ktime_get_ts64+0x52/0xf0
....
Oct 31 09:02:51 imx1 kernel: Out of memory: Kill process 5336 (clamd) score 92 or sacrifice child
Oct 31 09:02:51 imx1 kernel: Killed process 5336 (clamd), UID 8, total-vm:3399696kB, anon-rss:1774440kB, file-rss:0kB, shmem-rss:0kB
Oct 31 09:02:51 imx1 systemd: clamav-daemon.service: main process exited, code=killed, status=9/KILL
Oct 31 09:02:51 imx1 systemd: Unit clamav-daemon.service entered failed state.
Oct 31 09:02:51 imx1 systemd: clamav-daemon.service failed.

--
Mark G. Thomas <Mark@Misty.com>, KC3DRE

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] clamd RAM issue? [ In reply to ]

clamav-users at lists

Oct 31, 2021, 5:32 PM

Post #2 of 6 (588 views)

Permalink

On Sun, 2021-10-31 at 13:05 -0400, Mark G Thomas wrote:
>
> Has anyone else had similar experiences recently?
>

Not recently per se, but it happens. Do you limit the number of scans
that can be run simultaneously, if (for example) some doofus BCCs a
20MB nested zip file to everyone in his organization?

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] clamd RAM issue? [ In reply to ]

clamav-users at lists

Oct 31, 2021, 5:49 PM

Post #3 of 6 (588 views)

Permalink

Hi there,

On Sun, 31 Oct 2021, Mark G Thomas wrote:

> I'm running sendmail+mimedefang+clamav on a bunch of MX servers.
> This morning over a period of several hours each of my instances
> appear to have caused clamd to consume all RAM and swap. Normally
> swap is empty and 10GB of the 16GB per host is free. This happened
> immediately following db updates, but hours apart, and all the
> systems have matching db updates centrally distributed here ...

We find it less trouble to run a single clamd server which the mail
servers use via the network. We also run Sendmail, but we don't use
MIMEDefang any more - the MTAs talk to the clamd server via a milter.
The server is more or less dedicated to clamd, and if it does go OOM
there's less colateral damage. It generally uses around 2G of RAM and
almost no swap. It's been that way for at least a year (the RAM, swap
and a bunch of other stuff are graphed using Nagios which sends email
alerts if things get dicey). I really recommend it. In a case like
this you'd probably be able to see to within a few minutes the times
when the memory usage started to climb and that might help to identify
the culprit.

> I suspect some e-mail message payload was the commonality.

Seems plausible. Do you have any idea what that might have been?

> Has anyone else had similar experiences recently?

Nothing to report here I'm afraid, but I'll be very interested if you
can provide a sample message which demonstrates the issue. Which was
the update you mentioned? We saw daily bumped to 26338 yesterday at
about 15:06 BST and to 26339 at about 14:07 GMT today. Of course it's
just a coincidence that the clocks changed this morning. Or is it?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] clamd RAM issue? [ In reply to ]

Mark at Misty

Nov 1, 2021, 5:50 AM

Post #4 of 6 (587 views)

Permalink

Hi,

On Sun, Oct 31, 2021 at 08:32:00PM -0400, Michael Orlitzky via clamav-users wrote:
> On Sun, 2021-10-31 at 13:05 -0400, Mark G Thomas wrote:
> >
> > Has anyone else had similar experiences recently?
>
> Not recently per se, but it happens. Do you limit the number of scans
> that can be run simultaneously, if (for example) some doofus BCCs a
> 20MB nested zip file to everyone in his organization?

It is sometimes hard to tell from the logs after the fact what the culprit
is. In this case, it turned out we were getting flooded by small messages
from a single source. Enforcing ClientRate and ClientConn limits via
sendmail is doing the trick. I mistakenly thought we were already doing that.
I do not believe clamd was doing anything bad.

Mark

--
Mark G. Thomas <Mark@Misty.com>, KC3DRE

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] clamd RAM issue? [ In reply to ]

clamav-users at lists

Nov 1, 2021, 6:32 AM

Post #5 of 6 (587 views)

Permalink

On Sun, 31 Oct 2021, Mark G Thomas wrote:

> Date: Sun, 31 Oct 2021 13:05:35 -0400
> From: Mark G Thomas <Mark@Misty.com>
>
> I'm running sendmail+mimedefang+clamav on a bunch of MX servers.
>
> This morning over a period of several hours each of my instances
> appear to have caused clamd to consume all RAM and swap. Normally
> swap is empty and 10GB of the 16GB per host is free. This happened
> immediately following db updates, but hours apart, and all the
> systems have matching db updates centrally distributed here, so I
> suspect some e-mail message payload was the commonality.
>
> Has anyone else had similar experiences recently?
>
> All the clamd Limits settings are as default.
>
> CentOS Linux release 7.9.2009 (Core)
> clamav-0.104.0 (current stable release)
>
> Oct 31 09:01:39 imx1 clamd: Database correctly reloaded (12623346 signatures)
> Oct 31 09:01:39 imx1 clamd: Activating the newly loaded database...
> Oct 31 09:02:51 imx1 kernel: mimedefang.pl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
> Oct 31 09:02:51 imx1 kernel: mimedefang.pl cpuset=/ mems_allowed=0
> Oct 31 09:02:51 imx1 kernel: CPU: 2 PID: 30341 Comm: mimedefang.pl Kdump: loaded Not tainted 3.10.0-1160.42.2.el7.x86_64 #1
> Oct 31 09:02:51 imx1 kernel: Hardware name: ...
> Oct 31 09:02:51 imx1 kernel: Call Trace:
> Oct 31 09:02:51 imx1 kernel: [<ffffffffb9583539>] dump_stack+0x19/0x1b
> Oct 31 09:02:51 imx1 kernel: [<ffffffffb957e5d8>] dump_header+0x90/0x229
> Oct 31 09:02:51 imx1 kernel: [<ffffffffb8f06992>] ? ktime_get_ts64+0x52/0xf0
> ....
> Oct 31 09:02:51 imx1 kernel: Out of memory: Kill process 5336 (clamd) score 92 or sacrifice child
> Oct 31 09:02:51 imx1 kernel: Killed process 5336 (clamd), UID 8, total-vm:3399696kB, anon-rss:1774440kB, file-rss:0kB, shmem-rss:0kB
> Oct 31 09:02:51 imx1 systemd: clamav-daemon.service: main process exited, code=killed, status=9/KILL
> Oct 31 09:02:51 imx1 systemd: Unit clamav-daemon.service entered failed state.
> Oct 31 09:02:51 imx1 systemd: clamav-daemon.service failed.
>
The trouble starts with a tool called mimedefang.pl :

https://mimedefang.org :

"What is MIMEDefang?
MIMEDefang is an e-mail filtering tool that works with the Sendmail
"Milter" library. MIMEDefang lets you express your filtering policies
in Perl rather than C, making it quick and easy to filter or manipulate
your mail. MIMEDefang is mature software: The first version was
released in 2000. It's also in use in thousands of installations. It
remains under active development. MIMEDefang is free software: It's
released under the terms of the GNU General Public License. It runs
under Linux, FreeBSD, Solaris and most other UNIX or UNIX-like systems."

One of the reasons i selected clamd for my email system was that it was
written in 100% C. If clamd in your email system invokes a perl based
tool, anything can happen. Perl is a perfect tool for administering
complicated tasks, but when you allow it to get invoked for a unknown
number of times as part of a heavy duty service, the end result is
unclear.

--
Robert M. Stockmann - RHCE
Network Engineer - UNIX/Linux Specialist
crashrecovery.org stock@stokkie.net

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] clamd RAM issue? [ In reply to ]

clamav-users at lists

Nov 1, 2021, 9:45 AM

Post #6 of 6 (587 views)

Permalink

Hi there,

On Mon, 1 Nov 2021, Robert M. Stockmann via clamav-users wrote:

> ... If ... system invokes a perl based tool, anything can happen. ...

Er, if you let it. :)

I run a couple of Perl milters which do the heavy lifting for our mail
filtering. One of them _is_ pretty heavy but I only allow five copies
of it to run at any one time; the other is much lighter and at present
the limit is set to 63 concurrent copies. That's within the confines
of any one mail server, each of which has 750 Mbytes RAM and runs on
less than ten watts. The mail servers use a separate (single) clamd
server for all the scanning that they need to do (the ClamAV official
signatures, plus a truckload of third party signatures and a few dozen
custom Yara rules). My milters shoulder most of the filtering burden,
and from what's left after that the vast majority of scanner hits is
produced by the Yara rules. All of this is well within the processing
capacity of the typical modern laptop, by which I mean that IMHO mail
processing doesn't *have* to use a Core i9 and 16 gigabytes of RAM.

After all, we were doing it in the 1970s. Or at least I was. :(

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml