Mailing List Archive

Thanks! I've modified the setting. Seems very likely that this is the
culprit.

On Wed, 2 Dec 2020 at 09:41, Andrew C Aitchison <andrew@aitchison.me.uk>
wrote:

> On Wed, 2 Dec 2020, PenguinWhispererThe via clamav-users wrote:
>
> > Hi,
> >
> > I have a webserver with 4GB of memory that also functions as a
> mailserver.
> > The mail volume is rather low (perhaps a few hundred mails/day).
> > Almost every day around the same time I get a swap usage warning and once
> > in a while clamd crashes because it has no more swap space available
> > blocking mails from being processed.
> >
> > Right now it uses about 1.3GB and all is fine. I'm using FreeBSD. I'm
> > trying to see the logic why every day around the same time clamd decides
> to
> > need so much more memory. It's not like it needs it to process the
> emails.
> >
> > I've searched and read that clamd uses a lot of memory (30% is indeed
> quite
> > a lot). But nowhere I see these kinds of numbers (multiple gigabytes).
> > Having it use 60% memory (at least 2.3GB when it crashes) is getting
> > ridiculous. Since all mails are being processed just fine when clamd uses
> > 1.3GB I don't want to just increase the memory as it might start using
> 60GB
> > at some point for no clear reason.
> >
> > I didn't modify cronjobs recently so I'm unclear on why this seems to be
> a
> > periodic thing. I had it like weeks in a row at around 15:45. Then it
> > seemed to have switched to 17:45 and had it now once at 19:45. There
> seems
> > to be this 2 hour change in it or it's something that happens every 2
> hours
> > and circumstances get "just right" later and later.
> >
> > Anyone experienced the same? Knows what's going on? Has a solution to
> this
> > (not looking for "don't run clamAV as a daemon")?
>
> https://blog.clamav.net/2020/09/clamav-01030-released.html says
>
> Major changes
>
> clamd can now reload the signature database without blocking
> scanning.
> This multi-threaded database reload improvement was made possible thanks
> to a community effort.
> Non-blocking database reloads are now the default behavior. Some
> systems that are more constrained on RAM may need to disable non-blocking
> reloads, as it will temporarily consume double the amount of memory. We
> added a new clamd config option ConcurrentDatabaseReload, which may be set
> to no.
>
> --
> Andrew C. Aitchison Kendal, UK
> andrew@aitchison.me.uk
>

On 02/12/2020 09:34, PenguinWhispererThe via clamav-users wrote:
> Hi,
>
> I have a webserver with 4GB of memory that also functions as a mailserver.
> The mail volume is rather low (perhaps a few hundred mails/day).
> Almost every day around the same time I get a swap usage warning and
> once in a while clamd crashes because it has no more swap space
> available blocking mails from being processed.
>
> Right now it uses about 1.3GB and all is fine. I'm using FreeBSD. I'm
> trying to see the logic why every day around the same time clamd
> decides to need so much more memory. It's not like it needs it to
> process the emails.
>
> I've searched and read that clamd uses a lot of memory (30% is indeed
> quite a lot). But nowhere I see these kinds of numbers (multiple
> gigabytes). Having it use 60% memory (at least 2.3GB when it crashes)
> is getting ridiculous. Since all mails are being processed just fine
> when clamd uses 1.3GB I don't want to just increase the memory as it
> might start using 60GB at some point for no clear reason.
>
> I didn't modify cronjobs recently so I'm unclear on why this seems to
> be a periodic thing. I had it like weeks in a row at around 15:45.
> Then it seemed to have switched to 17:45 and had it now once at 19:45.
> There seems to be this 2 hour change in it or it's something that
> happens every 2 hours and circumstances get "just right" later and later.
>
> Anyone experienced the same? Knows what's going on? Has a solution to
> this (not looking for "don't run clamAV as a daemon")?
>
> Thanks in advance!
I have not seen this before, but let me ask if the extra memory usage
coincide with reloading the signatures? freshclam does a default 2 hour
database check.

If so, it is because of the design decision made by the clamav team.
When a database is reloaded, it is using newly allocated memory. To
allow continues virus scanning, the old contents is only thrashed after
reloading and processing continues with the freshly loaded databases. 
This approach is expected when the databases are kept growing - now
consumes around 1.2GB memory - and server memory is expected to grow also.

My situation is similar as yours, and as such I switch from a 32-bit
system with 750 MB to 64-bit and 16GB. The switch was needed because
some emails where not scanned due to the limited memory and processing
time by clamd.

Of course, running clamd is needed to avoid the cost of reloading
clamscan with every email and the latency incurred due to accessing the
database on disk.

An alternative would be to use compressed databases. Although I am not
sure if they get decompressed when loading the databases into memory. If
not decompressed, the processor speed might be a bottleneck.

As for your 60GB fear: Unless the design changes, it will keep on using
just twice the "normal" memory size. So as always, adding some GB's to
your system memory - if possible - is better as well as the cheapest way.

--- Frans.

--
A: Yes, just like that A: Ja, net zo
Q: Oh, Just like reading a book backwards Q: Oh, net als een boek achterstevoren lezen
A: Because it upsets the natural flow of a story A: Omdat het de natuurlijke gang uit het verhaal haalt
Q: Why is top-posting annoying? Q: Waarom is Top-posting zo irritant?

Citeren PenguinWhispererThe via clamav-users <clamav-users@lists.clamav.net>:

> Hi,
>
> I have a webserver with 4GB of memory that also functions as a mailserver.
> The mail volume is rather low (perhaps a few hundred mails/day).
> Almost every day around the same time I get a swap usage warning and once
> in a while clamd crashes because it has no more swap space available
> blocking mails from being processed.
>
> Right now it uses about 1.3GB and all is fine. I'm using FreeBSD. I'm
> trying to see the logic why every day around the same time clamd decides to
> need so much more memory. It's not like it needs it to process the emails.
>
> I've searched and read that clamd uses a lot of memory (30% is indeed quite
> a lot). But nowhere I see these kinds of numbers (multiple gigabytes).
> Having it use 60% memory (at least 2.3GB when it crashes) is getting
> ridiculous. Since all mails are being processed just fine when clamd uses
> 1.3GB I don't want to just increase the memory as it might start using 60GB
> at some point for no clear reason.
>
> I didn't modify cronjobs recently so I'm unclear on why this seems to be a
> periodic thing. I had it like weeks in a row at around 15:45. Then it
> seemed to have switched to 17:45 and had it now once at 19:45. There seems
> to be this 2 hour change in it or it's something that happens every 2 hours
> and circumstances get "just right" later and later.
>
> Anyone experienced the same? Knows what's going on? Has a solution to this
> (not looking for "don't run clamAV as a daemon")?
>
> Thanks in advance!

This is most likely due to loading the new signature database after
downloading. Previously, clamd would drop the existing database before
loading the new one. But this changed in version 0.103.0 as this to
load the new one first before dropping the old one. See the following
from /etc/clamd.conf which explains why and what you can do to change
back to the previous behaviour:

# Enable non-blocking (multi-threaded/concurrent) database reloads.
# This feature will temporarily load a second scanning engine while scanning
# continues using the first engine. Once loaded, the new engine takes over.
# The old engine is removed as soon as all scans using the old engine have
# completed.
# This feature requires more RAM, so this option is provided in case users are
# willing to block scans during reload in exchange for lower RAM requirements.
# Default: yes
#ConcurrentDatabaseReload no




_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

That 2 hour you mention seems to explain why it's always around the same
time and seems to come earlier or later by 2 hours.
I do not know if it literally coincides with that refresh (not sure if
that's logged on my system).

Having the database reloaded in chunks looks a lot more efficient to me. Ok
7xxMB memory is quite a small machine. But e.g. on a 16GB system the
standard memory usage is still almost 10% and so when reloading about 20%
(very rough numbers I admit). Of course I don't know how the database is
implemented so perhaps it's easier said than done.
In that regards "just twice" still is a lot ;)


On Wed, 2 Dec 2020 at 11:00, Frans de Boer <frans@fransdb.nl> wrote:

> On 02/12/2020 09:34, PenguinWhispererThe via clamav-users wrote:
>
> Hi,
>
> I have a webserver with 4GB of memory that also functions as a mailserver.
> The mail volume is rather low (perhaps a few hundred mails/day).
> Almost every day around the same time I get a swap usage warning and once
> in a while clamd crashes because it has no more swap space available
> blocking mails from being processed.
>
> Right now it uses about 1.3GB and all is fine. I'm using FreeBSD. I'm
> trying to see the logic why every day around the same time clamd decides to
> need so much more memory. It's not like it needs it to process the emails.
>
> I've searched and read that clamd uses a lot of memory (30% is indeed
> quite a lot). But nowhere I see these kinds of numbers (multiple
> gigabytes). Having it use 60% memory (at least 2.3GB when it crashes) is
> getting ridiculous. Since all mails are being processed just fine when
> clamd uses 1.3GB I don't want to just increase the memory as it might start
> using 60GB at some point for no clear reason.
>
> I didn't modify cronjobs recently so I'm unclear on why this seems to be a
> periodic thing. I had it like weeks in a row at around 15:45. Then it
> seemed to have switched to 17:45 and had it now once at 19:45. There seems
> to be this 2 hour change in it or it's something that happens every 2 hours
> and circumstances get "just right" later and later.
>
> Anyone experienced the same? Knows what's going on? Has a solution to this
> (not looking for "don't run clamAV as a daemon")?
>
> Thanks in advance!
>
> I have not seen this before, but let me ask if the extra memory usage
> coincide with reloading the signatures? freshclam does a default 2 hour
> database check.
>
> If so, it is because of the design decision made by the clamav team. When
> a database is reloaded, it is using newly allocated memory. To allow
> continues virus scanning, the old contents is only thrashed after reloading
> and processing continues with the freshly loaded databases. This approach
> is expected when the databases are kept growing - now consumes around 1.2GB
> memory - and server memory is expected to grow also.
>
> My situation is similar as yours, and as such I switch from a 32-bit
> system with 750 MB to 64-bit and 16GB. The switch was needed because some
> emails where not scanned due to the limited memory and processing time by
> clamd.
>
> Of course, running clamd is needed to avoid the cost of reloading clamscan
> with every email and the latency incurred due to accessing the database on
> disk.
>
> An alternative would be to use compressed databases. Although I am not
> sure if they get decompressed when loading the databases into memory. If
> not decompressed, the processor speed might be a bottleneck.
>
> As for your 60GB fear: Unless the design changes, it will keep on using
> just twice the "normal" memory size. So as always, adding some GB's to your
> system memory - if possible - is better as well as the cheapest way.
>
> --- Frans.
>
> --
> A: Yes, just like that A: Ja, net zo
> Q: Oh, Just like reading a book backwards Q: Oh, net als een boek achterstevoren lezen
> A: Because it upsets the natural flow of a story A: Omdat het de natuurlijke gang uit het verhaal haalt
> Q: Why is top-posting annoying? Q: Waarom is Top-posting zo irritant?
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

The signature patterns are loaded into a pair of large AC tries. Patterns can be added but not removed, so they have to be swapped out all at once. It is conceivable that we could do 1 trie at a time, which would split the reload process into 2 chunks, but no time/budget to work on that and improvement would be minimal.

We are making an effort to reduce the signature count be filtering out sigs that haven’t been effective. I’m hopeful you should start to see the database size slowly decrease. We will consider providing these archived signatures in an archived-signature database for folks who wish to run with as much coverage as possible.

-Micah

From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of PenguinWhispererThe via clamav-users
Sent: Wednesday, December 2, 2020 9:08 AM
To: ClamAV users ML <clamav-users@lists.clamav.net>
Cc: PenguinWhispererThe <th3penguinwhisperer@gmail.com>
Subject: Re: [clamav-users] Memory usage going up until OOM

That 2 hour you mention seems to explain why it's always around the same time and seems to come earlier or later by 2 hours.
I do not know if it literally coincides with that refresh (not sure if that's logged on my system).

Having the database reloaded in chunks looks a lot more efficient to me. Ok 7xxMB memory is quite a small machine. But e.g. on a 16GB system the standard memory usage is still almost 10% and so when reloading about 20% (very rough numbers I admit). Of course I don't know how the database is implemented so perhaps it's easier said than done.
In that regards "just twice" still is a lot ;)


On Wed, 2 Dec 2020 at 11:00, Frans de Boer <frans@fransdb.nl<mailto:frans@fransdb.nl>> wrote:
On 02/12/2020 09:34, PenguinWhispererThe via clamav-users wrote:
Hi,

I have a webserver with 4GB of memory that also functions as a mailserver.
The mail volume is rather low (perhaps a few hundred mails/day).
Almost every day around the same time I get a swap usage warning and once in a while clamd crashes because it has no more swap space available blocking mails from being processed.

Right now it uses about 1.3GB and all is fine. I'm using FreeBSD. I'm trying to see the logic why every day around the same time clamd decides to need so much more memory. It's not like it needs it to process the emails.

I've searched and read that clamd uses a lot of memory (30% is indeed quite a lot). But nowhere I see these kinds of numbers (multiple gigabytes). Having it use 60% memory (at least 2.3GB when it crashes) is getting ridiculous. Since all mails are being processed just fine when clamd uses 1.3GB I don't want to just increase the memory as it might start using 60GB at some point for no clear reason.

I didn't modify cronjobs recently so I'm unclear on why this seems to be a periodic thing. I had it like weeks in a row at around 15:45. Then it seemed to have switched to 17:45 and had it now once at 19:45. There seems to be this 2 hour change in it or it's something that happens every 2 hours and circumstances get "just right" later and later.

Anyone experienced the same? Knows what's going on? Has a solution to this (not looking for "don't run clamAV as a daemon")?

Thanks in advance!
I have not seen this before, but let me ask if the extra memory usage coincide with reloading the signatures? freshclam does a default 2 hour database check.

If so, it is because of the design decision made by the clamav team. When a database is reloaded, it is using newly allocated memory. To allow continues virus scanning, the old contents is only thrashed after reloading and processing continues with the freshly loaded databases. This approach is expected when the databases are kept growing - now consumes around 1.2GB memory - and server memory is expected to grow also.

My situation is similar as yours, and as such I switch from a 32-bit system with 750 MB to 64-bit and 16GB. The switch was needed because some emails where not scanned due to the limited memory and processing time by clamd.

Of course, running clamd is needed to avoid the cost of reloading clamscan with every email and the latency incurred due to accessing the database on disk.

An alternative would be to use compressed databases. Although I am not sure if they get decompressed when loading the databases into memory. If not decompressed, the processor speed might be a bottleneck.

As for your 60GB fear: Unless the design changes, it will keep on using just twice the "normal" memory size. So as always, adding some GB's to your system memory - if possible - is better as well as the cheapest way.

--- Frans.

--

A: Yes, just like that A: Ja, net zo

Q: Oh, Just like reading a book backwards Q: Oh, net als een boek achterstevoren lezen

A: Because it upsets the natural flow of a story A: Omdat het de natuurlijke gang uit het verhaal haalt

Q: Why is top-posting annoying? Q: Waarom is Top-posting zo irritant?

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml