Mailing List Archive

[clamav-users] Can’t allocate memory error
Hello. I am using clamav 103.1 on ubuntu 20.04. I am getting “can’t
allocate memory errors” on very large files ( 10GB +). I thought clamdscan
was supposed to skip files that are larger than what you set the
maxfilesize/maxscansize to.

Is there any workaround/fix for this?
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
I was not aware of any other way to avoid scanning large files. Where can I
find such solutions?

On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote:
>
> > ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate
> > memory errors” on very large files ( 10GB +). I thought clamdscan
> > was supposed to skip files that are larger than what you set the
> > maxfilesize/maxscansize to.
>
> Unfortunately this is a known issue:
>
> https://bugzilla.clamav.net/show_bug.cgi?id=12374
>
> Have you tried other ways to avoid scanning huge files?
>
> --
>
> 73,
> Ged.
> OA
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
Hello Michael

Le 02/03/2021 à 13:44, Michael Kyriacou via clamav-users a écrit :
> I was not aware of any other way to avoid scanning large files. Where
> can I find such solutions?

As an example scan all files below 50Mb :
find /your_path -type f -size -50M|parallel clamdscan -mi --fdpass
--no-summary {}


--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.60.47.09.81
E-mail : aj@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom
Signatures for ClamAV antivirus : http://ow.ly/LqfdL

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
My scanners have 16vcpus and 64 GB Ram allocated to them. (Each) I noticed
that the clamd process actually began hanging on some of the scanners. This
slowed the scanning by a lot. looking at the log, the only thing it says is
“... sleep”. After 5-10 minutes it will continue, and then pause again.

Do you know how I can troubleshoot this issue?

On Tue, Mar 2, 2021 at 9:40 AM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote:
> > On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote:
> >> On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote:
> >>
> >>> ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate
> >>> memory errors” on very large files ( 10GB +). I thought clamdscan
> >>> was supposed to skip files that are larger than what you set the
> >>> maxfilesize/maxscansize to.
> >>
> >> Unfortunately this is a known issue:
> >>
> >> https://bugzilla.clamav.net/show_bug.cgi?id=12374
> >>
> >> Have you tried other ways to avoid scanning huge files?
> >
> > I was not aware of any other way to avoid scanning large files. Where
> can I
> > find such solutions?
>
> The operating system offers ways to avoid shooting your own feet. You
> could just arrange for all the huge files to be in some corner of the
> filesystem which you don't normally scan - which begs the questions
> what are you scanning, and why? There will of course be pseudo-files
> in your system which you should _never_ scan. The 'find' utility will
> let you specify size limits. You will need to spend some quality time
> with the 'man' pages to gain familiarity with using standard utilities
> in conjunction with something like ClamAV. Using the 'man' pages is
> something of an acquired taste, which you do need to acquire if you're
> to get the most out of a Linux box. The 'man' page for clamd.conf
> contains information about usage of resources. Also there are some
> warnings, which to my mind are perhaps a little over the top, but they
> serve to remind us that the system's resources may be shared between a
> large number of processes; that these processes compete for resources;
> and that things can get ugly when there aren't enough to go around.
>
> The concept of "not scanning a file larger than X bytes" is a bit too
> simplistic when talking about scanning with something like ClamAV which
> (a) depending on the file type may use different approaches to scanning
> and (b) can extract the content from types of file (e.g. Zip, RAR, etc.)
> which can contain whole directory structures and also employ compression
> techniques, and which as a result are subject to various and sometimes
> non-obvious Denial-Of-Service type attacks. So there are numerous clamd
> configuration options which permit fine-tuning of the resource usage of
> the ClamAV tools. To make the best use of these options you'll need to
> be familiar with the your system's resources, and the constraints.
>
> How much memory does the box have? You'll probably need a gigabyte or
> so to store the signature database before you even start a scan, plus
> whatever the scanner uses when it scans something - that depends a lot
> on what it's scanning. Then if you keep the default configuration to
> permit scanning while reloading the databases, another gigabyte will
> be used (briefly) every time clamd reloads the database. Note that
> the extra memory will not be released until the completion of any scan
> which was started before the reload. I'd recommend that if you don't
> want to have to work on memory management, four gigabytes of RAM is
> about the minimum for a clamd server. The longer it takes to scan a
> file, the more likely it is that you'll try to reload the database
> during a scan, so if you're short on memory and you want to scan files
> which take a long time to scan then it's worth considering the option
> to scan data only while a database reload is not taking place.
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
I am scanning large Data sets for a company. These file systems have
hundreds of thousands of files in them. Most files are small in size, <1GB,
while a few are large, >10GB. Most files are documents, archives, and
executables. I am scanning them to detect if there are any malware.

These are virtual machines, running Ubuntu 20.04.
The cpu on the esxi host is an Intel Xeon Platinum 828 CPu @2.70GHz. I have
in total, 112 logical processors available, and 512 GB of RAM.

The message it says is the following:
————————————————
Got command FILDES(7,11) argument
RECVTH FILDES command complete
THMGR active jobs for ***********: 2
THRMGR: Contended, sleeping
————————————————
Nothin under this command, it pauses, then after a couple minutes it will
continue, repeating

On Tue, Mar 2, 2021 at 9:40 AM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote:
> > On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote:
> >> On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote:
> >>
> >>> ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate
> >>> memory errors” on very large files ( 10GB +). I thought clamdscan
> >>> was supposed to skip files that are larger than what you set the
> >>> maxfilesize/maxscansize to.
> >>
> >> Unfortunately this is a known issue:
> >>
> >> https://bugzilla.clamav.net/show_bug.cgi?id=12374
> >>
> >> Have you tried other ways to avoid scanning huge files?
> >
> > I was not aware of any other way to avoid scanning large files. Where
> can I
> > find such solutions?
>
> The operating system offers ways to avoid shooting your own feet. You
> could just arrange for all the huge files to be in some corner of the
> filesystem which you don't normally scan - which begs the questions
> what are you scanning, and why? There will of course be pseudo-files
> in your system which you should _never_ scan. The 'find' utility will
> let you specify size limits. You will need to spend some quality time
> with the 'man' pages to gain familiarity with using standard utilities
> in conjunction with something like ClamAV. Using the 'man' pages is
> something of an acquired taste, which you do need to acquire if you're
> to get the most out of a Linux box. The 'man' page for clamd.conf
> contains information about usage of resources. Also there are some
> warnings, which to my mind are perhaps a little over the top, but they
> serve to remind us that the system's resources may be shared between a
> large number of processes; that these processes compete for resources;
> and that things can get ugly when there aren't enough to go around.
>
> The concept of "not scanning a file larger than X bytes" is a bit too
> simplistic when talking about scanning with something like ClamAV which
> (a) depending on the file type may use different approaches to scanning
> and (b) can extract the content from types of file (e.g. Zip, RAR, etc.)
> which can contain whole directory structures and also employ compression
> techniques, and which as a result are subject to various and sometimes
> non-obvious Denial-Of-Service type attacks. So there are numerous clamd
> configuration options which permit fine-tuning of the resource usage of
> the ClamAV tools. To make the best use of these options you'll need to
> be familiar with the your system's resources, and the constraints.
>
> How much memory does the box have? You'll probably need a gigabyte or
> so to store the signature database before you even start a scan, plus
> whatever the scanner uses when it scans something - that depends a lot
> on what it's scanning. Then if you keep the default configuration to
> permit scanning while reloading the databases, another gigabyte will
> be used (briefly) every time clamd reloads the database. Note that
> the extra memory will not be released until the completion of any scan
> which was started before the reload. I'd recommend that if you don't
> want to have to work on memory management, four gigabytes of RAM is
> about the minimum for a clamd server. The longer it takes to scan a
> file, the more likely it is that you'll try to reload the database
> during a scan, so if you're short on memory and you want to scan files
> which take a long time to scan then it's worth considering the option
> to scan data only while a database reload is not taking place.
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
Hi Michael,

We have this bug report https://bugzilla.clamav.net/show_bug.cgi?id=12673 for 0.103.0+ which claims that files larger than 4GB are causing errors instead of obeying the MaxScanSize setting. I’ll work on it a little today to see if I can figure out what’s going wrong.

-Micah

From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of Michael Kyriacou via clamav-users
Sent: Tuesday, March 2, 2021 8:36 AM
To: ClamAV users ML <clamav-users@lists.clamav.net>
Cc: Michael Kyriacou <mkyriacou111@gmail.com>
Subject: Re: [clamav-users] Can’t allocate memory error

I am scanning large Data sets for a company. These file systems have hundreds of thousands of files in them. Most files are small in size, <1GB, while a few are large, >10GB. Most files are documents, archives, and executables. I am scanning them to detect if there are any malware.

These are virtual machines, running Ubuntu 20.04.
The cpu on the esxi host is an Intel Xeon Platinum 828 CPu @2.70GHz. I have in total, 112 logical processors available, and 512 GB of RAM.

The message it says is the following:
————————————————
Got command FILDES(7,11) argument
RECVTH FILDES command complete
THMGR active jobs for ***********: 2
THRMGR: Contended, sleeping
————————————————
Nothin under this command, it pauses, then after a couple minutes it will continue, repeating

On Tue, Mar 2, 2021 at 9:40 AM G.W. Haywood via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
Hi there,

On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote:
> On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote:
>> On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote:
>>
>>> ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate
>>> memory errors” on very large files ( 10GB +). I thought clamdscan
>>> was supposed to skip files that are larger than what you set the
>>> maxfilesize/maxscansize to.
>>
>> Unfortunately this is a known issue:
>>
>> https://bugzilla.clamav.net/show_bug.cgi?id=12374
>>
>> Have you tried other ways to avoid scanning huge files?
>
> I was not aware of any other way to avoid scanning large files. Where can I
> find such solutions?

The operating system offers ways to avoid shooting your own feet. You
could just arrange for all the huge files to be in some corner of the
filesystem which you don't normally scan - which begs the questions
what are you scanning, and why? There will of course be pseudo-files
in your system which you should _never_ scan. The 'find' utility will
let you specify size limits. You will need to spend some quality time
with the 'man' pages to gain familiarity with using standard utilities
in conjunction with something like ClamAV. Using the 'man' pages is
something of an acquired taste, which you do need to acquire if you're
to get the most out of a Linux box. The 'man' page for clamd.conf
contains information about usage of resources. Also there are some
warnings, which to my mind are perhaps a little over the top, but they
serve to remind us that the system's resources may be shared between a
large number of processes; that these processes compete for resources;
and that things can get ugly when there aren't enough to go around.

The concept of "not scanning a file larger than X bytes" is a bit too
simplistic when talking about scanning with something like ClamAV which
(a) depending on the file type may use different approaches to scanning
and (b) can extract the content from types of file (e.g. Zip, RAR, etc.)
which can contain whole directory structures and also employ compression
techniques, and which as a result are subject to various and sometimes
non-obvious Denial-Of-Service type attacks. So there are numerous clamd
configuration options which permit fine-tuning of the resource usage of
the ClamAV tools. To make the best use of these options you'll need to
be familiar with the your system's resources, and the constraints.

How much memory does the box have? You'll probably need a gigabyte or
so to store the signature database before you even start a scan, plus
whatever the scanner uses when it scans something - that depends a lot
on what it's scanning. Then if you keep the default configuration to
permit scanning while reloading the databases, another gigabyte will
be used (briefly) every time clamd reloads the database. Note that
the extra memory will not be released until the completion of any scan
which was started before the reload. I'd recommend that if you don't
want to have to work on memory management, four gigabytes of RAM is
about the minimum for a clamd server. The longer it takes to scan a
file, the more likely it is that you'll try to reload the database
during a scan, so if you're short on memory and you want to scan files
which take a long time to scan then it's worth considering the option
to scan data only while a database reload is not taking place.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
The 3 filesizes below are as follows:

70K
260M
80K

This is just one example of the many times the clamd process seems to pause
or hang.

Thu Mar 4 09:54:03 2021 ->
/imports/INPRDSS122/mnt1/Boot/et-EE/bootmgr.exe.mui: OK
Thu Mar 4 09:54:03 2021 -> Closed fd 44
Thu Mar 4 09:54:03 2021 -> Finished scanthread
Thu Mar 4 09:54:03 2021 -> THRMGR: group_finished: 0x561583a417b0, 10
Thu Mar 4 09:54:03 2021 -> THRMGR: active jobs for 0x561583a417b0: 9
Thu Mar 4 09:54:03 2021 -> THRMGR: queue (single) crossed low threshold ->
signaling
Thu Mar 4 09:54:03 2021 -> THRMGR: queue (bulk) crossed low threshold ->
signaling
Thu Mar 4 09:54:03 2021 -> THRMGR: contended, woken
Thu Mar 4 09:54:03 2021 -> Consumed entire command
Thu Mar 4 09:54:03 2021 -> got command IDSESSION (10, 13), argument:
Thu Mar 4 09:54:03 2021 -> THRMGR: new group: 0x5615837aa6e0
Thu Mar 4 09:54:03 2021 -> got command FILDES (7, 11), argument:
Thu Mar 4 09:54:03 2021 -> RECVTH: FILDES command complete
Thu Mar 4 09:54:03 2021 -> THRMGR: active jobs for 0x5615837aa6e0: 2
Thu Mar 4 09:54:03 2021 -> THRMGR: contended, sleeping
Thu Mar 4 09:55:04 2021 ->
/imports/INPRDSUPDS121_1/mnt2/UPDs/UVHD-S-1-5-21-3556941465-1377616134-1990869325-122888.vhdx:
OK
Thu Mar 4 09:55:04 2021 -> Closed fd 178
Thu Mar 4 09:55:04 2021 -> Finished scanthread
Thu Mar 4 09:55:04 2021 -> THRMGR: group_finished: 0x5615837bee00, 17
Thu Mar 4 09:55:04 2021 -> THRMGR: active jobs for 0x5615837bee00: 16
Thu Mar 4 09:55:04 2021 -> THRMGR: queue (single) crossed low threshold ->
signaling
Thu Mar 4 09:55:04 2021 -> THRMGR: queue (bulk) crossed low threshold ->
signaling
Thu Mar 4 09:55:04 2021 -> THRMGR: contended, woken
Thu Mar 4 09:55:04 2021 -> Consumed entire command
Thu Mar 4 09:55:04 2021 -> got command IDSESSION (10, 13), argument:
Thu Mar 4 09:55:04 2021 -> THRMGR: new group: 0x5615837aa760
Thu Mar 4 09:55:04 2021 -> got command FILDES (7, 11), argument:
Thu Mar 4 09:55:04 2021 -> RECVTH: FILDES command complete
Thu Mar 4 09:55:04 2021 -> THRMGR: active jobs for 0x5615837aa760: 2
Thu Mar 4 09:55:04 2021 -> THRMGR: contended, sleeping
Thu Mar 4 09:55:04 2021 ->
/imports/INPMBXFERS121/mnt1/Boot/de-DE/bootmgr.exe.mui: OK
Thu Mar 4 09:55:04 2021 -> Closed fd 46
Thu Mar 4 09:55:04 2021 -> Finished scanthread
Thu Mar 4 09:55:04 2021 -> THRMGR: group_finished: 0x5615837b87f0, 10
Thu Mar 4 09:55:04 2021 -> THRMGR: active jobs for 0x5615837b87f0: 9
Thu Mar 4 09:55:04 2021 -> THRMGR: queue (single) crossed low threshold ->
signaling
Thu Mar 4 09:55:04 2021 -> THRMGR: queue (bulk) crossed low threshold ->
signaling
Thu Mar 4 09:55:04 2021 -> THRMGR: contended, woken
Thu Mar 4 09:55:04 2021 -> Consumed entire command
Thu Mar 4 09:55:04 2021 -> got command IDSESSION (10, 13), argument:
Thu Mar 4 09:55:04 2021 -> THRMGR: new group: 0x5615837aa7e0


On Wed, Mar 3, 2021 at 4:30 AM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote:
>
> > I am scanning large Data sets for a company. These file systems have
> > hundreds of thousands of files in them. Most files are small in size,
> <1GB,
> > while a few are large, >10GB. Most files are documents, archives, and
> > executables. I am scanning them to detect if there are any malware.
> >
> > These are virtual machines, running Ubuntu 20.04.
> > The cpu on the esxi host is an Intel Xeon Platinum 828 CPu @2.70GHz. I
> have
> > in total, 112 logical processors available, and 512 GB of RAM.
> >
> > The message it says is the following:
> > ————————————————
> > Got command FILDES(7,11) argument
> > RECVTH FILDES command complete
> > THMGR active jobs for ***********: 2
> > THRMGR: Contended, sleeping
> > ————————————————
>
> Please _cut_and_paste_ the exact messages that you are seeing, not some
> approximation to it.
> It might also help if you can get your logging system(s) to prepend
> timestamps. It's usual
> to see (and I'm much more comfortable with) logs which look something like
> this:
>
> Wed Mar 3 09:13:45 2021 -> Got new connection, FD 10
> Wed Mar 3 09:13:45 2021 -> Received POLLIN|POLLHUP on fd 6
> Wed Mar 3 09:13:45 2021 -> fds_poll_recv: timeout after 30 seconds
> Wed Mar 3 09:13:45 2021 -> Received POLLIN|POLLHUP on fd 10
> Wed Mar 3 09:13:45 2021 -> got command ALLMATCHINSTREAM (17, 24),
> argument:
> Wed Mar 3 09:13:45 2021 -> Receive thread: INSTREAM:
> /EXPORTS/clamav/tmp/tcp3/clamav-dc107bae2242d939f16bcf2bf9dd409c.tmp fd 11
> Wed Mar 3 09:13:45 2021 -> Breaking command loop, mode is no longer
> MODE_COMMAND
> Wed Mar 3 09:13:45 2021 -> Moved partial command: 4086
> Wed Mar 3 09:13:45 2021 -> mode == MODE_STREAM
> Wed Mar 3 09:13:45 2021 -> Got chunksize: 8314
> Wed Mar 3 09:13:45 2021 -> Quota Remaining: 52420486
> Wed Mar 3 09:13:45 2021 -> Processed 4082 bytes of chunkdata, pos 4
> Wed Mar 3 09:13:45 2021 -> fds_poll_recv: timeout after 120 seconds
> Wed Mar 3 09:13:45 2021 -> Received POLLIN|POLLHUP on fd 10
> Wed Mar 3 09:13:45 2021 -> Moved partial command: 4104
> Wed Mar 3 09:13:45 2021 -> mode == MODE_STREAM
> Wed Mar 3 09:13:45 2021 -> Processed 4104 bytes of chunkdata, pos 0
> Wed Mar 3 09:13:45 2021 -> fds_poll_recv: timeout after 120 seconds
> Wed Mar 3 09:13:45 2021 -> Received POLLIN|POLLHUP on fd 10
> Wed Mar 3 09:13:45 2021 -> Moved partial command: 133
> Wed Mar 3 09:13:46 2021 -> mode == MODE_STREAM
> Wed Mar 3 09:13:46 2021 -> Processed 128 bytes of chunkdata, pos 0
> Wed Mar 3 09:13:46 2021 -> Got chunksize: 0
> Wed Mar 3 09:13:46 2021 -> Chunks complete
> Wed Mar 3 09:13:46 2021 -> Number of file descriptors polled: 1 fds
> Wed Mar 3 09:13:46 2021 -> fds_poll_recv: timeout after 9996 seconds
> Wed Mar 3 09:13:46 2021 -> THRMGR: queue (single) crossed low threshold
> -> signaling
> Wed Mar 3 09:13:46 2021 -> THRMGR: queue (bulk) crossed low threshold ->
> signaling
> Wed Mar 3 09:13:46 2021 -> instream(192.168.44.25@45066): OK
> Wed Mar 3 09:13:46 2021 -> Finished scanthread
> Wed Mar 3 09:13:46 2021 -> Scanthread: connection shut down (FD 10)
> Wed Mar 3 09:13:46 2021 -> THRMGR: queue (single) crossed low threshold
> -> signaling
> Wed Mar 3 09:13:46 2021 -> THRMGR: queue (bulk) crossed low threshold ->
> signaling
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
I am not using on-access scanning.
when I type man clamd, I don’t get any information on threads.

I’ve tested setting MaxThreads to 100 vs 16(the amount of vcpus the scanner
has), and I did not see a performance increase.
What is the recommend setting for MaxThreads/MaxQueue for a 16 core system?

Also another question- Is there anyway to force clamd to use more RAM to
increase performance.

On Thu, Mar 4, 2021 at 10:55 AM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Thu, 4 Mar 2021, Michael Kyriacou via clamav-users wrote:
>
> > ... This is just one example of the many times the clamd process
> > seems to pause or hang. ...
>
> That looks to me like clamd is working as designed. It has a queueing
> system which manages threads doing different kinds of work. If you
> are giving the daemon a lot of work to do, under some circumstances it
> will defer processing for some threads, based on a queuing algorithm.
>
> For more information see the 'man' page for clamd (look for 'threads',
> which should lead you to configuration options to experiment with) and
> perhaps also .../clamd/thrmgr.c in the ClamAV source.
>
> Are you using any on-access scanning?
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
Re: [clamav-users] Can’t allocate memory error [ In reply to ]
Yes I use 64 bit executables.

Just some more information:

My I/O speed caps around 500MB/s.

I’m running clamdscan across ~30 mounted file systems on each scanner. Some
of these are ntfs, others are ext4. Each filesystem has its own clamd
process run against it.


On Thu, Mar 4, 2021 at 12:37 PM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Thu, 4 Mar 2021, Michael Kyriacou via clamav-users wrote:
>
> > when I type man clamd, I don’t get any information on threads.
>
> Please accept my apologies. I meant the 'man' page for clamd.conf.
>
> > I’ve tested setting MaxThreads to 100 vs 16(the amount of vcpus the
> scanner
> > has), and I did not see a performance increase.
> > What is the recommend setting for MaxThreads/MaxQueue for a 16 core
> system?
>
> The 'man' page for clamd.conf should give you a better idea. There's
> more than one thing to think about. I don't know of any specific
> recommendations for particular systems or even classes of systems. I
> think you'll need to think about your particular scenario and perhaps
> experiment with settings. Now that we've identified the issue with
> use of special characters on your command lines you might find that
> things start to make more sense to you. Look again at the logs and
> check the process activity (for example with 'top', or better 'atop').
> You might need to look into the documentation about ClamAV's logging
> get it to help you more.
>
> > Also another question- Is there anyway to force clamd to use more RAM to
> > increase performance.
>
> If you're going to scan millions of files for millions of signatures,
> performance is always going to be an issue, and especially so if a lot
> of the files will be big archives. You might need to think about the
> approach, rather than just point the scanner at an enormous data store
> and hope for the best.
>
> You're probably seeing a number of limitations, not just memory. The
> performance of mass storage I/O, memory and CPU will all bear on the
> overall performance of your scans. The daemon will use what memory is
> available to it, but it seems to me that memory is unlikely to be the
> biggest issue in your system. You need to get a good feel for which
> resources are under pressure and which aren't. Given what you've said
> you're doing, I trust that you use exclusively 64-bit executables?
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>