Mailing List Archive

On 13.05.19 18:53, Avinash Sonawane via clamav-users wrote:
>I'm using ClamAV 0.100.3/25448 on Debian Stretch. It's a default
>install without any tweaks (as far as I remember).
>
>As seen from the below top o/p, without any active scan clamd is using
>~1Gb memory. That's unacceptable. May I know how do I reduce clamd
>memory usage?

redusing the number of signatures is the only way I know of.

>PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>8164 clamav 20 0 1255316 988.1m 31296 S 0.0 25.9 0:50.44 clamd

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2634 clamav 20 0 999856 866568 12912 S 0.0 21.0 265:55.12 clamd

but I apparently have lesser traffic.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
REALITY.SYS corrupted. Press any key to reboot Universe.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Mon, 13 May 2019 15:46:42 +0200
Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:

> On 13.05.19 18:53, Avinash Sonawane via clamav-users wrote:

> redusing the number of signatures is the only way I know of.

Actually, I was thinking if I could tweak some clamd conf without
removing AV databases/signatures thereby not reducing clamAv's
functionality/effectiveness.

> but I apparently have lesser traffic.

Single email account here. On average, I receive one email a day.
Devoting 1Gb memory all the time for that seems a poor bargain.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

>On Mon, 13 May 2019 15:46:42 +0200
>Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
>> redusing the number of signatures is the only way I know of.

On 13.05.19 19:30, Avinash Sonawane via clamav-users wrote:
>Actually, I was thinking if I could tweak some clamd conf without
>removing AV databases/signatures thereby not reducing clamAv's
>functionality/effectiveness.

I'm afraid that the virus database is the only thing that uses memory.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"The box said 'Requires Windows 95 or better', so I bought a Macintosh".

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Mon, 13 May 2019 19:30:12 +0530
Avinash Sonawane <rootkea@gmail.com> wrote:

> Single email account here. On average, I receive one email a day.
> Devoting 1Gb memory all the time for that seems a poor bargain.

Why can't clamd let databases/signatures stay in secondary memory
itself. Just load them when you actually receive message (or performing
the scan explicitly asked by user). Process and then again unload.
Waiting for next message.

Why clamd needs to have signatures/databases loaded in primary memory
all the time? Even when there is no active scan or incoming email? This
doesn't make sense.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

>On Mon, 13 May 2019 19:30:12 +0530
>Avinash Sonawane <rootkea@gmail.com> wrote:
>> Single email account here. On average, I receive one email a day.
>> Devoting 1Gb memory all the time for that seems a poor bargain.

On 13.05.19 19:46, Avinash Sonawane via clamav-users wrote:
>Why can't clamd let databases/signatures stay in secondary memory
>itself. Just load them when you actually receive message (or performing
>the scan explicitly asked by user). Process and then again unload.
>Waiting for next message.

loading takes time, much time. And, they still would take about the same
memory.

>Why clamd needs to have signatures/databases loaded in primary memory
>all the time? Even when there is no active scan or incoming email? This
>doesn't make sense.

there are many signatures, they must be parsed and understood by clamav.
The only place they can be stored at scanning time is the memory.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"Where do you want to go to die?" [Microsoft]

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Mon, 13 May 2019, Avinash Sonawane via clamav-users wrote:

> On Mon, 13 May 2019 19:30:12 +0530
> Avinash Sonawane <rootkea@gmail.com> wrote:
>
> > Single email account here. On average, I receive one email a day.
> > Devoting 1Gb memory all the time for that seems a poor bargain.
>
> Why can't clamd let databases/signatures stay in secondary memory
> itself. Just load them when you actually receive message (or performing
> the scan explicitly asked by user). Process and then again unload.
> Waiting for next message.
>
> Why clamd needs to have signatures/databases loaded in primary memory
> all the time? Even when there is no active scan or incoming email? This
> doesn't make sense.

What you're asking for is clamscan (as opposed to clamd and clamdscan).
It loads the signatures when it runs, and after scanning all the memory
is released.

Alan Stern


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

>> On Mon, 13 May 2019 19:30:12 +0530
>> Avinash Sonawane <rootkea@gmail.com> wrote:
>>
>> > Single email account here. On average, I receive one email a day.
>> > Devoting 1Gb memory all the time for that seems a poor bargain.

>On Mon, 13 May 2019, Avinash Sonawane via clamav-users wrote:
>> Why can't clamd let databases/signatures stay in secondary memory
>> itself. Just load them when you actually receive message (or performing
>> the scan explicitly asked by user). Process and then again unload.
>> Waiting for next message.
>>
>> Why clamd needs to have signatures/databases loaded in primary memory
>> all the time? Even when there is no active scan or incoming email? This
>> doesn't make sense.

On 13.05.19 10:34, Alan Stern wrote:
>What you're asking for is clamscan (as opposed to clamd and clamdscan).
>It loads the signatures when it runs, and after scanning all the memory
>is released.

however, it uses about the same memory:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2634 clamav 20 0 999856 866284 12656 S 0.0 21.0 265:55.79 clamd
24906 root 20 0 967288 875404 22844 R 98.3 21.2 0:38.71 clamscan

but much longer time:

# time clamscan /tmp/hwinfo
/tmp/hwinfo: OK

----------- SCAN SUMMARY -----------
Known viruses: 9157095
Engine version: 0.100.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.57 MB
Data read: 0.29 MB (ratio 1.95:1)
Time: 39.043 sec (0 m 39 s)
38.208u 0.652s 0:39.11 99.3% 0+0k 78984+0io 13pf+0w


# time clamdscan /tmp/hwinfo
/tmp/hwinfo: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.161 sec (0 m 0 s)
0.004u 0.000s 0:00.17 0.0% 0+0k 8+0io 0pf+0w


--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Enter any 12-digit prime number to continue.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Mon, 13 May 2019, Matus UHLAR - fantomas wrote:

> >> On Mon, 13 May 2019 19:30:12 +0530
> >> Avinash Sonawane <rootkea@gmail.com> wrote:
> >>
> >> > Single email account here. On average, I receive one email a day.
> >> > Devoting 1Gb memory all the time for that seems a poor bargain.
>
> >On Mon, 13 May 2019, Avinash Sonawane via clamav-users wrote:
> >> Why can't clamd let databases/signatures stay in secondary memory
> >> itself. Just load them when you actually receive message (or performing
> >> the scan explicitly asked by user). Process and then again unload.
> >> Waiting for next message.
> >>
> >> Why clamd needs to have signatures/databases loaded in primary memory
> >> all the time? Even when there is no active scan or incoming email? This
> >> doesn't make sense.
>
> On 13.05.19 10:34, Alan Stern wrote:
> >What you're asking for is clamscan (as opposed to clamd and clamdscan).
> >It loads the signatures when it runs, and after scanning all the memory
> >is released.
>
> however, it uses about the same memory:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2634 clamav 20 0 999856 866284 12656 S 0.0 21.0 265:55.79 clamd
> 24906 root 20 0 967288 875404 22844 R 98.3 21.2 0:38.71 clamscan
>
> but much longer time:
>
> # time clamscan /tmp/hwinfo
> /tmp/hwinfo: OK
>
> ----------- SCAN SUMMARY -----------
> Known viruses: 9157095
> Engine version: 0.100.3
> Scanned directories: 0
> Scanned files: 1
> Infected files: 0
> Data scanned: 0.57 MB
> Data read: 0.29 MB (ratio 1.95:1)
> Time: 39.043 sec (0 m 39 s)
> 38.208u 0.652s 0:39.11 99.3% 0+0k 78984+0io 13pf+0w
>
>
> # time clamdscan /tmp/hwinfo
> /tmp/hwinfo: OK
>
> ----------- SCAN SUMMARY -----------
> Infected files: 0
> Time: 0.161 sec (0 m 0 s)
> 0.004u 0.000s 0:00.17 0.0% 0+0k 8+0io 0pf+0w

True, but it has the behavior that Avinash asked for: It doesn't use up
1 GB of memory when it's not busy loading or scanning. For someone who
only receives about one email per day, trading off 39 seconds execution
time for 1 GB of permanently occupied memory might be worthwhile.

Alan Stern


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Mon, 13 May 2019 16:21:15 +0200
Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:

> loading takes time, much time.

How much time are we talking about here? I suppose by 'time' we mean
loading time (load binary and signatures) + processing time (comparing
signatures).

Now, for loading time, when I start firefox within 5-6 seconds it
immediately fills up 250+ Mb memory so for 950+ Mb (clamd) loading time
shouldn't be that of an issue.

Please note that processing time will be the same doesn't matter whether
you keep clamd and signatures loaded *all the time* or load on demand.

> And, they still would take about the same memory.

Yes. The difference is hogging memory *all the time* and loading *on
demand*


> there are many signatures, they must be parsed and understood by
> clamav. The only place they can be stored at scanning time is the
> memory.

Of course, at scanning time those signs/dbs need to be in memory. At
scanning time not *all the time*. e.g. I am expecting an email at 6 PM.
I don't mind clamd taking that much of a memory *at* 6 PM and then
release it. I find it absolutely inconvenient to having to forgo ~1GB
memory since the morning. As I said, a poor bargain.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Avinash Sonawane via clamav-users wrote:
> On Mon, 13 May 2019 16:21:15 +0200
> Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:
>
>> loading takes time, much time.
>
> How much time are we talking about here? I suppose by 'time' we mean
> loading time (load binary and signatures) + processing time (comparing
> signatures).
>
> Now, for loading time, when I start firefox within 5-6 seconds it
> immediately fills up 250+ Mb memory so for 950+ Mb (clamd) loading time
> shouldn't be that of an issue.

ClamAV isn't just pushing bits from disk to RAM; it does some active
processing to convert the signatures from their plaintext format on disk
into data structures for its pattern matching engine(s) to work with.

On lightly-loaded higher-end modern hardware, it should run about 15
seconds IME to load the signatures.

On older or less capable hardware, or systems with lots of other
processing going on, it can easily hit 30s to load the signatures.

On RAM-limited VPSes, you may be hitting swap, in which case load time
may well be several minutes at least. (And scanning isn't going to be
very fast either.)

> Of course, at scanning time those signs/dbs need to be in memory. At
> scanning time not *all the time*. e.g. I am expecting an email at 6 PM.
> I don't mind clamd taking that much of a memory *at* 6 PM and then
> release it. I find it absolutely inconvenient to having to forgo ~1GB
> memory since the morning. As I said, a poor bargain.

For your use case it sounds like you could do without ClamAV entirely.

-kgd

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Mon, 13 May 2019, Avinash Sonawane wrote:

> e.g. I am expecting an email at 6 PM. I don't mind clamd taking
> that much of a memory *at* 6 PM and then release it. I find it
> absolutely inconvenient to having to forgo ~1GB memory since the
> morning. As I said, a poor bargain.

The bargain is the one that you made when you installed ClamAV. If
you now feel that it is a poor one, you can of course uninstall it at
no extra charge.

Also consider that the email that you receive at 6PM might conceivably
contain something which could completely destroy _all_ the software in
your computer system. Perhaps not such a poor bargain then, if ClamAV
manages to prevent this malicious message from doing its nasty work?

You will probably agree that your use case is unusual (even I get more
mail than you do... :). Unfortunately it is difficult to accommodate
the needs of every user within a single package. It is unlikely that
the development team will schedule big changes to ClamAV for a single
user who receives one single email per day. The same install is used
by some people on this list to scan more than one message every single
second of every single day; the design of ClamAV appears to suit those
people better than it suits you.

There is still some hope, however.

The ClamAV source code is published. If you want to contribute code
which reduces the memory consumption of clamd without making serious
compromises in performance, I'm sure that people here will be pleased
to take a look at it.

Incidentally I normally run three copies of clamd on the a single mail
server. Each copy uses 1GB RAM. On a typical day, the server sees a
few thousand to a couple of tens of thousands of attempts to send mail
to it; thankfully most of the time it's at the lower end of the range.
The last time any of them found anything was on 26 September 2018, and
speaking personally I'm more than happy with that.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 13.05.19 16:40, Matus UHLAR - fantomas wrote:
>but much longer time:
>
># time clamscan /tmp/hwinfo
>/tmp/hwinfo: OK
>
>----------- SCAN SUMMARY -----------
>Known viruses: 9157095
>Engine version: 0.100.3
>Scanned directories: 0
>Scanned files: 1
>Infected files: 0
>Data scanned: 0.57 MB
>Data read: 0.29 MB (ratio 1.95:1)
>Time: 39.043 sec (0 m 39 s)
>38.208u 0.652s 0:39.11 99.3% 0+0k 78984+0io 13pf+0w

I should add that this is Xeon X3440 @2.53GHz
so you can try to compare...

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Chernobyl was an Windows 95 beta test site.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 13/05/2019, 15:57, Avinash Sonawane wrote:
> Of course, at scanning time those signs/dbs need to be in memory. At
> scanning time not *all the time*. e.g. I am expecting an email at 6 PM.
> I don't mind clamd taking that much of a memory *at* 6 PM and then
> release it. I find it absolutely inconvenient to having to forgo ~1GB
> memory since the morning. As I said, a poor bargain.

To paraphrase Apple's mistake from a few years back, "You're holding it wrong".

The ClamAV daemon is meant to be a permanently running, on-demand scanning service that can be called by the clamdscan binary, applications that have been linked against libclamav or accessed via the well-documented socket API.

If your use case is intermittent scanning, you probably don't need to use the daemon; as others have pointed out you can just use the clamscan binary instead.

I'm using clamd across a fleet of mail servers, which between them are scanning from 100k to 1 million messages per day. They're using more than just the default signatures and run at between 2GB to 2.5GB VSZ, 1GB RSS. We couldn't afford the signature loading time if we ran clamscan for every message.

If you need the immediacy, you need to accept the memory usage (but memory is cheap, right?). If you need the memory, use the slower method. If you're running a mail server that receives one email per day... maybe do that in a different way so you use your resources for something else!

Graeme


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Tue, 14 May 2019 14:53:27 +0000
Graeme Fowler <G.E.Fowler@lboro.ac.uk> wrote:

> If you're running a mail server that receives one email per
> day... maybe do that in a different way so you use your resources for
> something else!

I'm not running a mail server. I'm using clamAV at user end scanning
incoming email via mail client clamd plugin.

But perhaps you're right. Maybe I don't need the ClamAV mail scan since
my mail provider (non-Gmail account) must be scanning mails for
viruses/malware on server.

Though I intend to use clamscan for periodic filesystem scans through
cron job.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Mon 13/May/2019 16:55:57 +0200 Avinash Sonawane via clamav-users wrote:

> Now, for loading time, when I start firefox within 5-6 seconds it
> immediately fills up 250+ Mb memory so for 950+ Mb (clamd) loading time
> shouldn't be that of an issue.

I use more or less average ~1GB too:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12427 courier 20 0 1195912 1.023g 9000 S 0.0 15.3 1186:13 avfilter

Reload timing is about this:

May 16 09:43:40 22 north courierfilter: avfilter[12234]: spawned child while parent reloads
May 16 09:44:30 22 north courierfilter: avfilter[12234]: exiting
May 16 09:44:30 22 north avfilter_sig[12233]: avfilter[12427]: virus data loaded in: 50 sec(s)
May 16 09:44:30 22 north avfilter_sig[12233]: avfilter[12427]: previous load was: 16 May 2019 (50m 32s ago)
May 16 09:44:30 22 north avfilter_sig[12233]: avfilter[12427]: No. of viruses: 9154109
May 16 09:44:30 22 north avfilter_sig[12233]: avfilter[12427]: previous count was: 9153328 (781 increment)

If you have additional databases, it may take various minutes.


Best
Ale

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml