Mailing List Archive

On Wednesday 04 September 2019, Sergey wrote:

> Since some time there has been a noticeable increase a launch time.
> Startup with all databases takes about 220 seconds now. Startup
> without daily.cld takes 12 seconds. What's happened with daily.cld?

freshclam downloaded "daily.cvd" in next iteration:

142362624 Sep 3 13:21 daily.cld.bak
46896980 Sep 4 12:11 daily.cvd

nothing has changed, 223 seconds:

Wed Sep 4 12:17:37 2019 -> Bytecode: Security mode set to "TrustSigned".
Wed Sep 4 12:21:20 2019 -> Loaded 6304055 signatures.

--
Regards, Sergey

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Sergey wrote:

> Since some time there has been a noticeable increase a launch time.
> Startup with all databases takes about 220 seconds now. Startup
> without daily.cld takes 12 seconds. What's happened with daily.cld?

Hey,

sorry for breaking the thread, I've just subscribed and have found your
post in the archive:
https://lists.clamav.net/pipermail/clamav-users/2019-September/008496.html

So far I've found out, that the most time (about 90%) is lost in
"daily.ldb".

What I can say is, that there is no specific signature which slows it
down.
It is the algorithm... it looks like the time needed to load a new
signature depends on the number of sub-signatures (logic) and the number
of signatures already loaded in memory.
I think the Big-O-notation
(https://en.wikipedia.org/wiki/Big_O_notation) for this algorithm
wouldn't look good.

So during loading the lines (virus signatures) of ldb-file it becomes
slower and slower.
Very slow is if there are many sub-signatures for a virus signature.
There are more of this kind in the last 3rd or 4th of the ldb-file.

Maybe these signatures got added in the last months. My logfiles tell me
that the startup time of clamd was ok until February/March 2019.

greets
Markus

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Mon, Sep 09, 2019 at 13:46 PM, Markus Kolb wrote:
> Sergey wrote:
>
>> Since some time there has been a noticeable increase a launch time.
>> Startup with all databases takes about 220 seconds now. Startup
>> without daily.cld takes 12 seconds. What's happened with daily.cld?
>
> Hey,
>
> sorry for breaking the thread, I've just subscribed and have found your post in the archive:
> https://lists.clamav.net/pipermail/clamav-users/2019-September/008496.html <https://lists.clamav.net/pipermail/clamav-users/2019-September/008496.html>
>
> So far I've found out, that the most time (about 90%) is lost in "daily.ldb".
>
> What I can say is, that there is no specific signature which slows it down.
> It is the algorithm... it looks like the time needed to load a new signature depends on the number of sub-signatures (logic) and the number of signatures already loaded in memory.
> I think the Big-O-notation (https://en.wikipedia.org/wiki/Big_O_notation <https://en.wikipedia.org/wiki/Big_O_notation>) for this algorithm wouldn't look good.
>
> So during loading the lines (virus signatures) of ldb-file it becomes slower and slower.
> Very slow is if there are many sub-signatures for a virus signature. There are more of this kind in the last 3rd or 4th of the ldb-file.
>
> Maybe these signatures got added in the last months. My logfiles tell me that the startup time of clamd was ok until February/March 2019.
>
> greets
> Markus

The only thing I can add to this discussion is that in the past there have been a lot of false positives involving .ldb signatures, apparently caused by too few sub-signatures being used to positively identify a file as being malware. So that would appear to be the tradeoff here...load time vs. FPs.

-Al-
--
ClamXAV User

On Tuesday 10 September 2019, Markus Kolb wrote:

> Maybe these signatures got added in the last months. My logfiles tell
> me that the startup time of clamd was ok until February/March 2019.

Yes, I didn't notice any problems at the beginning of the year either.

Meanwhile, the time increased by another 14 seconds since 4 sep:

Fri Sep 13 13:56:47 2019 -> Bytecode: Security mode set to "TrustSigned".
Fri Sep 13 14:00:44 2019 -> Loaded 6315135 signatures.

--
Regards, Sergey

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Am 13.09.2019 12:09, schrieb Sergey:
> On Tuesday 10 September 2019, Markus Kolb wrote:
>
>> Maybe these signatures got added in the last months. My logfiles tell
>> me that the startup time of clamd was ok until February/March 2019.
>
> Yes, I didn't notice any problems at the beginning of the year either.
>
> Meanwhile, the time increased by another 14 seconds since 4 sep:
>
> Fri Sep 13 13:56:47 2019 -> Bytecode: Security mode set to
> "TrustSigned".
> Fri Sep 13 14:00:44 2019 -> Loaded 6315135 signatures.

It will increase further ;-)

I'm currently developing some file caching for the startup.
It is not so easy to implement.
The cache also needs to become invalid if there is a DB update or some
relevant config change.
Config changes could be handled by multiple file caches.
Maybe later there could be introduced some kind of patching of compiled
rules to circumvent the cache invalidation. But at the moment I don't
have a complete overview of the existing code. All the pointer linking
in data structs. OMG.

I would appreciate if someone with knowledge in developing C can jump in
to speed up the process.
There are some nice parts to split up.
If there is interest I'll create a Github repo.

greets
Markus

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

I've opened an enhacement bug for this:
https://bugzilla.clamav.net/show_bug.cgi?id=12389

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Friday 13 September 2019, Markus Kolb via clamav-users wrote:

> I've opened an enhacement bug for this:
> https://bugzilla.clamav.net/show_bug.cgi?id=12389

Thanks. But I have one more question. Do I understand correctly
that when loading main.cvd base rules are created quickly and
the problem is in their subsequent update from daily.* files?

Maybe it's time to update main.cvd and reduce daily.* while
bug 12389 is being processed?

--
Regards, Sergey

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 07/10/2019 08:57, Sergey wrote:
> On Friday 13 September 2019, Markus Kolb via clamav-users wrote:
>
>> I've opened an enhacement bug for this:
>> https://bugzilla.clamav.net/show_bug.cgi?id=12389
>
> Thanks. But I have one more question. Do I understand correctly
> that when loading main.cvd base rules are created quickly and
> the problem is in their subsequent update from daily.* files?
>
> Maybe it's time to update main.cvd and reduce daily.* while
> bug 12389 is being processed?
>

I support this idea. Daily.cvd is at the moment bigger than main.cvd and
main.cvd has not beeen updated at least two years (maybe even more).

--
Best Regards
Vladislav Kurz


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> > Maybe it's time to update main.cvd and reduce daily.* while
> > bug 12389 is being processed?
> >
>
> I support this idea. Daily.cvd is at the moment bigger than main.cvd and
> main.cvd has not beeen updated at least two years (maybe even more).

I don't know how the viruses are tracked, but maybe to reduce size (if
applicable) some of the more ancient viruses that only affect EOL
operating systems (or programs that should have long since been
patched) could be spun-off into a separate definition file (that could
be optionally disabled)? Seems like it would be quite a waste of
resources for most if there were like a million definitions that only
affected Windows XP or Office 2003 or something like that...

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 7 October 2019 15:25:41 "J.R. via clamav-users"
<clamav-users@lists.clamav.net> wrote:

> I don't know how the viruses are tracked, but maybe to reduce size (if
> applicable) some of the more ancient viruses that only affect EOL
> operating systems (or programs that should have long since been
> patched) could be spun-off into a separate definition file (that could
> be optionally disabled)? Seems like it would be quite a waste of
> resources for most if there were like a million definitions that only
> affected Windows XP or Office 2003 or something like that...

If you also take a peek at hashes:




Number of hashes:




36,49,543 main.hdb

23,657,708 daily.hdb




248,06,499 main.hsb

905,00,729 daily.hsb







file Size:




36,49,543 main.hdb

23,657,708 daily.hdb




24,806,499 main.hsb

905,00,729 daily.hsb




Example:




grep "130ae8f338cc705a26fa5fa635d8673a" daily.hsb




130ae8f338cc705a26fa5fa635d8673a:92160:Doc.Dropper.Agent-1453138:73







https://www.virustotal.com/gui/file/06f0af676b49d13c51b36e4d61f2d8751bd5ef5d5241a68e99691d68617c7415/detection




First Seen In The Wild ---> 2016-06-03 20:34:00

Last Submission ---> 2016-06-03 20:37:03

Document Name: Rotech AG_Faktur dot doc




So, is the above hash still relevant or should it moved into archived.hsb,
which by default doesn't load ?




Perhaps, daily.* are hashes up to a year old, main.* for hashes two years
old and everything else into archive.*




Or jsut drop document hashes over a year old ??




It's a difficult one to suit all uses of ClamAV I guess.
Cheers,


Steve
Twitter: @sanesecurity

Hi there,

On Mon, 7 Oct 2019, Steve Basford wrote:

> 36,49,543 main.hdb
> 23,657,708 daily.hdb
> 248,06,499 main.hsb
> 905,00,729 daily.hsb
> 36,49,543 main.hdb
> 23,657,708 daily.hdb
> 24,806,499 main.hsb
> 905,00,729 daily.hsb

Have you finally worn out your keyboard Steve? :O

> On 7 October 2019 15:25:41 "J.R. via clamav-users" wrote:
>
> I don't know how the viruses are tracked, but maybe to reduce size (if
> applicable) some of the more ancient viruses that only affect EOL
> operating systems (or programs that should have long since been
> patched) could be spun-off into a separate definition file (that could
> be optionally disabled)? Seems like it would be quite a waste of
> resources for most if there were like a million definitions that only
> affected Windows XP or Office 2003 or something like that...

Well I only run Linux systems and I'd _still_ want to scan for Windows
and Office 2003 malware. Call it social responsibility. Just because
my systems are immune to something malicious doesn't mean I'll want to
ignore it when it arrives. If my systems accepted such a thing from a
correspondent who has a vulnerable system, and then gave it to another
correspondent with yet another vulnerable system then I'd say that I'd
been irresponsible if I could have stopped it in its tracks with a bit
of effort and very little extra resource usage.

There are much more important things to be done with ClamAV, not the
least of which is improving the detection rates.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Oct 7, 2019, at 6:39 AM, Vladislav Kurz via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

On 07/10/2019 08:57, Sergey wrote:
On Friday 13 September 2019, Markus Kolb via clamav-users wrote:

I've opened an enhacement bug for this:
https://bugzilla.clamav.net/show_bug.cgi?id=12389

Thanks. But I have one more question. Do I understand correctly
that when loading main.cvd base rules are created quickly and
the problem is in their subsequent update from daily.* files?

Maybe it's time to update main.cvd and reduce daily.* while
bug 12389 is being processed?


I support this idea. Daily.cvd is at the moment bigger than main.cvd and
main.cvd has not beeen updated at least two years (maybe even more).


Thanks everyone, we are looking into this.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com

> Steve Basford:
> So, is the above hash still relevant or should it moved into archived.hsb,
> which by default doesn't load ?

I would *guess* the ClamAV team would have a *little* more detailed of
a back-end system tracking viruses (though I could be wrong)...

> G.W. Haywood:
> Well I only run Linux systems and I'd _still_ want to scan for Windows
> and Office 2003 malware. Call it social responsibility. Just because
> my systems are immune to something malicious doesn't mean I'll want to
> ignore it when it arrives. If my systems accepted such a thing from a
> correspondent who has a vulnerable system, and then gave it to another
> correspondent with yet another vulnerable system then I'd say that I'd
> been irresponsible if I could have stopped it in its tracks with a bit
> of effort and very little extra resource usage.

That's why I said "optionally disable" as in "enabled by default"...
and Office 2003 was just a random example (as it is 16 years old)...
Would you still feel necessary to scan for DOS viruses? Windows 3.1?
95? 98? 2K? It's sad that some people still today think Windows XP
should be supported (even though EXTENDED support ended in 2014), when
that OS has no business being connected to the internet with all the
out-of-date software on it.

When there's almost 1 MILLION new pieces of malware/viruses created
every DAY, there's a point of diminishing returns if the signature
database was going to contain everything since the dawn of
computing... Granted there aren't nearly that many new signatures
added to clamav, but the explosive growth in MODERN threats just goes
to show the direction things are going...

A logical approach would be to keep definitions in the "main.cvd" as
long as the product is currently supported... After it is declared EOL
and no longer supported by its creator, then move said definitions
into the (default enabled, but optionally disabled) "archived.cvd" or
whatever and give them an extended year before being removed out of
that. For the super-paranoid then maybe create a "historical.cvd" that
can hold all the old bloat and could would be default-disabled but
optionally-enabled.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Mon, 7 Oct 2019, J.R. via clamav-users wrote:
> G.W. Haywood wrote:
>> Well I only run Linux systems and I'd _still_ want to scan for Windows
>> and Office 2003 malware. Call it social responsibility. ...
>
> ... scan for DOS viruses? Windows 3.1? 95? 98? 2K?

mail6:/var/lib/clamav/databases$ grep -a DOS *ld | wc -l
214

> It's sad that some people still today think Windows XP should be
> supported (even though EXTENDED support ended in 2014), when that OS
> has no business being connected to the internet with all the
> out-of-date software on it.

Think a bit more about it. I've had clients who bought Windows 2000
servers and then had software built on top of it for their businesses
and in some cases their engineering processes. Some clients have CNC
machines driven by XP! They made a business plan, which was that the
package should pay for itself over a period of time, with a relatively
small outlay, further on down the road, to update the package when the
inevitable next Windows comes out. Then their software supplier goes
bankrupt. They've spent a couple of hundred grand on something which
will not work with the latest and greatest OS and there is absolutely
_no_ way to update it. They know the problems, but there really isn't
any option for them but to battle on. And yes, I warned them all, but
the alternatives were a lot more expensive - so at the time it really
just came down to gazing into a crystal ball.

> When there's almost 1 MILLION new pieces of malware/viruses created
> every DAY, there's a point of diminishing returns ...

I'm not sure what the numbers are, but it's very clear that it's a
numbers game, and that the Good Guys are very much in the minority.
The moral of that has to be don't play those numbers, you will lose.
I personally put a lot more store in blocking connections by country,
ASN and DNSBL for example than I do in scanning for malicious content,
but that isn't always workable for everyone.

> A logical approach would be to keep definitions in the "main.cvd" as
> long as the product is currently supported...

I think I just saw a flock of pigs flying by... :/

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Gotta keep detection around for “old” stuff. First of all, who defines old?

Second of all, when ClamAV is tested in third party analysis, we aren’t tested against “just new stuff”

Sent from my ? iPad

> On Oct 7, 2019, at 16:11, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?Hi there,
>
>> On Mon, 7 Oct 2019, J.R. via clamav-users wrote:
>> G.W. Haywood wrote:
>>> Well I only run Linux systems and I'd _still_ want to scan for Windows
>>> and Office 2003 malware. Call it social responsibility. ...
>>
>> ... scan for DOS viruses? Windows 3.1? 95? 98? 2K?
>
> mail6:/var/lib/clamav/databases$ grep -a DOS *ld | wc -l
> 214
>
>> It's sad that some people still today think Windows XP should be
>> supported (even though EXTENDED support ended in 2014), when that OS
>> has no business being connected to the internet with all the
>> out-of-date software on it.
>
> Think a bit more about it. I've had clients who bought Windows 2000
> servers and then had software built on top of it for their businesses
> and in some cases their engineering processes. Some clients have CNC
> machines driven by XP! They made a business plan, which was that the
> package should pay for itself over a period of time, with a relatively
> small outlay, further on down the road, to update the package when the
> inevitable next Windows comes out. Then their software supplier goes
> bankrupt. They've spent a couple of hundred grand on something which
> will not work with the latest and greatest OS and there is absolutely
> _no_ way to update it. They know the problems, but there really isn't
> any option for them but to battle on. And yes, I warned them all, but
> the alternatives were a lot more expensive - so at the time it really
> just came down to gazing into a crystal ball.
>
>> When there's almost 1 MILLION new pieces of malware/viruses created
>> every DAY, there's a point of diminishing returns ...
>
> I'm not sure what the numbers are, but it's very clear that it's a
> numbers game, and that the Good Guys are very much in the minority.
> The moral of that has to be don't play those numbers, you will lose.
> I personally put a lot more store in blocking connections by country,
> ASN and DNSBL for example than I do in scanning for malicious content,
> but that isn't always workable for everyone.
>
>> A logical approach would be to keep definitions in the "main.cvd" as
>> long as the product is currently supported...
>
> I think I just saw a flock of pigs flying by... :/
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

There are some occasion where malware becomes officially obsolete due to things like their C&C servers being seized by authorities. That would require a more detailed knowledge of such threats which I’m sure you don’t have the resources for. And eventually these threats could be resurrected by someone else and suddenly such legacy infections that are now ignored by ClamAV are operative again.

Sent from my iPad

-Al

> On Oct 7, 2019, at 13:16, Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Gotta keep detection around for “old” stuff. First of all, who defines old?


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Mon, 7 Oct 2019, Joel Esler (jesler) wrote:

> Gotta keep detection around for “old” stuff.
> First of all, who defines old? ...

You don't see many DOS packages nowadays, but they are still around.

Over thirty years ago, I wrote a package for product warehousing and
distribution in C and C++. Invoicing, purchasing, credit control,
stock control, reporting, all the usual day-to-day business stuff.

It's a multi-user system. It runs on MS-DOS (and FreeDOS, and yes, I
know they're not multi-user systems but one day my mother had an idea,
which, if you didn't also need multi-tasking, and admittedly after I'd
worked on it for a little while, turned out to be a rather good one.:)

Anyway, it's still in use today, and still on DOS, turning over some
millions of pounds per annum. It's never been compromised, although
it does now also run on Linux which I find a little scary. If there's
no network stack you have a lot less to worry about - but not nothing.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi,

You are forgetting things like embedded systems in hospitals that can't
reasonably be updated.

The NHS got stung by this with XP and Microsoft had to produce a post
EOL fix.

Outside of the computer industry, software and hardware move forward at
a snails pace. Many systems still use Windows 2K and DOS. Many systems
can't reasonably be updated as the company that made them no-longer
exists. The primary reason for change is that something breaks and the
equipment has to be scrapped not that the embedded software is not
supportable, 'out of date' and at risk from malicious software.

Regards
Mark.
On 07/10/19 18:38, J.R. via clamav-users wrote:
>> Steve Basford:
>> So, is the above hash still relevant or should it moved into archived.hsb,
>> which by default doesn't load ?
>
> I would *guess* the ClamAV team would have a *little* more detailed of
> a back-end system tracking viruses (though I could be wrong)...
>
>> G.W. Haywood:
>> Well I only run Linux systems and I'd _still_ want to scan for Windows
>> and Office 2003 malware. Call it social responsibility. Just because
>> my systems are immune to something malicious doesn't mean I'll want to
>> ignore it when it arrives. If my systems accepted such a thing from a
>> correspondent who has a vulnerable system, and then gave it to another
>> correspondent with yet another vulnerable system then I'd say that I'd
>> been irresponsible if I could have stopped it in its tracks with a bit
>> of effort and very little extra resource usage.
>
> That's why I said "optionally disable" as in "enabled by default"...
> and Office 2003 was just a random example (as it is 16 years old)...
> Would you still feel necessary to scan for DOS viruses? Windows 3.1?
> 95? 98? 2K? It's sad that some people still today think Windows XP
> should be supported (even though EXTENDED support ended in 2014), when
> that OS has no business being connected to the internet with all the
> out-of-date software on it.
>
> When there's almost 1 MILLION new pieces of malware/viruses created
> every DAY, there's a point of diminishing returns if the signature
> database was going to contain everything since the dawn of
> computing... Granted there aren't nearly that many new signatures
> added to clamav, but the explosive growth in MODERN threats just goes
> to show the direction things are going...
>
> A logical approach would be to keep definitions in the "main.cvd" as
> long as the product is currently supported... After it is declared EOL
> and no longer supported by its creator, then move said definitions
> into the (default enabled, but optionally disabled) "archived.cvd" or
> whatever and give them an extended year before being removed out of
> that. For the super-paranoid then maybe create a "historical.cvd" that
> can hold all the old bloat and could would be default-disabled but
> optionally-enabled.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Am 07.10.2019 08:57, schrieb Sergey:
> On Friday 13 September 2019, Markus Kolb via clamav-users wrote:
>
>> I've opened an enhacement bug for this:
>> https://bugzilla.clamav.net/show_bug.cgi?id=12389
>
> Thanks. But I have one more question. Do I understand correctly
> that when loading main.cvd base rules are created quickly and
> the problem is in their subsequent update from daily.* files?
>
> Maybe it's time to update main.cvd and reduce daily.* while
> bug 12389 is being processed?

It doesn't matter if the rules are in daily- or main-file.
The rules currently in daily are more complex and so slower.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 17/10/2019 15:40, Markus Kolb via clamav-users wrote:
> Am 07.10.2019 08:57, schrieb Sergey:
>> On Friday 13 September 2019, Markus Kolb via clamav-users wrote:
>>
>>> I've opened an enhacement bug for this:
>>> https://bugzilla.clamav.net/show_bug.cgi?id=12389
>>
>> Thanks. But I have one more question. Do I understand correctly
>> that when loading main.cvd base rules are created quickly and
>> the problem is in their subsequent update from daily.* files?
>>
>> Maybe it's time to update main.cvd and reduce daily.* while
>> bug 12389 is being processed?
>
> It doesn't matter if the rules are in daily- or main-file.
> The rules currently in daily are more complex and so slower.

Hello everybody,

just an idea, are all databases reloaded when the reload request comes?
Would it be possible to reload only those that have changed since last
reload? Then it would be beneficial to have large main.* and smaller
daily.* files, and forking and loading in background would not be needed.

So the question is - what would be easier to code?
- reloading in background thread
- reloading limited to new files

--
Best Regards
Vladislav Kurz

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Thu, 17 Oct 2019, Vladislav Kurz via clamav-users wrote:

> So the question is - what would be easier to code?
> - reloading in background thread
> - reloading limited to new files

It is not clear to me that the latter suggestion is feasible, but...

1. Reload in a separate thread was first coded about six years ago.

2. After other, more recent discussions on this list some weeks ago,
I provided a patch for the current version of clamd, using that
original code as a basis. Yet more recently, one of the ClamAV
developers provided a similar patch. Both are freely available.

3. I've been running the patched code without issue for months.

To begin with, just in case there were differences in behaviour
between the patched and unpatched versions of clamd and the new code
(v 0.102) and the older code, I ran three clamd instances in parallel
and configured a mail server (a) to permit connections from many more
suspect sources than I would normally permit to connect to the server
(so that the clamd instances get a lot more exercise than they would,
otherwise, here) and (b) to scan all our incoming mail with all three
versions of clamd concurrently. During the time of this experiment I
saw no difference between the detection performances of the three
clamd instances. After some weeks of running parallel daemons I shut
down those running the old code and now I'm running only the version
0.102rc code.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 17/10/2019 16:51, G.W. Haywood via clamav-users wrote:
> Hi there,
>
> On Thu, 17 Oct 2019, Vladislav Kurz via clamav-users wrote:
>
>> So the question is - what would be easier to code?
>> - reloading in background thread
>> - reloading limited to new files
>
> It is not clear to me that the latter suggestion is feasible, but...
>
> 1. Reload in a separate thread was first coded about six years ago.
>
> 2. After other, more recent discussions on this list some weeks ago,
>    I provided a patch for the current version of clamd, using that
>    original code as a basis.  Yet more recently, one of the ClamAV
>    developers provided a similar patch.  Both are freely available.
>
> 3. I've been running the patched code without issue for months.

Oh great. Thank you Ged for writing and testing it. I apologise for not
noticing that these patch is already done. Is there anything blocking
this patch from being accepted ?

I'm noticing clamav-reload related timeouts on more and more (mostly
older or low-end) servers, which were running just fine a year or two ago.

--
Best regards
Vladislav Kurz

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hello again,

On Thu, 17 Oct 2019, Vladislav Kurz via clamav-users wrote:

> Is there anything blocking this patch from being accepted ?

As far as I know, only the (significant) pressures and (AFAICT equally
significant) limitations on developer time at Cisco/Talos/SourceFire.

> I'm noticing clamav-reload related timeouts on more and more (mostly
> older or low-end) servers, which were running just fine a year or two ago.

There other ways of dealing with this, as I'm sure you're aware, but
using the patched daemon you only have to worry about the increased
memory consumption during databse reloads.

It seems to me that the amount of junk mail grows ever more quickly.
When not testing clamd, I routinely block for example all connections
from more than a hundred countries, a similar number of ASNs, and all
hosts which score a total of three or more in our weighted DNSBL list.
That's quite apart from the more targeted block lists. Obviously this
isn't an option for everyone, but here it makes the difference between
email being useful, and email being nothing but a nuisance.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Vladislav, Ged:

Reloading select databases is not feasible at this time, because signatures are loaded into the same structures in memory and that entire thing is recreated on reload.

Regarding the threaded reload feature ( ticket: https://bugzilla.clamav.net/show_bug.cgi?id=10979 )...

The main reason the "threaded reload" patch is held back at present is primarily because the recent work and interest in the patch came at the same time that 0.102 development was in code freeze while we tested and applied bug fixes for release. Reloading in a separate thread means that the memory usage will double (going from roughly ~750MB to ~1500MB) during the reload before it frees the original signatures and drops back to ~750MB.

We already have many complaints about freshclam and clamd memory usage, and this change in behavior could cause trouble for some users, so we want to provide an option to reload the traditional way. That's the second reason why the patch isn't been merged for 0.103 yet. We have to dedicate some time to code the ability to reload either way. It is absolutely on our to-do list.

Regarding the increase in daily.cvd signatures causing slow load times...

We identified the cause of slow load times. There are 2 loading algorithms identified that are inefficient and causing slow load times. The first has to do with signature ignore lists. This is easier to address for all users and I'm currently waiting for a larger update to daily & main that will drop load time by about 40% or so for most users. The 2nd is something harder to solve, triggered by 2 very common patterns in our signature set, used to identify functions and VBA macros. I was spending most of my time trying to figure out if we can optimize the signatures to get around the design issue, but had limited success. Alberto Wu, meanwhile, has found a way to make the loading algorithm more efficient (posted just yesterday).

His patch is really promising, though I'm still testing in a very large regression test and sorting through a large number of detection changes to identify the cause. For details, please read this ticket: https://bugzilla.clamav.net/show_bug.cgi?id=12389 I'll continue to update that ticket as testing continues and I review the results.

Best regards,
Micah

?On 10/17/19, 11:08 AM, "clamav-users on behalf of Vladislav Kurz via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

On 17/10/2019 16:51, G.W. Haywood via clamav-users wrote:
> Hi there,
>
> On Thu, 17 Oct 2019, Vladislav Kurz via clamav-users wrote:
>
>> So the question is - what would be easier to code?
>> - reloading in background thread
>> - reloading limited to new files
>
> It is not clear to me that the latter suggestion is feasible, but...
>
> 1. Reload in a separate thread was first coded about six years ago.
>
> 2. After other, more recent discussions on this list some weeks ago,
> I provided a patch for the current version of clamd, using that
> original code as a basis. Yet more recently, one of the ClamAV
> developers provided a similar patch. Both are freely available.
>
> 3. I've been running the patched code without issue for months.

Oh great. Thank you Ged for writing and testing it. I apologise for not
noticing that these patch is already done. Is there anything blocking
this patch from being accepted ?

I'm noticing clamav-reload related timeouts on more and more (mostly
older or low-end) servers, which were running just fine a year or two ago.

--
Best regards
Vladislav Kurz

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml