Mailing List Archive

The reload bug has been known for years, even has a ready patch.

https://bugzilla.clamav.net/show_bug.cgi?id=10979

But nothing you can do about it, ClamAV devs have a mind of their own.

Atleast servers in your scenario will (hopefully) retry sending.


On Sat, Aug 31, 2019 at 04:25:05PM +0200, Thomas Barth via clamav-users wrote:
> Hallo Mailinglist,
>
> sometimes I get in Postfix the error messages "451 4.3.0 Error: queue file
> write error". There is a warning timeout talking to localhost:10024 (Amavis)
>
>
> Aug 31 14:14:19 mx2 postfix/smtpd[15861]: connect from
> unknown[177.37.96.254]
> Aug 31 14:14:20 mx2 postfix/smtpd[15861]: NOQUEUE:
> client=unknown[177.37.96.254]
> Aug 31 14:16:02 mx2 postfix/smtpd[15861]: warning: timeout talking to proxy
> localhost:10024
> Aug 31 14:16:02 mx2 postfix/smtpd[15861]: proxy-reject: END-OF-MESSAGE: 451
> 4.3.0 Error: queue file write error; from=<BillyThompson@lookandwellness.it>
> to=<yr@domain.de> proto=ESMTP helo=<lookandwellness.it>
> Aug 31 14:16:02 mx2 postfix/smtpd[15861]: disconnect from
> unknown[177.37.96.254] ehlo=1 mail=1 rcpt=1 data=0/1 commands=3/4
>
> (Not hiding the from address, it s used by a spammer :))
>
> Normally postfix gets a response after 3 secondes.
>
> In the clamav.log I see at the same time, that reloading the database takes
> up to two minutes.
>
> /var/log/clamav/clamav.log
> Sat Aug 31 14:14:15 2019 -> Database correctly reloaded (10971844
> signatures)
> Sat Aug 31 14:14:15 2019 -> Reading databases from /var/lib/clamav
> Sat Aug 31 14:14:15 2019 ->
> /var/lib/amavis/tmp/amavis-20190831T125532-12347-lWbaS7Ci/parts/p001:
> Sanesecurity.Scam.12584.UNOFFICIAL(00000000000000000000000000000000:6617)
> FOUND
> Sat Aug 31 14:16:13 2019 -> Database correctly reloaded (10971844
> signatures)
> Sat Aug 31 14:16:13 2019 ->
> /var/lib/amavis/tmp/amavis-20190831T120830-10930-zSEWR54L/parts/p001:
> Sanesecurity.Scam.12559.UNOFFICIAL(00000000000000000000000000000000:6449)
> FOUND
>
> Is reloading a database blocking the e-Mail scanning? So how can I boost
> this process? It's a virtual server with 100% ssd and 6 cores (Intel(R)
> Xeon(R) CPU E5-2630 v4 @ 2.20GHz) and Debian Buster.
>
>
> Best regards,
> Thomas Barth
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Sat, 31 Aug 2019, Henrik K wrote:

> The reload bug has been known for years, even has a ready patch.
>
> https://bugzilla.clamav.net/show_bug.cgi?id=10979
>
> But nothing you can do about it...

Well not quite nothing, since you can download the source, apply the
patch, and rebuild ClamAV.

At the moment I'm scanning mail with two copies of clamd, one patched
and one plain vanilla. Despite some concerns about the reliability in
#10979, which is why I'm running an unpatched copy as well, the patched
version seems to be holding up - at least at fairly low mail volumes.

This is with my own Perl milter, see my recent post on the dev list
describing it. If anyone wants to try it they're more than welcome.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Am 2019-08-31 16:32, schrieb Henrik K:
> The reload bug has been known for years, even has a ready patch.

Wow, this is a masterpiece ignoring a problem for years :) Thanks for
pointing to the bugthread.

> But nothing you can do about it, ClamAV devs have a mind of their own.


Micah Snyder 2019-08-22 14:38:59 EDT
"We are not actively working on this, though it is on our list."


Realy bad attitude of developers! Fixing errors and problems must always
have highest priority before developing new things!






>
> Atleast servers in your scenario will (hopefully) retry sending.

Not the spammer.

>
>
> On Sat, Aug 31, 2019 at 04:25:05PM +0200, Thomas Barth via clamav-users
> wrote:
>> Hallo Mailinglist,
>>
>> sometimes I get in Postfix the error messages "451 4.3.0 Error: queue
>> file
>> write error". There is a warning timeout talking to localhost:10024
>> (Amavis)
>>
>>
>> Aug 31 14:14:19 mx2 postfix/smtpd[15861]: connect from
>> unknown[177.37.96.254]
>> Aug 31 14:14:20 mx2 postfix/smtpd[15861]: NOQUEUE:
>> client=unknown[177.37.96.254]
>> Aug 31 14:16:02 mx2 postfix/smtpd[15861]: warning: timeout talking to
>> proxy
>> localhost:10024
>> Aug 31 14:16:02 mx2 postfix/smtpd[15861]: proxy-reject:
>> END-OF-MESSAGE: 451
>> 4.3.0 Error: queue file write error;
>> from=<BillyThompson@lookandwellness.it>
>> to=<yr@domain.de> proto=ESMTP helo=<lookandwellness.it>
>> Aug 31 14:16:02 mx2 postfix/smtpd[15861]: disconnect from
>> unknown[177.37.96.254] ehlo=1 mail=1 rcpt=1 data=0/1 commands=3/4
>>
>> (Not hiding the from address, it s used by a spammer :))
>>
>> Normally postfix gets a response after 3 secondes.
>>
>> In the clamav.log I see at the same time, that reloading the database
>> takes
>> up to two minutes.
>>
>> /var/log/clamav/clamav.log
>> Sat Aug 31 14:14:15 2019 -> Database correctly reloaded (10971844
>> signatures)
>> Sat Aug 31 14:14:15 2019 -> Reading databases from /var/lib/clamav
>> Sat Aug 31 14:14:15 2019 ->
>> /var/lib/amavis/tmp/amavis-20190831T125532-12347-lWbaS7Ci/parts/p001:
>> Sanesecurity.Scam.12584.UNOFFICIAL(00000000000000000000000000000000:6617)
>> FOUND
>> Sat Aug 31 14:16:13 2019 -> Database correctly reloaded (10971844
>> signatures)
>> Sat Aug 31 14:16:13 2019 ->
>> /var/lib/amavis/tmp/amavis-20190831T120830-10930-zSEWR54L/parts/p001:
>> Sanesecurity.Scam.12559.UNOFFICIAL(00000000000000000000000000000000:6449)
>> FOUND
>>
>> Is reloading a database blocking the e-Mail scanning? So how can I
>> boost
>> this process? It's a virtual server with 100% ssd and 6 cores
>> (Intel(R)
>> Xeon(R) CPU E5-2630 v4 @ 2.20GHz) and Debian Buster.
>>
>>
>> Best regards,
>> Thomas Barth
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Sat, Aug 31, 2019 at 03:55:30PM +0100, G.W. Haywood via clamav-users wrote:
>
> Well not quite nothing, since you can download the source, apply the
> patch, and rebuild ClamAV.

Sure but it's not reality for majority of users..

While it's good that people try it out, I doubt if would take long for a dev
to verify the patch carefully and implement boolean for it's use. But I
guess new features pay more than having a robust engine.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 8/31/19 11:00 AM, Thomas Barth via clamav-users wrote:
>
> Realy bad attitude of developers!

Micah took the time to answer a question and provide a status update.
It's counterproductive to shame people for being honest.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Sat, Aug 31, 2019 at 11:18:00AM -0400, Michael Orlitzky via clamav-users wrote:
>
> Micah took the time to answer a question and provide a status update.
> It's counterproductive to shame people for being honest.

It's perfectly fine to shame a corporation for doing seemingly strange
things. Micah etc are paid developers and not volunteers maintaining some
stale Open Source thingy. Well atleast I hope they are not..

An existing patch has existed for 5 years, so I'm pretty interested in
hearing why such a basic and important feature is still not implemented.
Only thing that comes to mind is that the developers don't even actually use
ClamAV personally, or the use is so marginal that they don't even encounter
this problem.

If I encountered a bug like that on some project that I'm maintaining, I
would be shamed not to rapidly fix it. But perhaps it's the organization to
blame.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Sat, 31 Aug 2019, Henrik K wrote:
> On Sat, Aug 31, 2019, G.W. Haywood via clamav-users wrote:
>>
>> Well not quite nothing, since you can download the source, apply the
>> patch, and rebuild ClamAV.
>
> Sure but it's not reality for majority of users..
>
> While it's good that people try it out, I doubt if would take long for a dev
> to verify the patch carefully and implement boolean for it's use. But I
> guess new features pay more than having a robust engine.

It's not quite as simple as that. This software has to run reliably
on millions of systems with thousands of combinations and permutations
of configurations. It's doing that right now. There've occasionally
been examples of a change made perhaps a little too hastily which gave
grief to many users and rise to a lot of spleen-venting on the users'
mailing list. It would be a brave decision, in the face of the valid
concerns noted in #10979, to release a new version, world-wide, for
production use, which contains the patch that I'm running now merely
as an experiment with my eyes wide open on a server that crashed four
times this month because I'm also working on some netfilter stuff.

This is a community effort. If you're familiar with C it isn't at all
difficult to apply the patch, and I'd be happy to mail the two patched
files (56kBytes in total) to anyone who didn't feel up to applying the
patches themselves. Then, if you felt brave enough, it would _almost_
be as simple as

./configure && make && sudo make install

to build and install it.

Incidentally I'm a Sendmail dinosaur, and the default timeouts appear
to be longer for Sendmail than they are for Postfix. I'm sure it's
easy to make them longer for Postfix; then this issue would, if not
disappear, at least more or less be transparent. It really isn't that
big a deal if you know what you're doing.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Sat, 31 Aug 2019, Henrik K wrote:
> ...
> If I encountered a bug like that on some project that I'm maintaining, I
> would be shamed not to rapidly fix it.

If you called it a limitation I could agree, but I guess it's working
as designed. I'd call it an issue rather than a fault in the software.

If there _are_ bugs in this issue they're in the patch for it, which
may be why, AFAICT, I'm one of only about three people on the planet
who are actually running it.

More testing, by people prepared to chip in some effort instead of
complaining about something that they get for free, would be great.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote:
>
> More testing, by people prepared to chip in some effort instead of
> complaining about something that they get for free, would be great.

The final responsibility of implementing and testing the issue is still that
of the ClamAV team.

You are really making this much more complex and "scary" issue than it is.
New features and major versions have been constantly released these past
years. Just because someone in the bug had a random issue with patch that
wasn't even analyzed by devs, doesn't mean it will "break millions of
systems" - especially if it isn't enabled by default (which is wise, since
it would need more memory). It's simply a matter of willing to check and
implement it.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Saturday, August 31, 2019 12:04:36 PM EDT Henrik K wrote:
> On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users
wrote:
> > More testing, by people prepared to chip in some effort instead of
> > complaining about something that they get for free, would be great.
>
> The final responsibility of implementing and testing the issue is still that
> of the ClamAV team.
>
> You are really making this much more complex and "scary" issue than it is.
> New features and major versions have been constantly released these past
> years. Just because someone in the bug had a random issue with patch that
> wasn't even analyzed by devs, doesn't mean it will "break millions of
> systems" - especially if it isn't enabled by default (which is wise, since
> it would need more memory). It's simply a matter of willing to check and
> implement it.

Not to put too fine a point on it, but if you are unhappy with the service you
are receiving, you should switch to a different vendor. I suspect it's
unlikely you'll get the same value for money elsewhere.

Scott K



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,
On Sat, 31 Aug 2019, Henrik K wrote:
> On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote:
>
> The final responsibility of implementing and testing the issue is still that
> of the ClamAV team.

Agreed.

> You are really making this much more complex and "scary" issue than it is.

No, I don't think I am. How much experience do you have of writing
thread-safe code in C?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Sat, Aug 31, 2019 at 12:21:11PM -0400, Scott Kitterman via clamav-users wrote:
>
> Not to put too fine a point on it, but if you are unhappy with the service you
> are receiving, you should switch to a different vendor. I suspect it's
> unlikely you'll get the same value for money elsewhere.

Does this worn cliche really need posting? :-)

But hey, I'm just participating in the community.. sometimes things just
need a bit of nudging. I wouldn't even continue to nag about it, if this
was a basic volunteer project. But we are talking about a security company
that should be proud of it's code.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On August 31, 2019 4:32:00 PM UTC, Henrik K <hege@hege.li> wrote:
>On Sat, Aug 31, 2019 at 12:21:11PM -0400, Scott Kitterman via
>clamav-users wrote:
>>
>> Not to put too fine a point on it, but if you are unhappy with the
>service you
>> are receiving, you should switch to a different vendor. I suspect
>it's
>> unlikely you'll get the same value for money elsewhere.
>
>Does this worn cliche really need posting? :-)
>
>But hey, I'm just participating in the community.. sometimes things
>just
>need a bit of nudging. I wouldn't even continue to nag about it, if
>this
>was a basic volunteer project. But we are talking about a security
>company
>that should be proud of it's code.

There's no problem with nudging, but being nasty about isn't appropriate. I'll confess that I aimed a comment at you that should have been pointed at the OP. Sorry about that.

I've been maintaining clamav packages for over a decade through three different companies owning the project and overall I think the Talos/Cisco people are doing a pretty good job.

They are generally responsive.

Also, in terms of being proud of their code, you should compare the code quality when Sourcefire bought clamav to the current code base. It's night and day different. Also the rate of security bugs seems to have dropped off (not to mention we actually get bugfix releases now).

So yeah, they could do better, but looking back, I think they're doing pretty good.

In this case, the 'worn cliche' is important, because unlike lots of other FOSS projects, this one doesn't have non-proprietary alternatives, so it'd be pretty awful if the community were to convince Talos that publishing it was more trouble than it was worth.

Scott K

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> Normally postfix gets a response after 3 secondes.
>
> In the clamav.log I see at the same time, that reloading the database
> takes up to two minutes.

Yes, reloading the DB can take some time depending on which signature
DBs you are using. I can't speak for postfix (I run sendmail), but on
my server if it can't run the AV scan, then it simply tempfails the
email and the remote server (should) try later.

I wouldn't call the current design a "bug"... It works as intended.
However it would be nice if a fresh DB could be parsed & loaded, then
swapped, to prevent service interruption.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Sat, 31 Aug 2019, J.R. via clamav-users wrote:

> ...
> I wouldn't call the current design a "bug"... It works as intended.

+1

> However it would be nice if a fresh DB could be parsed & loaded, then
> swapped, to prevent service interruption.

That's exactly what the patch in #10979 does. Unfortunately, although
as I've said it's simple enough to apply the patch, it's by no means a
simple patch and it would greatly benefit from some serious testing by
the community - especially by people who see higher volumes of mail
than I do.

Perhaps we should call it "crowd-sourcing"? Would that be better? :)

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users:

>
> That's exactly what the patch in #10979 does. Unfortunately, although
> as I've said it's simple enough to apply the patch, it's by no means a
> simple patch and it would greatly benefit from some serious testing by
> the community - especially by people who see higher volumes of mail
> than I do.
>

And where can I find this patch? It s not on the download page
(https://www.clamav.net/downloads), so it s not official. I would like
to test it on my private server first, just to see if I get it work. And
what happens if I update my system (# aptitude update && aptitude
safe-upgrade) and a new verson of clamav is being installed. Do I always
have to repatch clamav?

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

>Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users:
>>That's exactly what the patch in #10979 does. Unfortunately, although
>>as I've said it's simple enough to apply the patch, it's by no means a
>>simple patch and it would greatly benefit from some serious testing by
>>the community - especially by people who see higher volumes of mail
>>than I do.

On 01.09.19 18:08, Thomas Barth via clamav-users wrote:
>And where can I find this patch? It s not on the download page
>(https://www.clamav.net/downloads), so it s not official. I would like
>to test it on my private server first, just to see if I get it work.
>And what happens if I update my system (# aptitude update && aptitude
>safe-upgrade) and a new verson of clamav is being installed. Do I
>always have to repatch clamav?

it's attached to the bugreport
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Saving Private Ryan...
Private Ryan exists. Overwrite? (Y/N)

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote:

> Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users:
>>
>> That's exactly what the patch in #10979 does. ...
>
> And where can I find this patch?

If you navigate to

https://bugzilla.clamav.net/show_bug.cgi?id=10979

and then down to "Comment 2" (dated 2016-11-28 12:16:52 EST) you will
see a link "attachment 7196". This is a modification to the original
#10979 patch. If you navigate to that link you will see a page which
gives a representation of the patch 'diff'. Near the top of that page
there is a link "Raw Unified", which takes you to the raw unified diff
text which is here:

https://bugzilla.clamav.net/attachment.cgi?id=7196&action=diff&context=patch&collapsed=&headers=1&format=raw

You might be able to use this as input to 'patch' but I didn't try it,
I did not expect it to work well on code which is years younger than
that on which the patch is based. Instead, I applied the patch by
hand with an editor. It was tedious but not difficult. Even if you
do not believe that you can trust my patched files (which I think is a
perfectly reasonable belief:) I should be happy to mail the patched
files to you so that you can compare the results of patching to give
you some confidence that it will work.

> ... what happens if I update my system (# aptitude update &&
> aptitude safe-upgrade) and a new verson of clamav is being
> installed. Do I always have to repatch clamav?

If you want to use this patch you must compile and install ClamAV from
the sources distributed on the clamav.net Website. You cannot use the
package management system of any Operating System (OS) distribution to
install any version of the ClamAV package(s) from the OS distribution.
Of course you could create your own package from the patched sources,
and then use the package management system to install your own package.
Many administrators do that when they have large numbers of machines
to be installed but they have some reason to avoid using the packages
produced by the OS publisher. If 'upstream' produces a new version of
the package which (still) does not contain the patch then yes, you do
have to re-apply the patch.

Your package manager will probably set up ClamAV in a way which is
very different from the way it is set up after building from source,
e.g. using directory paths like /usr/bin and /usr/sbin instead of
/usr/local/bin, /usr/local/sbin etc. - here are some samples from a
machine with both kinds of package installed:

mail6:~$ >>> l /usr/sbin/clam*
-rwxr-xr-x 1 root root 223296 Apr 15 22:12 /usr/sbin/clamd
-rwxr-xr-x 1 root root 233424 Apr 15 22:12 /usr/sbin/clamav-milter
mail6:~$ >>> l /usr/local/sbin/clam*
-rwxr-xr-x 1 root staff 581080 Aug 21 18:43 /usr/local/sbin/clamd
-rwxr-xr-x 1 root staff 581368 Aug 22 14:33 /usr/local/sbin/clamd_patched
mail6:~$ >>> l /usr/bin/freshclam
-rwxr-xr-x 1 root root 202816 Apr 15 22:12 /usr/bin/freshclam
mail6:~$ >>> l /usr/local/bin/freshclam
-rwxr-xr-x 1 root staff 442616 Aug 22 14:33 /usr/local/bin/freshclam

Note that there are THREE versions of clamd on this machine - the OS
distribution version and two versions built from source. The versions
built from source are the two which are currently running on the machine:

mail6:~$ >>> top -n1 -b -u clamav
top - 18:04:21 up 9 days, 1:49, 9 users, load average: 0.11, 0.33, 0.29
Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.1 us, 0.5 sy, 0.1 ni, 92.5 id, 0.3 wa, 0.0 hi, 4.4 si, 0.0 st
KiB Mem: 16469180 total, 15243004 used, 1226176 free, 232408 buffers
KiB Swap: 3212284 total, 0 used, 3212284 free. 11851656 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3846 clamav 20 0 61220 5644 4568 S 0.0 0.0 4:07.37 freshclam
5479 clamav 20 0 1430760 1.058g 4604 S 0.0 6.7 115:21.15 clamd
7689 clamav 20 0 1490600 1.061g 4656 S 0.0 6.8 123:10.10 clamd_patched

There will be other path differences too, for configuration and data
file stores. If you do something like this then you need to make sure
that you're running the right binaries, and that the binaries will use
the right configurations and libraries. If you aren't sure you can do
that then it would be best to uninstall and *purge* the OS versions of
the packages before you install the package from source. This applies
not just to ClamAV, but to any package where there may be conflicts of
this kind.

HTH

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Alright. I think we’ve beat the proverbial dead horse here. The devs know this is a request and they will get it into their dev queue for examination.

Sent from my ? iPhone

> On Sep 1, 2019, at 13:21, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?Hi there,
>
>> On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote:
>>
>> Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users:
>>> That's exactly what the patch in #10979 does. ...
>>
>> And where can I find this patch?
>
> If you navigate to
>
> https://bugzilla.clamav.net/show_bug.cgi?id=10979
>
> and then down to "Comment 2" (dated 2016-11-28 12:16:52 EST) you will
> see a link "attachment 7196". This is a modification to the original
> #10979 patch. If you navigate to that link you will see a page which
> gives a representation of the patch 'diff'. Near the top of that page
> there is a link "Raw Unified", which takes you to the raw unified diff
> text which is here:
>
> https://bugzilla.clamav.net/attachment.cgi?id=7196&action=diff&context=patch&collapsed=&headers=1&format=raw
>
> You might be able to use this as input to 'patch' but I didn't try it,
> I did not expect it to work well on code which is years younger than
> that on which the patch is based. Instead, I applied the patch by
> hand with an editor. It was tedious but not difficult. Even if you
> do not believe that you can trust my patched files (which I think is a
> perfectly reasonable belief:) I should be happy to mail the patched
> files to you so that you can compare the results of patching to give
> you some confidence that it will work.
>
>> ... what happens if I update my system (# aptitude update &&
>> aptitude safe-upgrade) and a new verson of clamav is being
>> installed. Do I always have to repatch clamav?
>
> If you want to use this patch you must compile and install ClamAV from
> the sources distributed on the clamav.net Website. You cannot use the
> package management system of any Operating System (OS) distribution to
> install any version of the ClamAV package(s) from the OS distribution.
> Of course you could create your own package from the patched sources,
> and then use the package management system to install your own package.
> Many administrators do that when they have large numbers of machines
> to be installed but they have some reason to avoid using the packages
> produced by the OS publisher. If 'upstream' produces a new version of
> the package which (still) does not contain the patch then yes, you do
> have to re-apply the patch.
>
> Your package manager will probably set up ClamAV in a way which is
> very different from the way it is set up after building from source,
> e.g. using directory paths like /usr/bin and /usr/sbin instead of
> /usr/local/bin, /usr/local/sbin etc. - here are some samples from a
> machine with both kinds of package installed:
>
> mail6:~$ >>> l /usr/sbin/clam*
> -rwxr-xr-x 1 root root 223296 Apr 15 22:12 /usr/sbin/clamd
> -rwxr-xr-x 1 root root 233424 Apr 15 22:12 /usr/sbin/clamav-milter
> mail6:~$ >>> l /usr/local/sbin/clam*
> -rwxr-xr-x 1 root staff 581080 Aug 21 18:43 /usr/local/sbin/clamd
> -rwxr-xr-x 1 root staff 581368 Aug 22 14:33 /usr/local/sbin/clamd_patched
> mail6:~$ >>> l /usr/bin/freshclam
> -rwxr-xr-x 1 root root 202816 Apr 15 22:12 /usr/bin/freshclam
> mail6:~$ >>> l /usr/local/bin/freshclam
> -rwxr-xr-x 1 root staff 442616 Aug 22 14:33 /usr/local/bin/freshclam
>
> Note that there are THREE versions of clamd on this machine - the OS
> distribution version and two versions built from source. The versions
> built from source are the two which are currently running on the machine:
>
> mail6:~$ >>> top -n1 -b -u clamav
> top - 18:04:21 up 9 days, 1:49, 9 users, load average: 0.11, 0.33, 0.29
> Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 2.1 us, 0.5 sy, 0.1 ni, 92.5 id, 0.3 wa, 0.0 hi, 4.4 si, 0.0 st
> KiB Mem: 16469180 total, 15243004 used, 1226176 free, 232408 buffers
> KiB Swap: 3212284 total, 0 used, 3212284 free. 11851656 cached Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 3846 clamav 20 0 61220 5644 4568 S 0.0 0.0 4:07.37 freshclam
> 5479 clamav 20 0 1430760 1.058g 4604 S 0.0 6.7 115:21.15 clamd
> 7689 clamav 20 0 1490600 1.061g 4656 S 0.0 6.8 123:10.10 clamd_patched
>
> There will be other path differences too, for configuration and data
> file stores. If you do something like this then you need to make sure
> that you're running the right binaries, and that the binaries will use
> the right configurations and libraries. If you aren't sure you can do
> that then it would be best to uninstall and *purge* the OS versions of
> the packages before you install the package from source. This applies
> not just to ClamAV, but to any package where there may be conflicts of
> this kind.
>
> HTH
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

Hi Joel,

On Sun, 1 Sep 2019, Joel Esler (jesler) wrote:

> Alright. I think we?ve beat the proverbial dead horse here. ...

I don't think anybody's beating anything here Joel. Just we users,
discussing, on the users' list, ways of dealing with an issue.

On Sat, 31 Aug 2019, G.W. Haywood wrote:

> It really isn't that big a deal if you know what you're doing.

You saw that part?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi Joel,

On Sun, 1 Sep 2019, Joel Esler (jesler) wrote:

> Alright. I think we?ve beat the proverbial dead horse here. ...

I don't think anybody's beating anything here Joel. Just we users,
discussing, on the users' list, ways of dealing with an issue.

On Sat, 31 Aug 2019, G.W. Haywood wrote:

> It really isn't that big a deal if you know what you're doing.

You saw that part?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Am 2019-09-01 19:12, schrieb G.W. Haywood via clamav-users:
> Hi there,
>
> On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote:
>
>> Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users:
>>>
>>> That's exactly what the patch in #10979 does. ...
>>
>> And where can I find this patch?
>
> If you navigate to
>
> https://bugzilla.clamav.net/show_bug.cgi?id=10979
>
> [...]

Thank you for your detailed and patient explanation. I thought it s just
a configure && install ;-) But in the moment, it would be too much for
me to manage clamav by hand.





_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Am 2019-09-01 19:30, schrieb Joel Esler (jesler) via clamav-users:
> Alright. I think we’ve beat the proverbial dead horse here. The devs
> know this is a request and they will get it into their dev queue for
> examination.

I saw that clamd use just one core at a time to load the databases.

top - 16:09:43 up 23:33, 2 users, load average: 0.47, 0.13, 0.04
Tasks: 176 total, 2 running, 174 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si,
1.0 st
%Cpu1 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
%Cpu2 : 85.4 us, 6.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si,
7.8 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
%Cpu4 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
MiB Mem : 16042.2 total, 14207.5 free, 813.4 used, 1021.3
buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 14921.4 avail
Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
455 clamav 20 0 469352 207432 10840 R 100.0 1.3 30:33.51
/usr/sbin/clamd --foreground=true
22861 root 20 0 11316 3648 3108 R 2.0 0.0 0:24.91 top

Always 2 minute loading time
Wed Sep 4 16:09:17 2019 -> Reading databases from /var/lib/clamav
Wed Sep 4 16:11:24 2019 -> Database correctly reloaded (10966440
signatures)


Why not using half of the cores to also reduce the loading time? Many
years ago when I used eMule for downloading big files, I was so
fascinated by the download mechanism: one big file, many download
sources to get the file together piece by piece. And it didn't have to
follow any order. That would be fun to programm for loading the
databases, am I right? :-)



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

The database load process reads signatures and uses the data to populate a couple of pseudo-tries (https://en.wikipedia.org/wiki/Trie). The tries themselves could only be modified by a single thread at a time, with a mutex around each trie. There might be some performance to be gained by using multiple threads. I'm not certain. Definitely a bunch of thread safety code would need to be written.

-Micah

?On 9/4/19, 1:40 PM, "clamav-users on behalf of Thomas Barth via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

Am 2019-09-01 19:30, schrieb Joel Esler (jesler) via clamav-users:
> Alright. I think we’ve beat the proverbial dead horse here. The devs
> know this is a request and they will get it into their dev queue for
> examination.

I saw that clamd use just one core at a time to load the databases.

top - 16:09:43 up 23:33, 2 users, load average: 0.47, 0.13, 0.04
Tasks: 176 total, 2 running, 174 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si,
1.0 st
%Cpu1 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
%Cpu2 : 85.4 us, 6.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si,
7.8 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
%Cpu4 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
MiB Mem : 16042.2 total, 14207.5 free, 813.4 used, 1021.3
buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 14921.4 avail
Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
455 clamav 20 0 469352 207432 10840 R 100.0 1.3 30:33.51
/usr/sbin/clamd --foreground=true
22861 root 20 0 11316 3648 3108 R 2.0 0.0 0:24.91 top

Always 2 minute loading time
Wed Sep 4 16:09:17 2019 -> Reading databases from /var/lib/clamav
Wed Sep 4 16:11:24 2019 -> Database correctly reloaded (10966440
signatures)


Why not using half of the cores to also reduce the loading time? Many
years ago when I used eMule for downloading big files, I was so
fascinated by the download mechanism: one big file, many download
sources to get the file together piece by piece. And it didn't have to
follow any order. That would be fun to programm for loading the
databases, am I right? :-)



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml