Mailing List Archive

Hello Bowie,

Le 19/03/2019 à 15:35, Bowie Bailey a écrit :
> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
> seems to frequently happen when we are sending an email, so the Thunderbird will time
> out on the send (although the message will frequently go through anyway).
>
> I do have a bunch of third party signatures installed from Sanesecurity and
> SecuriteInfo.  Is there a way to get timing information on which signature files are
> taking the longest to load?  Or is this mainly a function of file size?

javascript.ndb is taking most of the reload time.
You can test the time to reload without this file.

--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.44.39.76.46
E-mail : aj@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom

Securiteinfo.com
La Sécurité Informatique - La Sécurité des Informations.
266, rue de Villers
60123 Bonneuil en Valois


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/19/2019 10:44 AM, Arnaud Jacques wrote:
> Hello Bowie,
>
> Le 19/03/2019 à 15:35, Bowie Bailey a écrit :
>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>> out on the send (although the message will frequently go through anyway).
>>
>> I do have a bunch of third party signatures installed from Sanesecurity and
>> SecuriteInfo.  Is there a way to get timing information on which signature files are
>> taking the longest to load?  Or is this mainly a function of file size?
>
> javascript.ndb is taking most of the reload time.
> You can test the time to reload without this file

Without javascript.ndb, the reload time drops to 2 minutes.  Better, but still longer
than I'd like.

Is there a way to get the details on how long each file take to load, or do I just
have to test them one by one?

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/19/2019 4:27 PM, Bowie Bailey wrote:
> On 3/19/2019 10:44 AM, Arnaud Jacques wrote:
>> Hello Bowie,
>>
>> Le 19/03/2019 à 15:35, Bowie Bailey a écrit :
>>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
>>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>>> out on the send (although the message will frequently go through anyway).
>>>
>>> I do have a bunch of third party signatures installed from Sanesecurity and
>>> SecuriteInfo.  Is there a way to get timing information on which signature files are
>>> taking the longest to load?  Or is this mainly a function of file size?
>> javascript.ndb is taking most of the reload time.
>> You can test the time to reload without this file
> Without javascript.ndb, the reload time drops to 2 minutes.  Better, but still longer
> than I'd like.
>
> Is there a way to get the details on how long each file take to load, or do I just
> have to test them one by one?

I did a check on the SecuriteInfo signatures.  I grepped my clamd logs for hits on
SecuriteInfo signatures and then matched them to the file they came from.

#1 was spam_marketing.ndb with 110 hits
#2 was javascript.ndb with 10 hits

And that was it.  securiteinfo.hdb, securiteinfohtml.hdb, and securiteinfoascii.hdb
did not produce a single hit on my system in the past year.  Unfortunately, removing
those signatures does not have much of an effect on the reload times.

--
Bowie


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 19 March 2019 21:01:03 Bowie Bailey <Bowie_Bailey@BUC.com> wrote:

> On 3/19/2019 4:27 PM, Bowie Bailey wrote:
>>
>>
>>
>> Is there a way to get the details on how long each file take to load, or do
>> I just
>> have to test them one by one?
A very simple per Database scan time test... Sorry not sorted in time order
but might help anyway...

badmacro.ndb: 984 ms

blurl.ndb: 1469 ms

bofhland_cracked_URL.ndb: 875 ms

bofhland_malware_attach.hdb: 891 ms

bofhland_malware_URL.ndb: 891 ms

bofhland_phishing_URL.ndb: 890 ms

crdfam.clamav.hdb: 891 ms

doppelstern-phishtank.ndb: 891 ms

doppelstern.hdb: 890 ms

doppelstern.ndb: 906 ms

foxhole_all.cdb: 891 ms

foxhole_all.ndb: 921 ms

foxhole_filename.cdb: 1000 ms

foxhole_generic.cdb: 907 ms

foxhole_js.cdb: 890 ms

foxhole_js.ndb: 875 ms

foxhole_mail.cdb: 891 ms

junk.ndb: 1812 ms

jurlbl.ndb: 1047 ms

jurlbla.ndb: 937 ms

lott.ndb: 890 ms

malware.expert.hdb: 875 ms

malware.expert.ldb: 875 ms

malware.expert.ndb: 906 ms

MiscreantPunch099-INFO-Low.ldb: 953 ms

MiscreantPunch099-Low.ldb:
Possible Performance Issue: 10422 ms

phish.ndb: 4406 ms

phishtank.ndb: 1156 ms

porcupine.ndb: 969 ms

rogue.hdb: 922 ms

scam.ndb: 1235 ms

scamnailer.ndb: 4016 ms

shelter.ldb: 906 ms

spam.ldb: 922 ms

spamattach.hdb: 906 ms

spamimg.hdb: 891 ms

spear.ndb: 1594 ms

spearl.ndb: 906 ms

winnow.attachments.hdb: 891 ms

winnow.complex.patterns.ldb: 891 ms

winnow_bad_cw.hdb: 891 ms

winnow_extended_malware.hdb: 984 ms

winnow_extended_malware_links.ndb: 891 ms

winnow_malware.hdb: 907 ms

winnow_malware_links.ndb: 890 ms

winnow_phish_complete.ndb: 890 ms

winnow_phish_complete_url.ndb: 875 ms

winnow_spam_complete.ndb: 875 ms


Cheers,


Steve
Twitter: @sanesecurity

Hello Bowie,

> I did a check on the SecuriteInfo signatures.  I grepped my clamd logs for hits on
> SecuriteInfo signatures and then matched them to the file they came from.
>
> #1 was spam_marketing.ndb with 110 hits
> #2 was javascript.ndb with 10 hits
>
> And that was it.  securiteinfo.hdb, securiteinfohtml.hdb, and securiteinfoascii.hdb
> did not produce a single hit on my system in the past year.  Unfortunately, removing
> those signatures does not have much of an effect on the reload times.
Do you have Professional subscription ?
If no : you'll have many more hits if you have Professional subscription.
If yes : you forget some important databases to get more hits. Please
login to you account.

Anyway, here are sorted loading time on my system (Intel i7-6700 CPU)
using time clamscan -d :

securiteinfo0hour.hdb : 0.021s
securiteinfo.mdb : 0.033s
securiteinfopdf.hdb : 0.047s
securiteinfohtml.hdb : 0.076s
securiteinfoascii.hdb : 0.163s
securiteinfoandroid.hdb : 0.214s
spam_marketing.ndb : 0.332s
securiteinfo.hdb : 0.894s
securiteinfoold.hdb : 6.801s
javascript.ndb : 21.133s

An antivirus is like any other software : it has minimum requirements.
Every years, it is dozen of thousands malwares in the wild. If we want
to detect all of them, we have to include them in antivirus databases.
That's why I recommend to get a *fast* CPU and a lot of RAM, because it
still growing past years !
In my opinion it is a bad idea to run clamAV+our provided databases
signatures on low performance CPU (VMs, embeeded systems, old hardware, ...)

Could you please tell us the CPU you use ?

--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.44.39.76.46
E-mail : aj@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom

Securiteinfo.com
La Sécurité Informatique - La Sécurité des Informations.
266, rue de Villers
60123 Bonneuil en Valois


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 2019-03-19 14:35, Bowie Bailey wrote:

> I do have a bunch of third party signatures installed from Sanesecurity
> and
> SecuriteInfo.  Is there a way to get timing information on which
> signature files are
> taking the longest to load?  Or is this mainly a function of file size?

Here's a quick sorted output of scanning each database with 1 file to
test db loading and scan times....

eg.

clamscan --database=xxxxx.xxx


Sanesecurity:

MiscreantPunch099-Low.ldb Execution time 10.534 s
phish.ndb Execution time 4.400 s
scamnailer.ndb Execution time 4.070 s
junk.ndb Execution time 1.840 s
spear.ndb Execution time 1.610 s
blurl.ndb Execution time 1.387 s
scam.ndb Execution time 1.292 s
phishtank.ndb Execution time 1.124 s
jurlbl.ndb Execution time 1.062 s
winnow_extended_malware.hdb Execution time 1.017 s
foxhole_filename.cdb Execution time 1.013 s
MiscreantPunch099-INFO-Low.ldb Execution time 1.007 s
porcupine.ndb Execution time 0.990 s
jurlbla.ndb Execution time 0.974 s
doppelstern.hdb Execution time 0.953 s
doppelstern-phishtank.ndb Execution time 0.945 s
rogue.hdb Execution time 0.945 s
foxhole_all.ndb Execution time 0.937 s
foxhole_js.ndb Execution time 0.932 s
foxhole_generic.cdb Execution time 0.931 s
bofhland_malware_URL.ndb Execution time 0.930 s
winnow.complex.patterns.ldb Execution time 0.928 s
winnow_phish_complete_url.ndb Execution time 0.926 s
foxhole_all.cdb Execution time 0.922 s
lott.ndb Execution time 0.919 s
malware.expert.ndb Execution time 0.919 s
bofhland_cracked_URL.ndb Execution time 0.917 s
foxhole_mail.cdb Execution time 0.917 s
winnow_malware.hdb Execution time 0.917 s
winnow.attachments.hdb Execution time 0.916 s
winnow_bad_cw.hdb Execution time 0.916 s
crdfam.clamav.hdb Execution time 0.915 s
winnow_extended_malware_links.ndb Execution time 0.915 s
spearl.ndb Execution time 0.914 s
badmacro.ndb Execution time 0.913 s
foxhole_js.cdb Execution time 0.913 s
winnow_malware_links.ndb Execution time 0.913 s
winnow_spam_complete.ndb Execution time 0.911 s
bofhland_phishing_URL.ndb Execution time 0.910 s
malware.expert.ldb Execution time 0.910 s
bofhland_malware_attach.hdb Execution time 0.909 s
doppelstern.ndb Execution time 0.909 s
winnow_phish_complete.ndb Execution time 0.908 s
shelter.ldb Execution time 0.907 s
spamattach.hdb Execution time 0.905 s
spamimg.hdb Execution time 0.896 s
malware.expert.hdb Execution time 0.895 s
spam.ldb Execution time 0.889 s

and the same setup for ClamAV Official DB's:

daily.ldb Execution time 35.893 s
main.mdb Execution time 13.474 s
daily.hsb Execution time 10.514 s
daily.hdb Execution time 3.533 s
main.ndb Execution time 2.676 s
main.hsb Execution time 2.020 s
daily.mdb Execution time 1.610 s
main.hdb Execution time 1.054 s
daily.ndb Execution time 0.983 s
daily.pdb Execution time 0.946 s
daily.ldu Execution time 0.942 s
daily.idb Execution time 0.940 s
daily.ign2 Execution time 0.937 s
daily.ign Execution time 0.930 s
daily.fp Execution time 0.921 s
daily.wdb Execution time 0.919 s
daily.ftm Execution time 0.918 s
main.sfp Execution time 0.914 s
daily.hsu Execution time 0.911 s
daily.sfp Execution time 0.910 s
daily.cfg Execution time 0.909 s
daily.crb Execution time 0.907 s
daily.msu Execution time 0.906 s
daily.hdu Execution time 0.902 s
daily.msb Execution time 0.902 s
daily.ndu Execution time 0.902 s
daily.cdb Execution time 0.901 s
main.msb Execution time 0.896 s
main.crb Execution time 0.894 s
main.fp Execution time 0.891 s
daily.mdu Execution time 0.889 s

Just in case that's useful for anyone ;)

--
Cheers,

Steve
Twitter: @sanesecurity

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

All these times, I would imagine, would be based on the amount of CPU and RAM, even disk read speed, available to the machine loading. So these times are relative.

Sent from my ? iPhone

> On Mar 20, 2019, at 07:48, Steve Basford <steveb_clamav@sanesecurity.com> wrote:
>
>> On 2019-03-19 14:35, Bowie Bailey wrote:
>>
>> I do have a bunch of third party signatures installed from Sanesecurity and
>> SecuriteInfo. Is there a way to get timing information on which
>> signature files are
>> taking the longest to load? Or is this mainly a function of file size?
>
> Here's a quick sorted output of scanning each database with 1 file to test db loading and scan times....
>
> eg.
>
> clamscan --database=xxxxx.xxx
>
>
> Sanesecurity:
>
> MiscreantPunch099-Low.ldb Execution time 10.534 s
> phish.ndb Execution time 4.400 s
> scamnailer.ndb Execution time 4.070 s
> junk.ndb Execution time 1.840 s
> spear.ndb Execution time 1.610 s
> blurl.ndb Execution time 1.387 s
> scam.ndb Execution time 1.292 s
> phishtank.ndb Execution time 1.124 s
> jurlbl.ndb Execution time 1.062 s
> winnow_extended_malware.hdb Execution time 1.017 s
> foxhole_filename.cdb Execution time 1.013 s
> MiscreantPunch099-INFO-Low.ldb Execution time 1.007 s
> porcupine.ndb Execution time 0.990 s
> jurlbla.ndb Execution time 0.974 s
> doppelstern.hdb Execution time 0.953 s
> doppelstern-phishtank.ndb Execution time 0.945 s
> rogue.hdb Execution time 0.945 s
> foxhole_all.ndb Execution time 0.937 s
> foxhole_js.ndb Execution time 0.932 s
> foxhole_generic.cdb Execution time 0.931 s
> bofhland_malware_URL.ndb Execution time 0.930 s
> winnow.complex.patterns.ldb Execution time 0.928 s
> winnow_phish_complete_url.ndb Execution time 0.926 s
> foxhole_all.cdb Execution time 0.922 s
> lott.ndb Execution time 0.919 s
> malware.expert.ndb Execution time 0.919 s
> bofhland_cracked_URL.ndb Execution time 0.917 s
> foxhole_mail.cdb Execution time 0.917 s
> winnow_malware.hdb Execution time 0.917 s
> winnow.attachments.hdb Execution time 0.916 s
> winnow_bad_cw.hdb Execution time 0.916 s
> crdfam.clamav.hdb Execution time 0.915 s
> winnow_extended_malware_links.ndb Execution time 0.915 s
> spearl.ndb Execution time 0.914 s
> badmacro.ndb Execution time 0.913 s
> foxhole_js.cdb Execution time 0.913 s
> winnow_malware_links.ndb Execution time 0.913 s
> winnow_spam_complete.ndb Execution time 0.911 s
> bofhland_phishing_URL.ndb Execution time 0.910 s
> malware.expert.ldb Execution time 0.910 s
> bofhland_malware_attach.hdb Execution time 0.909 s
> doppelstern.ndb Execution time 0.909 s
> winnow_phish_complete.ndb Execution time 0.908 s
> shelter.ldb Execution time 0.907 s
> spamattach.hdb Execution time 0.905 s
> spamimg.hdb Execution time 0.896 s
> malware.expert.hdb Execution time 0.895 s
> spam.ldb Execution time 0.889 s
>
> and the same setup for ClamAV Official DB's:
>
> daily.ldb Execution time 35.893 s
> main.mdb Execution time 13.474 s
> daily.hsb Execution time 10.514 s
> daily.hdb Execution time 3.533 s
> main.ndb Execution time 2.676 s
> main.hsb Execution time 2.020 s
> daily.mdb Execution time 1.610 s
> main.hdb Execution time 1.054 s
> daily.ndb Execution time 0.983 s
> daily.pdb Execution time 0.946 s
> daily.ldu Execution time 0.942 s
> daily.idb Execution time 0.940 s
> daily.ign2 Execution time 0.937 s
> daily.ign Execution time 0.930 s
> daily.fp Execution time 0.921 s
> daily.wdb Execution time 0.919 s
> daily.ftm Execution time 0.918 s
> main.sfp Execution time 0.914 s
> daily.hsu Execution time 0.911 s
> daily.sfp Execution time 0.910 s
> daily.cfg Execution time 0.909 s
> daily.crb Execution time 0.907 s
> daily.msu Execution time 0.906 s
> daily.hdu Execution time 0.902 s
> daily.msb Execution time 0.902 s
> daily.ndu Execution time 0.902 s
> daily.cdb Execution time 0.901 s
> main.msb Execution time 0.896 s
> main.crb Execution time 0.894 s
> main.fp Execution time 0.891 s
> daily.mdu Execution time 0.889 s
>
> Just in case that's useful for anyone ;)
>
> --
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote:

> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
> seems to frequently happen when we are sending an email, so the Thunderbird will time
> out on the send (although the message will frequently go through anyway).


The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database.


Best
Ale
--

https://www.tana.it/sw/avfilter/






_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/20/2019 2:57 AM, Arnaud Jacques wrote:
> Hello Bowie,
>
>> I did a check on the SecuriteInfo signatures.  I grepped my clamd logs for hits on
>> SecuriteInfo signatures and then matched them to the file they came from.
>>
>> #1 was spam_marketing.ndb with 110 hits
>> #2 was javascript.ndb with 10 hits
>>
>> And that was it.  securiteinfo.hdb, securiteinfohtml.hdb, and securiteinfoascii.hdb
>> did not produce a single hit on my system in the past year.  Unfortunately, removing
>> those signatures does not have much of an effect on the reload times.
> Do you have Professional subscription ?
> If no : you'll have many more hits if you have Professional subscription.
> If yes : you forget some important databases to get more hits. Please login to you
> account.
>
> Anyway, here are sorted loading time on my system (Intel i7-6700 CPU) using time
> clamscan -d :
>
> securiteinfo0hour.hdb : 0.021s
> securiteinfo.mdb : 0.033s
> securiteinfopdf.hdb : 0.047s
> securiteinfohtml.hdb : 0.076s
> securiteinfoascii.hdb : 0.163s
> securiteinfoandroid.hdb : 0.214s
> spam_marketing.ndb : 0.332s
> securiteinfo.hdb : 0.894s
> securiteinfoold.hdb : 6.801s
> javascript.ndb : 21.133s
>
> An antivirus is like any other software : it has minimum requirements.
> Every years, it is dozen of thousands malwares in the wild. If we want to detect
> all of them, we have to include them in antivirus databases.
> That's why I recommend to get a *fast* CPU and a lot of RAM, because it still
> growing past years !
> In my opinion it is a bad idea to run clamAV+our provided databases signatures on
> low performance CPU (VMs, embeeded systems, old hardware, ...)
>
> Could you please tell us the CPU you use ?

No, I don't have a subscription.

My system is a bit old.  It's running a quad-core Xeon 3Ghz with 4G of RAM.  The RAM
doesn't seem to be an issue, just the slow loading due to the CPU.  It scans a bit
slow as well, but that's not an issue since we don't have a high email volume.

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote:
> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote:
>
>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>> out on the send (although the message will frequently go through anyway).
>
> The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database.

That would be ideal, but it doesn't seem to be happening that way.  If I look at my
logs, I see "SelfCheck", then "Reading databases", and at that point all scanning
stops until the "Database correctly reloaded" message 2 1/2 minutes later.

Is there a setting somewhere to allow scanning to continue with the existing child
processes while the reload happens?

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

I think Alessandro was suggesting how it could work, not how it does work. Clamd doesn't work that way at present. It has been a feature request for a very long time, one that I hope we can address sometime soon, but I don't know when.

Micah

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.



?On 3/20/19, 10:04 AM, "clamav-users on behalf of Bowie Bailey" <clamav-users-bounces@lists.clamav.net on behalf of Bowie_Bailey@BUC.com> wrote:

On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote:
> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote:
>
>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server. This
>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>> out on the send (although the message will frequently go through anyway).
>
> The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database.

That would be ideal, but it doesn't seem to be happening that way. If I look at my
logs, I see "SelfCheck", then "Reading databases", and at that point all scanning
stops until the "Database correctly reloaded" message 2 1/2 minutes later.

Is there a setting somewhere to allow scanning to continue with the existing child
processes while the reload happens?

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Wed 20/Mar/2019 14:53:28 +0100 Bowie Bailey wrote:

> On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote:
>> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote:
>>
>>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
>>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>>> out on the send (although the message will frequently go through anyway).
>>
>> The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database.
>
> That would be ideal, but it doesn't seem to be happening that way.  If I look at my
> logs, I see "SelfCheck", then "Reading databases", and at that point all scanning
> stops until the "Database correctly reloaded" message 2 1/2 minutes later.


Are you using clamd directly (rather than libclamav)? How is it connected to
the mail server?


> Is there a setting somewhere to allow scanning to continue with the existing child
> processes while the reload happens?


It has to be coded inside the mail filter; avfilter for Courier works that way,
for one.


Best
Ale
--





_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/20/2019 2:57 PM, Alessandro Vesely via clamav-users wrote:
> On Wed 20/Mar/2019 14:53:28 +0100 Bowie Bailey wrote:
>
>> On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote:
>>> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote:
>>>
>>>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
>>>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>>>> out on the send (although the message will frequently go through anyway).
>>> The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database.
>> That would be ideal, but it doesn't seem to be happening that way.  If I look at my
>> logs, I see "SelfCheck", then "Reading databases", and at that point all scanning
>> stops until the "Database correctly reloaded" message 2 1/2 minutes later.
>
> Are you using clamd directly (rather than libclamav)? How is it connected to
> the mail server?
>
>
>> Is there a setting somewhere to allow scanning to continue with the existing child
>> processes while the reload happens?
>
> It has to be coded inside the mail filter; avfilter for Courier works that way,
> for one.

I am running Courier.  I am currently calling clamd via pythonfilter.  I'll have to
take a look at avfilter.

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Wed, 20 Mar 2019, Micah Snyder wrote:
> On 3/20/19, 10:04 AM, "clamav-users on behalf of Bowie Bailey" <clamav-users-bounces@lists.clamav.net on behalf of Bowie_Bailey@BUC.com> wrote:
>> On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote:
>>> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote:
>>>
>>>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server. This
>>>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>>>> out on the send (although the message will frequently go through anyway).
>>>
>>> The mail server should scan the message with the database at hand.
>>> A forked child can do the filtering while the parent reloads.
>>> Upon loading, the child exits and new messages will be scanned by
>>> the parent with the updated database.
>>
>> That would be ideal, but it doesn't seem to be happening that way. If I look at my
>> logs, I see "SelfCheck", then "Reading databases", and at that point all scanning
>> stops until the "Database correctly reloaded" message 2 1/2 minutes later.
>>
>> Is there a setting somewhere to allow scanning to continue with the existing child
>> processes while the reload happens?
>
> I think Alessandro was suggesting how it could work, not how it does
> work. Clamd doesn't work that way at present. It has been a
> feature request for a very long time, one that I hope we can address
> sometime soon, but I don't know when.

The simplest way to achieve this right now would probably be to use
two servers for scanning, and arrange for them to update their DBs
at different times. A simple milter with a knowledge of the update
schedule could choose which scanner to use just by checking the time.
I imagine that it wouldn't be difficult, for example, to modify the
well-known MimeDefang to do that. It wouldn't be much more difficult
to create a milter especially for the purpose.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/20/2019 3:53 PM, Bowie Bailey wrote:
> On 3/20/2019 2:57 PM, Alessandro Vesely via clamav-users wrote:
>> On Wed 20/Mar/2019 14:53:28 +0100 Bowie Bailey wrote:
>>
>>> On 3/20/2019 8:42 AM, Alessandro Vesely via clamav-users wrote:
>>>> On Tue 19/Mar/2019 15:35:39 +0100 Bowie Bailey wrote:
>>>>
>>>>> ClamAV is taking about 2 1/2 minutes to reload its database on my mail server.  This
>>>>> seems to frequently happen when we are sending an email, so the Thunderbird will time
>>>>> out on the send (although the message will frequently go through anyway).
>>>> The mail server should scan the message with the database at hand. A forked child can do the filtering while the parent reloads. Upon loading, the child exits and new messages will be scanned by the parent with the updated database.
>>> That would be ideal, but it doesn't seem to be happening that way.  If I look at my
>>> logs, I see "SelfCheck", then "Reading databases", and at that point all scanning
>>> stops until the "Database correctly reloaded" message 2 1/2 minutes later.
>> Are you using clamd directly (rather than libclamav)? How is it connected to
>> the mail server?
>>
>>
>>> Is there a setting somewhere to allow scanning to continue with the existing child
>>> processes while the reload happens?
>> It has to be coded inside the mail filter; avfilter for Courier works that way,
>> for one.
> I am running Courier.  I am currently calling clamd via pythonfilter.  I'll have to
> take a look at avfilter.

I'm trying to build avfilter, but it is complaining that it can't find libclamav.  I
have ClamAV 0.101.1 installed (built from source).  I was able to find libclamav.so
in /usr/local/lib64/.  Do I need to specify that directory somewhere, or am I missing
something else?  Do I need to do something special with ClamAV to get the dev stuff
installed for the build?

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> The simplest way to achieve this right now would probably be to use
> two servers for scanning, and arrange for them to update their DBs
> at different times. A simple milter with a knowledge of the update
> schedule could choose which scanner to use just by checking the time.
> I imagine that it wouldn't be difficult, for example, to modify the
> well-known MimeDefang to do that. It wouldn't be much more difficult
> to create a milter especially for the purpose.

Or just have the mail server send a 'tempfail' and the remote mail
server will retry sending usually within 10 minutes...

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Wed 20/Mar/2019 21:52:49 +0100 Bowie Bailey wrote:
>
> I'm trying to build avfilter, but it is complaining that it can't find libclamav.  I
> have ClamAV 0.101.1 installed (built from source).  I was able to find libclamav.so
> in /usr/local/lib64/.  Do I need to specify that directory somewhere, or am I missing
> something else?  Do I need to do something special with ClamAV to get the dev stuff
> installed for the build?


V.3.5 configure script runs 'pkg-config --atleast-version=0.101.0 libclamav'
and complains "libclamav missing or too old" if that fails. pkg-config is also
used to determine -l, -L, and -I. Those tests should all succeed with a source
built.

If you cannot find libclamav at runtime, it may be due to the library path
having been moved to /usr/local. In that case, you may have to run ldconfig or
adjust LD_LIBRARY_PATH or something to that end.


Best
Ale
--












_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/21/2019 5:07 AM, Alessandro Vesely via clamav-users wrote:
> On Wed 20/Mar/2019 21:52:49 +0100 Bowie Bailey wrote:
>> I'm trying to build avfilter, but it is complaining that it can't find libclamav.  I
>> have ClamAV 0.101.1 installed (built from source).  I was able to find libclamav.so
>> in /usr/local/lib64/.  Do I need to specify that directory somewhere, or am I missing
>> something else?  Do I need to do something special with ClamAV to get the dev stuff
>> installed for the build?
>
> V.3.5 configure script runs 'pkg-config --atleast-version=0.101.0 libclamav'
> and complains "libclamav missing or too old" if that fails. pkg-config is also
> used to determine -l, -L, and -I. Those tests should all succeed with a source
> built.
>
> If you cannot find libclamav at runtime, it may be due to the library path
> having been moved to /usr/local. In that case, you may have to run ldconfig or
> adjust LD_LIBRARY_PATH or something to that end.

That was helpful.

$ pkg-config --atleast-version=0.101.0 libclamav --print-errors
Package libclamav was not found in the pkg-config search path.
Perhaps you should add the directory containing `libclamav.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libclamav' found

Once I found that file (in /usr/local/lib64/pkgconfig) and added the directory to the
config path, I was able to complete the configure and make without any further errors.

When I ran "make check", it failed on 11 of 12 tests.  I was able to fix most of the
tests by adding '/usr/local/lib64' to the LD_LIBRARY_PATH so it could find
libclamav.so.  I assume I'll need to make a similar change somewhere when I add the
filter to Courier.

Now, it's only failing the per-virus behavior test.  The relevant section of the
testsuite.log looks like this:

9. testsuite.at:540: testing per-virus behavior ...
./testsuite.at:551: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
avtest.conf  -t two.mail
./testsuite.at:561: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
avtest.conf  -t two.mail
./testsuite.at:571: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
avtest.conf  -t two.mail
./testsuite.at:582: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
avtest.conf  -t two.mail
./testsuite.at:593: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
avtest.conf  -t two.mail
./testsuite.at:606: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
avtest.conf  -t eicar.mail
./testsuite.at:612: egrep '^ClamAV-Found: Eicar' eicar.mail | $WC -l
--- -   2019-03-21 09:56:42.104212351 -0400
+++ /home/bowieb/source/avfilter-3.5/tests/testsuite.dir/at-groups/9/stdout    
2019-03-21 09:56:42.100299898 -0400
@@ -1,2 +1,2 @@
-1
+0

9. testsuite.at:540: 9. per-virus behavior (testsuite.at:540): FAILED (testsuite.at:612)

Suggestions?

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Thu, 21 Mar 2019, J.R. wrote:

> > The simplest way to achieve this right now would probably be to use
> > two servers for scanning ...
>
> Or just have the mail server send a 'tempfail' and the remote mail
> server will retry sending usually within 10 minutes...

The OP specifically mentioned Thunderbird, not a remote mail server.
You often can't be sure what a mail client will do in this kind of
situation. Especially when it's Thunderbird. :(

Agreed the tempfail would be fine most of the time, but some people do
seem to confuse email with Instant Messaging. Just the other day, in
a discussion about email security, a guy grumbled about my greylisting
his mail for ten minutes! I decided not to tell him that the default
greylist period here is a couple of hours, and just whitelisted him...

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Thu 21/Mar/2019 15:05:59 +0100 Bowie Bailey wrote:
>
> $ pkg-config --atleast-version=0.101.0 libclamav --print-errors
> Package libclamav was not found in the pkg-config search path.
> Perhaps you should add the directory containing `libclamav.pc'
> to the PKG_CONFIG_PATH environment variable
> No package 'libclamav' found
>
> Once I found that file (in /usr/local/lib64/pkgconfig) and added the directory to the
> config path, I was able to complete the configure and make without any further errors.


Hm.. that way pkg-config couldn't find itself?


> When I ran "make check", it failed on 11 of 12 tests.  I was able to fix most of the
> tests by adding '/usr/local/lib64' to the LD_LIBRARY_PATH so it could find
> libclamav.so.  I assume I'll need to make a similar change somewhere when I add the
> filter to Courier.


Or you might add it to ld.so.conf? Otherwise, you may try building with CFLAGS=-Wl,-rpath,/usr/local/lib64 and check ldd. The issue is the ability to have libraries of different versions simultaneously installed on the same system. See e.g.:
https://unix.stackexchange.com/questions/356624/why-isnt-usr-local-lib-on-the-library-path-by-default


> Now, it's only failing the per-virus behavior test.  The relevant section of the
> testsuite.log looks like this:
>
> 9. testsuite.at:540: testing per-virus behavior ...
> ./testsuite.at:551: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
> avtest.conf  -t two.mail
> ./testsuite.at:561: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
> avtest.conf  -t two.mail
> ./testsuite.at:571: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
> avtest.conf  -t two.mail
> ./testsuite.at:582: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
> avtest.conf  -t two.mail
> ./testsuite.at:593: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
> avtest.conf  -t two.mail

Well, the above is my laziness of including multiple tests into one.


> ./testsuite.at:606: unset DEBUG_FILTER; $VALGRIND_AND_OPTS $AVFILTEREXEC -f
> avtest.conf  -t eicar.mail
> ./testsuite.at:612: egrep '^ClamAV-Found: Eicar' eicar.mail | $WC -l
> --- -   2019-03-21 09:56:42.104212351 -0400
> +++ /home/bowieb/source/avfilter-3.5/tests/testsuite.dir/at-groups/9/stdout    
> 2019-03-21 09:56:42.100299898 -0400
> @@ -1,2 +1,2 @@
> -1
> +0


That last test tests the following config setting:

action_default drop
virus_header ClamAV-Found
pass Eicar-Test*

That is, drop infected messages, but if the virus is named "Eicar-Test*" just add a header field and pass the message. The test message, eicar.mail, should actually be infected:

ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ clamscan -i -d ../../small eicar.mail
eicar.mail: Eicar-Test-Signature.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 1
Engine version: 0.101.1
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.008 sec (0 m 0 s)


Running avfilter on that should add a header field:

ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ ../../../src/tavfilter -f avtest.conf -t eicar.mail
250 Ok.

More verbosely:

ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ sed -i '/^verbose/s/0/4/' avtest.conf
ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ ../../../src/tavfilter -f avtest.conf -t eicar.mail
ERR: avfilter[12341]: unwritable pid dir directory /var/run/courier: Permission denied
avfilter[12341]: Virus engine version: 0.101.1 (f=102)
avfilter[12341]: Virus data version: 0
avfilter[12341]: Virus data date: 1 Jan 1970
avfilter[12341]: No. of viruses: 1
avfilter: running test on 1 files
avfilter[12341]: waiting for 1 child(ren)
INFO: avfilter: found virus "Eicar-Test-Signature.UNOFFICIAL"; specific action: pass
INFO: avfilter: found virus "Eicar-Test-Signature.UNOFFICIAL"; specific action: pass
INFO: avfilter: found virus Eicar-Test-Signature.UNOFFICIAL; final action pass; not saved
250 Ok.


At that point, the top of the header should be plenty of virus_header's (one for each invocation):

ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ head eicar.mail
ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
Eicar-Test-Signature.UNOFFICIAL
From: author@example.com
To: victim@example.net
Subject: test message
Virus-Header: what does this mean?
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=_1_1553193777_12188"

And hence:

ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ egrep '^ClamAV-Found: Eicar' eicar.mail | wc -l
1

Instead, at yours wc wrote "0". Why?

> 9. testsuite.at:540: 9. per-virus behavior (testsuite.at:540): FAILED (testsuite.at:612)
>
> Suggestions?
>

I'd guess something must have gone wrong in the testsuite script. In the testsuite, wc is $WC, after a definition in tests/atlocal, but egrep was not checked during configure, so maybe it should have been grep -E or similar. Is that the culprit?

Best
Ale
--





_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/21/2019 3:14 PM, Alessandro Vesely via clamav-users wrote:
> On Thu 21/Mar/2019 15:05:59 +0100 Bowie Bailey wrote:
>> $ pkg-config --atleast-version=0.101.0 libclamav --print-errors
>> Package libclamav was not found in the pkg-config search path.
>> Perhaps you should add the directory containing `libclamav.pc'
>> to the PKG_CONFIG_PATH environment variable
>> No package 'libclamav' found
>>
>> Once I found that file (in /usr/local/lib64/pkgconfig) and added the directory to the
>> config path, I was able to complete the configure and make without any further errors.
>
> Hm.. that way pkg-config couldn't find itself?
>
>
>> When I ran "make check", it failed on 11 of 12 tests.  I was able to fix most of the
>> tests by adding '/usr/local/lib64' to the LD_LIBRARY_PATH so it could find
>> libclamav.so.  I assume I'll need to make a similar change somewhere when I add the
>> filter to Courier.
>
> Or you might add it to ld.so.conf? Otherwise, you may try building with CFLAGS=-Wl,-rpath,/usr/local/lib64 and check ldd. The issue is the ability to have libraries of different versions simultaneously installed on the same system. See e.g.:
> https://unix.stackexchange.com/questions/356624/why-isnt-usr-local-lib-on-the-library-path-by-default

Adding it to ld.so.conf worked once I figured out I had to run ldconfig to load the
changes.

>
> At that point, the top of the header should be plenty of virus_header's (one for each invocation):
>
> ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ head eicar.mail
> ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
> Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
> Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
> Eicar-Test-Signature.UNOFFICIAL
> From: author@example.com
> To: victim@example.net
> Subject: test message
> Virus-Header: what does this mean?
> MIME-Version: 1.0
> Content-Type: multipart/mixed; boundary="=_1_1553193777_12188"

And here's the problem.  SecuriteInfo has their own Eicar signatures, so ClamAV found
those first and not the one you were expecting.  My header looks like this:

ClamAV-Found: SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL
  SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL
  SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL
  Eicar-Test-Signature.UNOFFICIAL
  SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL
  SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL
  SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL
  Eicar-Test-Signature.UNOFFICIAL

Not sure why everything is duplicated...

> And hence:
>
> ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ egrep '^ClamAV-Found: Eicar' eicar.mail | wc -l
> 1
>
> Instead, at yours wc wrote "0". Why?
>
>> 9. testsuite.at:540: 9. per-virus behavior (testsuite.at:540): FAILED (testsuite.at:612)
>>
>> Suggestions?
>>
> I'd guess something must have gone wrong in the testsuite script. In the testsuite, wc is $WC, after a definition in tests/atlocal, but egrep was not checked during configure, so maybe it should have been grep -E or similar. Is that the culprit?

No, egrep works fine once the regex is adjusted to match the header.

I guess I should have specified that I'm running this on CentOS 7, not that it
matters at this point.

Looks like everything is working now.  I'll try integrating it with Courier
tomorrow.  If I just want to reject any email that is flagged by ClamAV, I shouldn't
need to adjust the default config, right?

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Thu 21/Mar/2019 21:21:45 +0100 Bowie Bailey wrote:
>
>>
>> At that point, the top of the header should be plenty of virus_header's (one for each invocation):
>>
>> ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ head eicar.mail
>> ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
>> Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
>> Old-ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
>> Eicar-Test-Signature.UNOFFICIAL
>> From: author@example.com
>> To: victim@example.net
>> Subject: test message
>> Virus-Header: what does this mean?
>> MIME-Version: 1.0
>> Content-Type: multipart/mixed; boundary="=_1_1553193777_12188"
>
> And here's the problem.  SecuriteInfo has their own Eicar signatures, so ClamAV found
> those first and not the one you were expecting.  My header looks like this:
>
> ClamAV-Found: SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL
>   SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL
>   SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL
>   Eicar-Test-Signature.UNOFFICIAL
>   SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL
>   SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL
>   SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL
>   Eicar-Test-Signature.UNOFFICIAL


Uh, yeah, can be. The small database is made like so:

test.ndb:
sigtool -f Eicar-Test-Signature > $@

where the argument to -f is a regular expression. So the database seems to contain four matching signatures. Your database (like mine) probably has much more matching signatures, such as Win.Test.EICAR_NDB-1, whose names don't match the above regex.


> Not sure why everything is duplicated...
>

This is something we should ask to ClamAV developers. avtest.conf also contains allmatch (since it's useful when configuring different actions for different viruses). With --allmatch, clamscan duplicates (or triplicates) messages too, even if the small database contains a single virus:

ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ clamscan --allmatch -d ../../small eicar.mail
eicar.mail: Eicar-Test-Signature.UNOFFICIAL FOUND
eicar.mail: Eicar-Test-Signature.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 1
Engine version: 0.101.1
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.007 sec (0 m 0 s)


>> And hence:
>>
>> ale@pcale:~/tmp/courier/avfilter/svn/tests/testsuite.dir/09$ egrep '^ClamAV-Found: Eicar' eicar.mail | wc -l
>> 1
>>
>> Instead, at yours wc wrote "0". Why?
>>
>>> 9. testsuite.at:540: 9. per-virus behavior (testsuite.at:540): FAILED (testsuite.at:612)
>>>
>>> Suggestions?
>>>
>> I'd guess something must have gone wrong in the testsuite script. In the testsuite, wc is $WC, after a definition in tests/atlocal, but egrep was not checked during configure, so maybe it should have been grep -E or similar. Is that the culprit?
>
> No, egrep works fine once the regex is adjusted to match the header.


Fine. I'll change that command to "egrep -i '^ClamAV-Found: .*Eicar' eicar.mail".

Thank you for the fix.


> I guess I should have specified that I'm running this on CentOS 7, not that it
> matters at this point.
>
> Looks like everything is working now.  I'll try integrating it with Courier
> tomorrow.  If I just want to reject any email that is flagged by ClamAV, I shouldn't
> need to adjust the default config, right?


Correct, reject is the default. You probably need to set "database" to the same directory you configured as "DatabaseDirectory" in freshclam.conf. Also, recall that clamd.conf is not read; please see avfilter.conf(5) if you need to set clamav options.


Best
Ale
--






_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/22/2019 6:58 AM, Alessandro Vesely via clamav-users wrote:
> On Thu 21/Mar/2019 21:21:45 +0100 Bowie Bailey wrote:
>>
>> And here's the problem.  SecuriteInfo has their own Eicar signatures, so ClamAV found
>> those first and not the one you were expecting.  My header looks like this:
>>
>> ClamAV-Found: SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL
>>   SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL
>>   SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL
>>   Eicar-Test-Signature.UNOFFICIAL
>>   SecuriteInfo.com.Eicar-Test-Signature.UNOFFICIAL
>>   SecuriteInfo.com.Eicar-Test-Signature-4.UNOFFICIAL
>>   SecuriteInfo.com.Eicar-Test-Signature-2.UNOFFICIAL
>>   Eicar-Test-Signature.UNOFFICIAL
>
> Uh, yeah, can be. The small database is made like so:
>
> test.ndb:
> sigtool -f Eicar-Test-Signature > $@
>
> where the argument to -f is a regular expression. So the database seems to contain four matching signatures. Your database (like mine) probably has much more matching signatures, such as Win.Test.EICAR_NDB-1, whose names don't match the above regex.

Maybe you should limit that regex so it only matches the official ClamAV signature.

sigtool -f '^Eicar-Test-Signature' > $@

>
>> Not sure why everything is duplicated...
>>

After testing the sigtool command above, it looks like the duplication comes from
matching multiple types of files in the database directory.  daily.ndu, daily.ndb,
daily.ldu, and daily.ldb all matched an Eicar signature on my system.  I don't see a
way with sigtool to limit the match to a particular file type without having to
specify the directory.  Maybe you could just drop all but the first match from the
results.  (Although I guess it really doesn't matter since it is only used for the test)

Something like this:
sigtool -f '^Eicar-Test-Signature' | grep -m 1 . > $@

Actually, after testing that, I still get the header duplication with only 1
signature in the database.  Oh well...

ClamAV-Found: Eicar-Test-Signature.UNOFFICIAL Eicar-Test-Signature.UNOFFICIAL
  Eicar-Test-Signature.UNOFFICIAL

>> No, egrep works fine once the regex is adjusted to match the header.
>
> Fine. I'll change that command to "egrep -i '^ClamAV-Found: .*Eicar' eicar.mail".
>
> Thank you for the fix.

If you change how the test database is generated, then you can stick with the
original regex.

>
>> I guess I should have specified that I'm running this on CentOS 7, not that it
>> matters at this point.
>>
>> Looks like everything is working now.  I'll try integrating it with Courier
>> tomorrow.  If I just want to reject any email that is flagged by ClamAV, I shouldn't
>> need to adjust the default config, right?
>
> Correct, reject is the default. You probably need to set "database" to the same directory you configured as "DatabaseDirectory" in freshclam.conf. Also, recall that clamd.conf is not read; please see avfilter.conf(5) if you need to set clamav options.

The database option defaulted to the right location, so I didn't need to adjust
that.  I did need to go through the other clam options and adjust them to match what
I was using with clamd.  Oddly, the defaults in clamd aren't quite the same as the
defaults in clamscan.  Annoyingly, the option names aren't quite the same either...

Everything is working now.  I have avfilter installed and running on Courier with
pythonfilter now just doing the rate limiting.  The only problem I have found is that
since my databases take so long to load, avfilter_sig times out when reloading the
virus definitions.  Is there a way to increase the timeout for avfilter_sig?

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 3/22/2019 10:54 AM, Bowie Bailey wrote:
>
> The only problem I have found is that
> since my databases take so long to load, avfilter_sig times out when reloading the
> virus definitions.  Is there a way to increase the timeout for avfilter_sig?

Nevermind.  I updated the hardcoded value in avfilter_sig.c and recompiled.  Maybe
that could be added as an option for the next version?

$ avfilter_sig -t 300 -l reload virus

Sorry, my C programming skills aren't good enough to offer a patch.

--
Bowie

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml