Mailing List Archive

[clamav-users] looks like I have a problem too
Greetings;

I just reduced my freshclam fetch from 24 to 6 times a day. But I do see
some errors when it does try to update:
copy/paste, wordwrap off:
from freshclam.log:

Wed Mar 10 08:09:24 2021 -> Received signal: wake up
Wed Mar 10 08:09:24 2021 -> ClamAV update process started at Wed Mar 10 08:09:24 2021
Wed Mar 10 08:09:24 2021 -> daily database available for update (local version: 26103, remote version: 26104)
Wed Mar 10 08:09:24 2021 -> Testing
database: '/var/lib/clamav/tmp.9230f/clamav-e29c7bfba68291a21e41dbe83fb8c776.tmp-daily.cld' ...
Wed Mar 10 08:09:29 2021 -> WARNING: [LibClamAV] cli_tgzload: Invalid checksum for file daily.hsb
Wed Mar 10 08:09:29 2021 -> WARNING: [LibClamAV] Can't
load /var/lib/clamav/tmp.9230f/clamav-e29c7bfba68291a21e41dbe83fb8c776.tmp-daily.cld: Malformed database
Wed Mar 10 08:09:29 2021 -> ERROR: Failed to load new database: Malformed database
Wed Mar 10 08:09:29 2021 -> Database test passed.
Wed Mar 10 08:09:31 2021 -> daily.cld updated (version: 26104, sigs: 3958880, f-level: 63, builder: raynman)
Wed Mar 10 08:09:31 2021 -> main.cld database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
Wed Mar 10 08:09:31 2021 -> bytecode.cld database is up to date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
Wed Mar 10 08:09:31 2021 -> WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such
file or directory

So obviously something is aglay with my config which I haven't touched
since debian stretch was installed. But it has been kept uptodate at
least weekly. synaptic says I have version 102-4.

What should I fix?

Thanks folks.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] looks like I have a problem too [ In reply to ]
On Wednesday 10 March 2021 12:25:18 Gene Heskett via clamav-users wrote:

> Greetings;
>
> I just reduced my freshclam fetch from 24 to 6 times a day. But I do
> see some errors when it does try to update:
> copy/paste, wordwrap off:
> from freshclam.log:
>
> Wed Mar 10 08:09:24 2021 -> Received signal: wake up
> Wed Mar 10 08:09:24 2021 -> ClamAV update process started at Wed Mar
> 10 08:09:24 2021 Wed Mar 10 08:09:24 2021 -> daily database available
> for update (local version: 26103, remote version: 26104) Wed Mar 10
> 08:09:24 2021 -> Testing
> database:
> '/var/lib/clamav/tmp.9230f/clamav-e29c7bfba68291a21e41dbe83fb8c776.tmp
>-daily.cld' ... Wed Mar 10 08:09:29 2021 -> WARNING: [LibClamAV]
> cli_tgzload: Invalid checksum for file daily.hsb Wed Mar 10 08:09:29
> 2021 -> WARNING: [LibClamAV] Can't
> load
> /var/lib/clamav/tmp.9230f/clamav-e29c7bfba68291a21e41dbe83fb8c776.tmp-
>daily.cld: Malformed database Wed Mar 10 08:09:29 2021 -> ERROR: Failed
> to load new database: Malformed database Wed Mar 10 08:09:29 2021 ->
> Database test passed.
> Wed Mar 10 08:09:31 2021 -> daily.cld updated (version: 26104, sigs:
> 3958880, f-level: 63, builder: raynman) Wed Mar 10 08:09:31 2021 ->
> main.cld database is up to date (version: 59, sigs: 4564902, f-level:
> 60, builder: sigmgr) Wed Mar 10 08:09:31 2021 -> bytecode.cld database
> is up to date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
> Wed Mar 10 08:09:31 2021 -> WARNING: Clamd was NOT notified: Can't
> connect to clamd through /var/run/clamav/clamd.ctl: No such file or
> directory
>
> So obviously something is aglay with my config which I haven't touched
> since debian stretch was installed. But it has been kept uptodate at
> least weekly. synaptic says I have version 102-4.
>
> What should I fix?
>
> Thanks folks.
>
> Cheers, Gene Heskett

Maybe its fixed, maybe not, I ran dpkg-reconfigure on it and selected
unix sockets. then restarted both, but there is still
no /var/run/clamav/clamd.ctl file. But procmail is using clamdscan, not
clamd. And its silently diverting stuff to /var/mail/virii occasionally.
I *think* that's a success clue.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] looks like I have a problem too [ In reply to ]
I wrote a little script that run off cron every hour or so. But it *only* invokes freshclam after querying ClamAV's DNS TXT record to see if any advertised versions of 'daily', 'bytecode' or 'main' are newer than the local versions of the CVD files, as determined by 'head', not the files' timestamps. Only if something newer is supposed to be available (Cloudflare BOS mirror had some problems in the past), does it invoke freshclam. (It also keeps backup CVD/CLD files just in case.)

This provides a very low bandwidth way of getting ClamAV updates ASAP. (I even cron my different ClamAV machines at different odd times during the day.)

If anyone is interested, I could provide my scripts: 'getfreshclam' (Bash) which uses 'testclam-dns' (Perl).



On Wed, 10 Mar 2021 12:25:18 -0500
Gene Heskett via clamav-users <clamav-users@lists.clamav.net> wrote:

> Greetings;
>
> I just reduced my freshclam fetch from 24 to 6 times a day. But I do see
> some errors when it does try to update:
> copy/paste, wordwrap off:
> from freshclam.log:
>
> Wed Mar 10 08:09:24 2021 -> Received signal: wake up
> Wed Mar 10 08:09:24 2021 -> ClamAV update process started at Wed Mar 10 08:09:24 2021
> Wed Mar 10 08:09:24 2021 -> daily database available for update (local version: 26103, remote version: 26104)
> Wed Mar 10 08:09:24 2021 -> Testing
> database: '/var/lib/clamav/tmp.9230f/clamav-e29c7bfba68291a21e41dbe83fb8c776.tmp-daily.cld' ...
> Wed Mar 10 08:09:29 2021 -> WARNING: [LibClamAV] cli_tgzload: Invalid checksum for file daily.hsb
> Wed Mar 10 08:09:29 2021 -> WARNING: [LibClamAV] Can't
> load /var/lib/clamav/tmp.9230f/clamav-e29c7bfba68291a21e41dbe83fb8c776.tmp-daily.cld: Malformed database
> Wed Mar 10 08:09:29 2021 -> ERROR: Failed to load new database: Malformed database
> Wed Mar 10 08:09:29 2021 -> Database test passed.
> Wed Mar 10 08:09:31 2021 -> daily.cld updated (version: 26104, sigs: 3958880, f-level: 63, builder: raynman)
> Wed Mar 10 08:09:31 2021 -> main.cld database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
> Wed Mar 10 08:09:31 2021 -> bytecode.cld database is up to date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
> Wed Mar 10 08:09:31 2021 -> WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such
> file or directory
>
> So obviously something is aglay with my config which I haven't touched
> since debian stretch was installed. But it has been kept uptodate at
> least weekly. synaptic says I have version 102-4.
>
> What should I fix?
>
> Thanks folks.
>
> Cheers, Gene Heskett

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] looks like I have a problem too [ In reply to ]
On 10/03/2021 20:29, Paul Kosinski via clamav-users wrote:
> I wrote a little script that run off cron every hour or so. But it *only* invokes freshclam after querying ClamAV's DNS TXT record to see if any advertised versions of 'daily', 'bytecode' or 'main' are newer than the local versions of the CVD files

As I understand it, Freshclam already won't do anything if the DNS
record shows the same versions as the locally available CVD files, so
you don't need to do that.

That's certainly how it seems to behave here. If the DNS record hasn't
changed, then it just says "everything's fine" and does nothing else.
So, if you ran Freshclam every minute, it wouldn't download anything
except lots of DNS queries (which would be cached more locally).

The bandwidth problem is due to people NOT using Freshclam at all.

--
Paul


--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] looks like I have a problem too [ In reply to ]
Citeren Paul Smith via clamav-users <clamav-users@lists.clamav.net>:

> That's certainly how it seems to behave here. If the DNS record
> hasn't changed, then it just says "everything's fine" and does
> nothing else. So, if you ran Freshclam every minute, it wouldn't
> download anything except lots of DNS queries (which would be cached
> more locally).

It reads the TXT record for 'current.cvd.clamav.net' which has a TTL
of 1800 seconds. Running freshclam more frequent than once every
thirty minutes is therefor in most cases just wasting CPU cycles, as
the previous answer will be used then. Even if the answer isn't cached
locally, chances are it will be cached somewhere upstream.




_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] looks like I have a problem too [ In reply to ]
> On Mar 10, 2021, at 3:58 PM, Arjen de Korte via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Citeren Paul Smith via clamav-users <clamav-users@lists.clamav.net>:
>
>> That's certainly how it seems to behave here. If the DNS record hasn't changed, then it just says "everything's fine" and does nothing else. So, if you ran Freshclam every minute, it wouldn't download anything except lots of DNS queries (which would be cached more locally).
>
> It reads the TXT record for 'current.cvd.clamav.net' which has a TTL of 1800 seconds. Running freshclam more frequent than once every thirty minutes is therefor in most cases just wasting CPU cycles, as the previous answer will be used then. Even if the answer isn't cached locally, chances are it will be cached somewhere upstream.

Right. Run it once an hour and you’re good.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] looks like I have a problem too [ In reply to ]
On Mar 10, 2021, at 3:29 PM, Paul Kosinski via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

I wrote a little script that run off cron every hour or so. But it *only* invokes freshclam after querying ClamAV's DNS TXT record to see if any advertised versions of 'daily', 'bytecode' or 'main' are newer than the local versions of the CVD files, as determined by 'head', not the files' timestamps. Only if something newer is supposed to be available (Cloudflare BOS mirror had some problems in the past), does it invoke freshclam. (It also keeps backup CVD/CLD files just in case.)

This provides a very low bandwidth way of getting ClamAV updates ASAP. (I even cron my different ClamAV machines at different odd times during the day.)

If anyone is interested, I could provide my scripts: 'getfreshclam' (Bash) which uses 'testclam-dns' (Perl).

As someone else said, Freshclam does this for you.
Re: [clamav-users] looks like I have a problem too [ In reply to ]
Hi Gene,

Regarding the errors you're observing:

I don't know why the daily.cld check is failing with:
WARNING: [LibClamAV] cli_tgzload: Invalid checksum for file daily.hsb

My guess is that there was a truncated download but that Freshclam didn't realize it (see below).

The second issue I see is that database verification failed, but then it claimed the database test passed. This is a knowns issue that we fixed recently, here:
- ticket: https://bugzilla.clamav.net/show_bug.cgi?id=12522
- commit: https://github.com/Cisco-Talos/clamav-devel/commit/ade9352d9a168f3560f28da806b48e010ff009b7

I believe this issue was fixed in 0.103.1: https://blog.clamav.net/2021/02/clamav-01031-patch-release.html

This does raise a point that we could improve the error handling in freshclam to try to not only verify the database with a load test, but also first verify that the content-length in the HTTP headers matches the size of the downloaded files. I'll make a ticket to add this extra check just in case users have database verification disabled (or the verification is broken again in some unexpected way).

Sorry for all the trouble!

Regards,
-Micah

> -----Original Message-----
> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> Gene Heskett via clamav-users
> Sent: Wednesday, March 10, 2021 9:25 AM
> To: clamav-users@lists.clamav.net
> Cc: Gene Heskett <gheskett@shentel.net>
> Subject: [clamav-users] looks like I have a problem too
>
> Greetings;
>
> I just reduced my freshclam fetch from 24 to 6 times a day. But I do see some
> errors when it does try to update:
> copy/paste, wordwrap off:
> from freshclam.log:
>
> Wed Mar 10 08:09:24 2021 -> Received signal: wake up Wed Mar 10 08:09:24
> 2021 -> ClamAV update process started at Wed Mar 10 08:09:24 2021 Wed
> Mar 10 08:09:24 2021 -> daily database available for update (local version:
> 26103, remote version: 26104) Wed Mar 10 08:09:24 2021 -> Testing
> database: '/var/lib/clamav/tmp.9230f/clamav-
> e29c7bfba68291a21e41dbe83fb8c776.tmp-daily.cld' ...
> Wed Mar 10 08:09:29 2021 -> WARNING: [LibClamAV] cli_tgzload: Invalid
> checksum for file daily.hsb Wed Mar 10 08:09:29 2021 -> WARNING:
> [LibClamAV] Can't load /var/lib/clamav/tmp.9230f/clamav-
> e29c7bfba68291a21e41dbe83fb8c776.tmp-daily.cld: Malformed database
> Wed Mar 10 08:09:29 2021 -> ERROR: Failed to load new database: Malformed
> database Wed Mar 10 08:09:29 2021 -> Database test passed.
> Wed Mar 10 08:09:31 2021 -> daily.cld updated (version: 26104, sigs:
> 3958880, f-level: 63, builder: raynman) Wed Mar 10 08:09:31 2021 -> main.cld
> database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
> Wed Mar 10 08:09:31 2021 -> bytecode.cld database is up to date (version:
> 333, sigs: 92, f-level: 63, builder: awillia2) Wed Mar 10 08:09:31 2021 ->
> WARNING: Clamd was NOT notified: Can't connect to clamd through
> /var/run/clamav/clamd.ctl: No such file or directory
>
> So obviously something is aglay with my config which I haven't touched since
> debian stretch was installed. But it has been kept uptodate at least weekly.
> synaptic says I have version 102-4.
>
> What should I fix?
>
> Thanks folks.
>
> Cheers, Gene Heskett
> --
> "There are four boxes to be used in defense of liberty:
> soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> If we desire respect for the law, we must first make the law respectable.
> - Louis D. Brandeis
> Genes Web page <http://geneslinuxbox.net:6309/gene>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] looks like I have a problem too [ In reply to ]
On Tuesday 16 March 2021 15:29:04 Micah Snyder (micasnyd) wrote:

> Hi Gene,
>
> Regarding the errors you're observing:
>
> I don't know why the daily.cld check is failing with:
> WARNING: [LibClamAV] cli_tgzload: Invalid checksum for file
> daily.hsb
>
> My guess is that there was a truncated download but that Freshclam
> didn't realize it (see below).
>
> The second issue I see is that database verification failed, but then
> it claimed the database test passed. This is a knowns issue that we
> fixed recently, here: - ticket:
> https://bugzilla.clamav.net/show_bug.cgi?id=12522
> - commit:
> https://github.com/Cisco-Talos/clamav-devel/commit/ade9352d9a168f3560f
>28da806b48e010ff009b7
>
> I believe this issue was fixed in 0.103.1:
> https://blog.clamav.net/2021/02/clamav-01031-patch-release.html
>
> This does raise a point that we could improve the error handling in
> freshclam to try to not only verify the database with a load test, but
> also first verify that the content-length in the HTTP headers matches
> the size of the downloaded files. I'll make a ticket to add this
> extra check just in case users have database verification disabled (or
> the verification is broken again in some unexpected way).
>
> Sorry for all the trouble!
>
Well, I'm not sure why it errored, it has not repeated, ahh, logrotate
killed my tail on the 13th. What its doing is showing that message
everytime it increments the database serial number, but on the next
wakeup, the new version is used and it goes on its merry way until the
next number is released. Currently on 26110. So I'm inclined to think
the error is a corner case in handling the update of daily.cvd.

The reason I didn't see it before is theres so much bs being fed to the
syslog from stuff which did, a year ago, have its own log, so
freshclam's little 6 line thingy was lost in the noise. Something is
restarting cups quite a few times a day, and that is 30+ lines of
useless noise spamming the syslog. With 6 machines all capable of
hitting that printer, and they all have to reestablish connections, the
log is busier than that famous cat on the equally famous tin roof.

Many thanks for the reply Micah. Take care and stay safe and well.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml