Mailing List Archive

Hi there,

On Tue, 8 Dec 2020, Gal Cohen wrote:

> I'm serving cvd files from a local server, when I run freshclam on my
> server it takes some runes until the daily.cvd is updated even though the
> remote version was updated a while ago.

Please clarify that the daily.cvd file which you are talking about is
the one on the "local server" which serves the .cvd file, and that you
are not talking about .cvd files on the clients of your local server.

Please explain exactly how you installed ClamAV on the local server.

> ...
> DatabaseOwner root
> ...

Bad idea, but probably not related to your problem.

> "daily database available for update (local version: 26009, remote version: 26010)
> *The daily.cvd database downloaded from https://database.clamav.net is one version older than advertised in the DNS TXT record.
> Database test passed.
> daily.cvd updated (version: 26009, sigs: 4351133, f-level: 63, builder: raynman)"

Please include timestamps on the logs next time. Here's my freshclam
log for the update to version 26009. It took about 34 seconds total
and it isn't a very fast box (a Raspberry Pi 4B). As you see I have
daily.cld not daily.cvd, I wonder if it might make a difference.

Sat Dec 5 18:15:49 2020 -> Received signal: wake up
Sat Dec 5 18:15:49 2020 -> ClamAV update process started at Sat Dec 5 18:15:49 2020
Sat Dec 5 18:15:50 2020 -> daily database available for update (local version: 26008, remote version: 26009)
Sat Dec 5 18:15:57 2020 -> Testing database: '/EXPORTS/clamav/databases/tmp.06d3200a0e/clamav-223cf5c8f023bca440730a45c874e079.tmp-daily.cld' ...
Sat Dec 5 18:16:23 2020 -> Database test passed.
Sat Dec 5 18:16:23 2020 -> daily.cld updated (version: 26009, sigs: 4351133, f-level: 63, builder: raynman)
Sat Dec 5 18:16:23 2020 -> main.cvd database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
Sat Dec 5 18:16:23 2020 -> bytecode.cvd database is up to date (version: 331, sigs: 94, f-level: 63, builder: anvilleg)

I have never seen freshclam download an older file than the one which
is claimed in the DNS to be the latest. How often are you checking
for updates?

> Do I need to change my configuration or is it a bug on the 102.4 clamav version?

There will be other possible explanations. I don't see how a fault or
a misconfiguration at the client end might cause the remote server to
serve an out of date database file but perhaps you can let us see the
output of 'clamconf' for completeness anyway.

If one of the ClamAV mirrors is serving outdated files then I'm sure
that the ClamAV team will want to know about it. Please give full log
details including timestamps and the IP address(es) from which the
.cvd files were downloaded. Without more information it's difficult
to know what the problem might be, it might help if you investigate
with something like Wireshark. I wonder if your DNS setup might need
some work but that's just a guess.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi all,

1. the daily.cvd file I referring to is on the local server which acts as
the private local mirror (not referring to clients at all)

2. The freshclam is running in a docker , his image came from `alpine:3.12`
and clamav was install with that command: `apk add --no-cache
clamav=0.102.4-r11 clamav-libunrar=0.102.4-r11`

3. I checking the updates twice a day

4. it's not that straightforward to run wireshark on that server, but i can
route it to a specific dns record (will update)

5. here are the full logs of the latest update failure (26011 ->
26012),freshclam run takes 19 sec
Tue Dec 8 22:00:02 2020 -> ClamAV update process started at Tue Dec 8
22:00:02 2020
Tue Dec 8 22:00:02 2020 -> *Current working dir is /data/
Tue Dec 8 22:00:02 2020 -> *Querying current.cvd.clamav.net
Tue Dec 8 22:00:02 2020 -> *TTL: 30
Tue Dec 8 22:00:02 2020 -> *fc_dns_query_update_info: Software version
from DNS: 0.103.0
Tue Dec 8 22:00:02 2020 -> *Current working dir is /data/
Tue Dec 8 22:00:02 2020 -> *check_for_new_database_version: Local copy of
daily found: daily.cvd.
Tue Dec 8 22:00:02 2020 -> *query_remote_database_version: daily.cvd
version from DNS: 26012
Tue Dec 8 22:00:02 2020 -> daily database available for update (local
version: 26011, remote version: 26012)
Tue Dec 8 22:00:02 2020 -> *Retrieving
https://database.clamav.net/daily.cvd
Tue Dec 8 22:00:02 2020 -> *downloadFile: Download source:
https://database.clamav.net/daily.cvd
Tue Dec 8 22:00:02 2020 -> *downloadFile: Download destination:
/data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp
* Trying 104.16.218.84:443...
* Connected to database.clamav.net (104.16.218.84) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=
sni.cloudflaressl.com
* start date: Aug 15 00:00:00 2020 GMT
* expire date: Aug 15 12:00:00 2021 GMT
* subjectAltName: host "database.clamav.net" matched cert's "
database.clamav.net"
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade:
len=0
* Using Stream ID: 1 (easy handle 0x56459985de60)
> GET /daily.cvd HTTP/2
Host: database.clamav.net
user-agent: ClamAV/0.102.4 (OS: linux-musl, ARCH: x86_64, CPU: x86_64)
accept: */*
connection: close

* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Tue, 08 Dec 2020 22:00:02 GMT
< content-type: application/octet-stream
< content-length: 114885026
< set-cookie: __cfduid=dc7afe2099393f2517fefc5bfc70645881607464802;
expires=Thu, 07-Jan-21 22:00:02 GMT; path=/; domain=.clamav.net; HttpOnly;
SameSite=Lax
< last-modified: Mon, 07 Dec 2020 14:37:00 GMT
< etag: "5fce3e0c-6d901a2"
< expires: Wed, 09 Dec 2020 10:00:02 GMT
< cache-control: public, max-age=43200
< cf-cache-status: HIT
< age: 109
< accept-ranges: bytes
< cf-request-id: 06e5f76fd70000dfa591a49000000001
< expect-ct: max-age=604800, report-uri="
https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 5fe9c1c62d72dfa5-FRA
<
* Connection #0 to host database.clamav.net left intact
Tue Dec 8 22:00:05 2020 -> *The daily.cvd database downloaded from
https://database.clamav.net is one version older than advertised in the DNS
TXT record.
Tue Dec 8 22:00:05 2020 -> *updatedb: Running g_cb_download_complete
callback...
Tue Dec 8 22:00:05 2020 -> *download_complete_callback: Download complete
for database :
/data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp-daily.cvd
Tue Dec 8 22:00:05 2020 -> *download_complete_callback:
fc_context->bTestDatabases : 1
Tue Dec 8 22:00:05 2020 -> *download_complete_callback:
fc_context->bBytecodeEnabled : 1
Tue Dec 8 22:00:05 2020 -> Testing database:
'/data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp-daily.cvd' ...
Tue Dec 8 22:00:05 2020 -> *Loading signatures from
/data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp-daily.cvd
Tue Dec 8 22:00:20 2020 -> *Properly loaded 4397905 signatures from
/data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp-daily.cvd
Tue Dec 8 22:00:21 2020 -> Database test passed.
Tue Dec 8 22:00:21 2020 -> daily.cvd updated (version: 26011, sigs:
4351421, f-level: 63, builder: raynman)
Tue Dec 8 22:00:21 2020 -> *fc_update_database: daily.cvd updated.
Tue Dec 8 22:00:21 2020 -> *Current working dir is /data/
Tue Dec 8 22:00:21 2020 -> *check_for_new_database_version: Local copy of
main found: main.cvd.
Tue Dec 8 22:00:21 2020 -> *query_remote_database_version: main.cvd
version from DNS: 59
Tue Dec 8 22:00:21 2020 -> main.cvd database is up to date (version: 59,
sigs: 4564902, f-level: 60, builder: sigmgr)
Tue Dec 8 22:00:21 2020 -> *fc_update_database: main.cvd already
up-to-date.
Tue Dec 8 22:00:21 2020 -> *Current working dir is /data/
Tue Dec 8 22:00:21 2020 -> *check_for_new_database_version: Local copy of
bytecode found: bytecode.cvd.
Tue Dec 8 22:00:21 2020 -> *query_remote_database_version: bytecode.cvd
version from DNS: 331
Tue Dec 8 22:00:21 2020 -> bytecode.cvd database is up to date (version:
331, sigs: 94, f-level: 63, builder: anvilleg)
Tue Dec 8 22:00:21 2020 -> *fc_update_database: bytecode.cvd already
up-to-date.
Tue Dec 8 22:00:21 2020 -> *Current working dir is /data/
Tue Dec 8 22:00:21 2020 -> *check_for_new_database_version: Local copy of
safebrowsing found: safebrowsing.cvd.
Tue Dec 8 22:00:21 2020 -> *query_remote_database_version:
safebrowsing.cvd version from DNS: 49191
Tue Dec 8 22:00:21 2020 -> safebrowsing.cvd database is up to date
(version: 49191, sigs: 2213119, f-level: 63, builder: google)
Tue Dec 8 22:00:21 2020 -> *fc_update_database: safebrowsing.cvd already
up-to-date.


On Tue, Dec 8, 2020 at 8:01 PM Gal Cohen <gal.cohen@zooz.com> wrote:

> Hello,
>
> I'm serving cvd files from a local server, when I run freshclam on my
> server it takes some runes until the daily.cvd is updated even though the
> remote version was updated a while ago.
>
> - the clamav version I'm using is 0.102.4-r1
> - freshclam.conf I'm using is:
> DatabaseDirectory /data
> LogSyslog yes
> UpdateLogFile /logs/freshclam.log
> LogTime yes
> PidFile /run/clamav/freshclam.pid
> DatabaseOwner root
> LogVerbose yes
> DatabaseMirror database.clamav.net
> ScriptedUpdates no. (for serving as local server)
> SafeBrowsing yes
> Bytecode yes
>
> some focused logs from freshclam run which not update the local daily.cvd
> even though it indicates a newer version remotely:
> "daily database available for update (local version: 26009, remote
> version: 26010)
> *The daily.cvd database downloaded from https://database.clamav.net is
> one version older than advertised in the DNS TXT record.
> Database test passed.
> daily.cvd updated (version: 26009, sigs: 4351133, f-level: 63, builder:
> raynman)"
>
> Do I need to change my configuration or is it a bug on the 102.4 clamav
> version?
>
> Thanks
> Gal
>

Hi there,

On Wed, 9 Dec 2020, Gal Cohen wrote:

> 5. here are the full logs of the latest update failure (26011 -> 26012),freshclam run takes 19 sec
> Tue Dec 8 22:00:02 2020 -> ClamAV update process started at Tue Dec 8 22:00:02 2020
> ...
> Tue Dec 8 22:00:02 2020 -> *check_for_new_database_version: Local copy of daily found: daily.cvd.
> Tue Dec 8 22:00:02 2020 -> *query_remote_database_version: daily.cvd version from DNS: 26012
> Tue Dec 8 22:00:02 2020 -> daily database available for update (local version: 26011, remote version: 26012)
> Tue Dec 8 22:00:02 2020 -> *Retrieving https://database.clamav.net/daily.cvd
> Tue Dec 8 22:00:02 2020 -> *downloadFile: Download source: https://database.clamav.net/daily.cvd
> Tue Dec 8 22:00:02 2020 -> *downloadFile: Download destination: /data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp
> * Trying 104.16.218.84:443...

This is one of the IPs which I was expecting to see. I wouldn't expect any
problems with it, our ClamAV server updated from it at 1818 GMT last night.

Maybe you have a proxy between you and the Cloudflare servers which is caching
the data downloads? Try downloading the 'daily' file with 'wget' from several
different places and check which versions you receive.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

The same problem happens to me now when trying to update from 26012
to 26013 version
I was trying to wget directly from each IP (104.16.218.84, 104.16.219.8)
and still I'm getting 26012 version from them.
maybe this relates to the fact `scriptUpdates no`? even though the server
should be configure that way base on clamav's docs

On Wed, Dec 9, 2020 at 1:13 PM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Wed, 9 Dec 2020, Gal Cohen wrote:
>
> > 5. here are the full logs of the latest update failure (26011 ->
> 26012),freshclam run takes 19 sec
> > Tue Dec 8 22:00:02 2020 -> ClamAV update process started at Tue Dec 8
> 22:00:02 2020
> > ...
> > Tue Dec 8 22:00:02 2020 -> *check_for_new_database_version: Local copy
> of daily found: daily.cvd.
> > Tue Dec 8 22:00:02 2020 -> *query_remote_database_version: daily.cvd
> version from DNS: 26012
> > Tue Dec 8 22:00:02 2020 -> daily database available for update (local
> version: 26011, remote version: 26012)
> > Tue Dec 8 22:00:02 2020 -> *Retrieving
> https://database.clamav.net/daily.cvd
> > Tue Dec 8 22:00:02 2020 -> *downloadFile: Download source:
> https://database.clamav.net/daily.cvd
> > Tue Dec 8 22:00:02 2020 -> *downloadFile: Download destination:
> /data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp
> > * Trying 104.16.218.84:443...
>
> This is one of the IPs which I was expecting to see. I wouldn't expect any
> problems with it, our ClamAV server updated from it at 1818 GMT last night.
>
> Maybe you have a proxy between you and the Cloudflare servers which is
> caching
> the data downloads? Try downloading the 'daily' file with 'wget' from
> several
> different places and check which versions you receive.
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

"This is one of the IPs which I was expecting to see. I wouldn't
expect any problems with it, our ClamAV server updated from it at
1818 GMT last night."

Unfortunately, given the way Cloudflare works, the IP address
(e.g., 104.16.218.84) isn't the whole story. A particular Anycast IP
address such as this will route to the "nearest" server for that IP
address, and different servers may behave differently.

The HTTP(S) response header indicates which of the Cloudflare
servers the IP address actually routed to, for example:

CF-RAY: 433942cde659ae1a-BOS

But I think you have to pretend you are ClamAV, or the server rejects
you, as in:

User-Agent: ClamAV/0.103.0 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

(At least this is the way it was in 2018.)

In the summer of 2018 (just after ClamAV started using Cloudflare) we
were having trouble in that our local BOS server was often behind the
latest ClamAV CVD file which was advertised by the DNS TXT record. I
finally gave up trying to have a local mirror for CVD files, and just
changed all our ClamAV machines to use the "scripted update" (CDIFF)
method individually. There are so few machines that it turned out to
*save* bandwidth in practice.

P.S. There are a lot of emails about this in the ClamAV list for July
2018 et seq with subject lines: "We STILL cannot reliably get virus
updates (since new mirrors)".





On Wed, 9 Dec 2020 11:12:28 +0000 (GMT)
"G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Wed, 9 Dec 2020, Gal Cohen wrote:
>
> > 5. here are the full logs of the latest update failure (26011 -> 26012),freshclam run takes 19 sec
> > Tue Dec 8 22:00:02 2020 -> ClamAV update process started at Tue Dec 8 22:00:02 2020
> > ...
> > Tue Dec 8 22:00:02 2020 -> *check_for_new_database_version: Local copy of daily found: daily.cvd.
> > Tue Dec 8 22:00:02 2020 -> *query_remote_database_version: daily.cvd version from DNS: 26012
> > Tue Dec 8 22:00:02 2020 -> daily database available for update (local version: 26011, remote version: 26012)
> > Tue Dec 8 22:00:02 2020 -> *Retrieving https://database.clamav.net/daily.cvd
> > Tue Dec 8 22:00:02 2020 -> *downloadFile: Download source: https://database.clamav.net/daily.cvd
> > Tue Dec 8 22:00:02 2020 -> *downloadFile: Download destination: /data/tmp.7624b/clamav-cde3734f56b3b9351a0261c3b140966f.tmp
> > * Trying 104.16.218.84:443...
>
> This is one of the IPs which I was expecting to see. I wouldn't expect any
> problems with it, our ClamAV server updated from it at 1818 GMT last night.
>
> Maybe you have a proxy between you and the Cloudflare servers which is caching
> the data downloads? Try downloading the 'daily' file with 'wget' from several
> different places and check which versions you receive.
>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hey Gal,

This message: “*The daily.cvd database downloaded from https://database.clamav.net is one version older than advertised in the DNS TXT record.” is a verbose-level message from freshclam but is not an error to worry about.
It indicates a known issue with CloudFlare caching that our team has been unable to resolve. You can safely ignore the message. Update again in an hour or two and you should get the latest database.

Regards,
Micah


From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of Gal Cohen
Sent: Tuesday, December 8, 2020 10:02 AM
To: clamav-users@lists.clamav.net
Subject: [clamav-users] local server takes time to update clamav db

Hello,

I'm serving cvd files from a local server, when I run freshclam on my server it takes some runes until the daily.cvd is updated even though the remote version was updated a while ago.

- the clamav version I'm using is 0.102.4-r1
- freshclam.conf I'm using is:
DatabaseDirectory /data
LogSyslog yes
UpdateLogFile /logs/freshclam.log
LogTime yes
PidFile /run/clamav/freshclam.pid
DatabaseOwner root
LogVerbose yes
DatabaseMirror database.clamav.net<http://database.clamav.net>
ScriptedUpdates no. (for serving as local server)
SafeBrowsing yes
Bytecode yes

some focused logs from freshclam run which not update the local daily.cvd even though it indicates a newer version remotely:
"daily database available for update (local version: 26009, remote version: 26010)
*The daily.cvd database downloaded from https://database.clamav.net is one version older than advertised in the DNS TXT record.
Database test passed.
daily.cvd updated (version: 26009, sigs: 4351133, f-level: 63, builder: raynman)"

Do I need to change my configuration or is it a bug on the 102.4 clamav version?

Thanks
Gal

Hi there,

On Wed, 9 Dec 2020, Micah Snyder (micasnyd) via clamav-users wrote:

> This message:
>
> “*The daily.cvd database downloaded from https://database.clamav.net
> is one version older than advertised in the DNS TXT record.”
>
> is a verbose-level message from freshclam but is not an error to
> worry about. It indicates a known issue with CloudFlare caching
> that our team has been unable to resolve. You can safely ignore the
> message. Update again in an hour or two and you should get the
> latest database.

But...but...I thought Joel said that this was fixed in February! [*]

Shouldn't this be documented?

At least in the error message itself, and preferably the manual and Bugzilla.

[*] https://marc.info/?l=clamav-announce&m=158092409813720&w=2

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> On Dec 10, 2020, at 6:06 AM, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Hi there,
>
> On Wed, 9 Dec 2020, Micah Snyder (micasnyd) via clamav-users wrote:
>
>> This message:
>> “*The daily.cvd database downloaded from https://database.clamav.net
>> is one version older than advertised in the DNS TXT record.”
>> is a verbose-level message from freshclam but is not an error to
>> worry about. It indicates a known issue with CloudFlare caching
>> that our team has been unable to resolve. You can safely ignore the
>> message. Update again in an hour or two and you should get the
>> latest database.
>
> But...but...I thought Joel said that this was fixed in February! [*]
>
> Shouldn't this be documented?
>
> At least in the error message itself, and preferably the manual and Bugzilla.
>
> [*] https://marc.info/?l=clamav-announce&m=158092409813720&w=2
>

By “unable to resolve” Micah means: “There’s nothing more we can do to solve the problem”.

When we generate a new cvd/cld, etc, we push it to our local mirrors. We then force Cloudflare to fetch the files through purging the cache for the files updated, and then requesting them through cloudflare (which causes the CF PoP (Point of Presence) to fetch the file from our mirror). Then we update the TXT record in DNS.

So, there are occasions where one PoP from Cloudflare is behind and hasn’t yet fetched the file from the other PoP or from our mirror directly. This might be the case that you’re the first one that’s asked for it from your PoP location in the world, or its just not caught up yet. Waiting a bit ensures that the PoP will fetch the file, and when you try again, it’s there (as you can see).

I think the way to fix this is, freshclam, if it receives an “I’m behind” error from the PoP, to do a sleep for awhile and then try again. If the second attempt still fails then give the error to the user.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com | https://www.snort.org

Date: Thu, 10 Dec 2020 14:07:08 +0000 (GMT)
From: Andrew C Aitchison <andrew@aitchison.me.uk>
To: clamav-users@lists.clamav.net
Cc: "Joel Esler (jesler)" <jesler@cisco.com>
Subject: Re: [clamav-users] local server takes time to update clamav db

On Thu, 10 Dec 2020, Joel Esler (jesler) via clamav-users wrote:

>> On Dec 10, 2020, at 6:06 AM, G.W. Haywood via clamav-users
>> <clamav-users@lists.clamav.net> wrote:
>>
>> Hi there,
>>
>> On Wed, 9 Dec 2020, Micah Snyder (micasnyd) via clamav-users wrote:
>>
>>> This message:
>>> ???*The daily.cvd database downloaded from https://database.clamav.net
>>> is one version older than advertised in the DNS TXT record.???
>>> is a verbose-level message from freshclam but is not an error to
>>> worry about. It indicates a known issue with CloudFlare caching
>>> that our team has been unable to resolve. You can safely ignore the
>>> message. Update again in an hour or two and you should get the
>>> latest database.
>>
>> But...but...I thought Joel said that this was fixed in February! [*]
>>
>> Shouldn't this be documented?
>>
>> At least in the error message itself, and preferably the manual and
>> Bugzilla.
>>
>> [*] https://marc.info/?l=clamav-announce&m=158092409813720&w=2
>>
>
> By ???unable to resolve??? Micah means: ???There???s nothing more we can do
> to solve the problem???.
>
> When we generate a new cvd/cld, etc, we push it to our local
> mirrors. We then force Cloudflare to fetch the files through
> purging the cache for the files updated, and then requesting them
> through cloudflare (which causes the CF PoP (Point of Presence) to
> fetch the file from our mirror). Then we update the TXT record in
> DNS.

Would you be able to request them from multiple cloudflare PoPs
before updating the DNS record ?
Not necessarily waiting for the update, but making the request.

> So, there are occasions where one PoP from Cloudflare is behind and
> hasn???t yet fetched the file from the other PoP or from our mirror
> directly. This might be the case that you???re the first one that???s
> asked for it from your PoP location in the world, or its just not
> caught up yet. Waiting a bit ensures that the PoP will fetch the
> file, and when you try again, it???s there (as you can see).
>
> I think the way to fix this is, freshclam, if it receives an ???I???m
> behind??? error from the PoP, to do a sleep for awhile and then try
> again. If the second attempt still fails then give the error to the
> user.

Would it be sensible for freshclam to update the file when a newer
version is available, even if it is not the newest ?

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> On Dec 10, 2020, at 9:07 AM, Andrew C Aitchison <andrew@aitchison.me.uk> wrote:
>
> On Thu, 10 Dec 2020, Joel Esler (jesler) via clamav-users wrote:
>
>>> <snip>
>>>
>>
>> By â??unable to resolveâ? Micah means: â??Thereâ??s nothing more we can do
>> to solve the problem�.
>>
>> When we generate a new cvd/cld, etc, we push it to our local
>> mirrors. We then force Cloudflare to fetch the files through
>> purging the cache for the files updated, and then requesting them
>> through cloudflare (which causes the CF PoP (Point of Presence) to
>> fetch the file from our mirror). Then we update the TXT record in
>> DNS.
>
> Would you be able to request them from multiple cloudflare PoPs
> before updating the DNS record ?
> Not necessarily waiting for the update, but making the request.

Possibly. But only in places around the world where we have a datacenter we can do it from, for obvious reasons.

>
>> So, there are occasions where one PoP from Cloudflare is behind and
>> hasnâ??t yet fetched the file from the other PoP or from our mirror
>> directly. This might be the case that youâ??re the first one thatâ??s
>> asked for it from your PoP location in the world, or its just not
>> caught up yet. Waiting a bit ensures that the PoP will fetch the
>> file, and when you try again, itâ??s there (as you can see).
>>
>> I think the way to fix this is, freshclam, if it receives an â??Iâ??m
>> behind� error from the PoP, to do a sleep for awhile and then try
>> again. If the second attempt still fails then give the error to the
>> user.
>
> Would it be sensible for freshclam to update the file when a newer
> version is available, even if it is not the newest ?

Not sure what you mean here?

> On Thu, 10 Dec 2020, Joel Esler (jesler) via clamav-users wrote:
>> So, there are occasions where one PoP from Cloudflare is behind and
>> hasn't yet fetched the file from the other PoP or from our mirror
>> directly. This might be the case that you're the first one that's
>> asked for it from your PoP location in the world, or its just not
>> caught up yet. Waiting a bit ensures that the PoP will fetch the
>> file, and when you try again, it's there (as you can see).
>>
>> I think the way to fix this is, freshclam, if it receives an "I'm
>> behind" error from the PoP, to do a sleep for awhile and then try
>> again. If the second attempt still fails then give the error to the
>> user.

I asked:
> Would it be sensible for freshclam to update the file when a newer
> version is available, even if it is not the newest ?

To be clearer, say I have version 26011, the DNS says 26013 is current
but the newest that freshclam can find on any configured mirror is 26012,
it might be better to update to 26012 than wait for 26013.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

With regard to "sleep for awhile".

I remember that Cloudflare's BOS server on occasion remained behind the
latest CVD version (according to the DNS TXT record) for more than one
hour!

Might the following be possible instead?

I would imagine that Cloudflare has a means of fetching a specific file
from any of their own mirror servers (via its unique, non-anycast, IP
address) to check its operation. If ClamAV DB files could be requested
from specific (i.e., all) CF servers, it would cause them to be pulled
from ClamAV's master server(s).

Is this something CF could do for ClamAV? AV software helps improve
Internet security, which seems to be part of CF's mission.


On Thu, 10 Dec 2020 13:54:22 +0000
"Joel Esler \(jesler\) via clamav-users" <clamav-users@lists.clamav.net> wrote:

> I think the way to fix this is, freshclam, if it receives an “I’m
> behind” error from the PoP, to do a sleep for awhile and then try
> again. If the second attempt still fails then give the error to the
> user.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Thu, 10 Dec 2020, Joel Esler (jesler) via clamav-users wrote:

>>> I think the way to fix this is, freshclam, if it receives an "I'm
>>> behind" error from the PoP, to do a sleep for awhile and then try
>>> again. ...

Maybe the workaround is simpler than that.

The document at

https://www.clamav.net/documents/private-local-mirrors

tells the reader to set the 'ScriptedUpdates' option to 'no' for
_both_ the local mirror _and_ that mirror's clients.

I can understand the logic of setting the option to 'no' for clients
of the local mirror, because a local mirror won't serve '.cdiff' files
and if they ask the local mirror for such a file they'll get a 404.

But the local mirror could grab the .cdiff files from the Cloudflare
mirrors using freshclam, just as does any client which does _not_ use
a local mirror, no?

What reason is there for not using 'ScriptedUpdates yes' on the mirror?

As I said earlier to the OP, I've never seen the problem that he's
complaining of and I'm beginning to suspect that he's right - that
it's the use of the

ScriptedUpdates no

option which is at the root of the problem. (Well, that and the fact
that Cloudflare apparently isn't providing the service that Cisco has
presumably contracted it to provide - if all that's necessary in order
for the Cloudflare PoP to update its copy of the .cvd file is for some
random client to request a download of it, then you'd expect that the
OP's request would trigger that, and apparently it doesn't).

Most freshclam daemons will be configured to make just a few attempts
per day to update, and a failure will mean using outdated databases
(on a server which by definition is providing service to many clients)
until at least the time of the next scheduled update. That and the
"try again in an hour or two" suggestion seem to fly in the face of
the freshclam man page:

--on-error-execute=COMMAND Execute COMMAND if error occurred.
Remember, that virus database freshness is the most important thing in
anti-virus system. ...

I wonder if another workaround might be to use the 'DatabaseMirror' or
'PrivateMirror' options in freshclam.conf to avoid Cloudflare issues.

But the real fix must be in the hands of Cloudflare, or perhaps those
of Cloudflare's customers (making more fuss about something which, at
first sight, could very easily be remedied).

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> On Dec 10, 2020, at 11:58 AM, Paul Kosinski via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> I would imagine that Cloudflare has a means of fetching a specific file
> from any of their own mirror servers (via its unique, non-anycast, IP
> address) to check its operation. If ClamAV DB files could be requested
> from specific (i.e., all) CF servers, it would cause them to be pulled
> from ClamAV's master server(s).


We already do this.

> On Dec 10, 2020, at 12:21 PM, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> But the real fix must be in the hands of Cloudflare, or perhaps those
> of Cloudflare's customers (making more fuss about something which, at
> first sight, could very easily be remedied).

While I agree, I am sure tweaks can be done, the fact that CF serves over 200TB of data a day for just clamav in terms of updates, and the problems are very minor, I’d say they are doing pretty good.

Ged, Joel, Andrew, Paul:

Ged wrote:
> As I said earlier to the OP, I've never seen the problem that he's complaining of and I'm beginning to suspect that he's right - that it's the use of the `ScriptedUpdates no` option which is at the root of the problem.

This is correct -- there is no issue getting the latest patch when using scripted updates. The issue is when trying to download the whole CVD. The whole CVD filename is not versioned (always "daily.cvd") which is why the CloudFlare caching issue may result in serving the previous version.

Andrew wrote:
> Would it be sensible for freshclam to update the file when a newer version is available, even if it is not the newest ?
> ...
> To be clearer, say I have version 26011, the DNS says 26013 is current but the newest that freshclam can find on any configured mirror is 26012, it might be better to update to 26012 than wait for 26013.

It should already do this. If you have version 26011 and it says 26013, but only 26012 is available, it should get 26012. If that's not working -- let me know, we'd have a bug to fix.

Joel wrote:
> I think the way to fix this is, freshclam, if it receives an "I'm behind" error from the PoP, to do a sleep for awhile and then try again. If the second attempt still fails then give the error to the user.

I want to be clear -- the message that was originally reported is not an error message. It's a verbose (a.k.a debug-level) message. If you're running freshclam relatively frequently, then this "wait a while and try again" thing is transparent to you. Disable the `Verbose` option in freshclam.conf and don't worry about it.

-Micah

> -----Original Message-----
> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> G.W. Haywood via clamav-users
> Sent: Thursday, December 10, 2020 9:21 AM
> To: Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net>
> Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
> Subject: Re: [clamav-users] local server takes time to update clamav db
>
> Hi there,
>
> On Thu, 10 Dec 2020, Joel Esler (jesler) via clamav-users wrote:
>
> >>> I think the way to fix this is, freshclam, if it receives an "I'm
> >>> behind" error from the PoP, to do a sleep for awhile and then try
> >>> again. ...
>
> Maybe the workaround is simpler than that.
>
> The document at
>
> https://www.clamav.net/documents/private-local-mirrors
>
> tells the reader to set the 'ScriptedUpdates' option to 'no' for _both_ the local
> mirror _and_ that mirror's clients.
>
> I can understand the logic of setting the option to 'no' for clients of the local
> mirror, because a local mirror won't serve '.cdiff' files and if they ask the local
> mirror for such a file they'll get a 404.
>
> But the local mirror could grab the .cdiff files from the Cloudflare mirrors
> using freshclam, just as does any client which does _not_ use a local mirror,
> no?
>
> What reason is there for not using 'ScriptedUpdates yes' on the mirror?
>
> As I said earlier to the OP, I've never seen the problem that he's complaining
> of and I'm beginning to suspect that he's right - that it's the use of the
>
> ScriptedUpdates no
>
> option which is at the root of the problem. (Well, that and the fact that
> Cloudflare apparently isn't providing the service that Cisco has presumably
> contracted it to provide - if all that's necessary in order for the Cloudflare PoP
> to update its copy of the .cvd file is for some random client to request a
> download of it, then you'd expect that the OP's request would trigger that,
> and apparently it doesn't).
>
> Most freshclam daemons will be configured to make just a few attempts per
> day to update, and a failure will mean using outdated databases (on a server
> which by definition is providing service to many clients) until at least the time
> of the next scheduled update. That and the "try again in an hour or two"
> suggestion seem to fly in the face of the freshclam man page:
>
> --on-error-execute=COMMAND Execute COMMAND if error occurred.
> Remember, that virus database freshness is the most important thing in
> anti-virus system. ...
>
> I wonder if another workaround might be to use the 'DatabaseMirror' or
> 'PrivateMirror' options in freshclam.conf to avoid Cloudflare issues.
>
> But the real fix must be in the hands of Cloudflare, or perhaps those of
> Cloudflare's customers (making more fuss about something which, at first
> sight, could very easily be remedied).
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Does ClamAV (Talos?) check *all* the Cloudflare anycast servers?
I thought it could only check those "near" to ClamAV POPs.


On Thu, 10 Dec 2020 18:00:15 +0000
"Joel Esler (jesler)" <jesler@cisco.com> wrote:

> > On Dec 10, 2020, at 11:58 AM, Paul Kosinski via clamav-users <clamav-users@lists.clamav.net> wrote:
> >
> > I would imagine that Cloudflare has a means of fetching a specific file
> > from any of their own mirror servers (via its unique, non-anycast, IP
> > address) to check its operation. If ClamAV DB files could be requested
> > from specific (i.e., all) CF servers, it would cause them to be pulled
> > from ClamAV's master server(s).
>
>
> We already do this.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

"The whole CVD filename is not versioned (always "daily.cvd") which is
why the CloudFlare caching issue may result in serving the previous
version."

HTML filenames for Web pages are not versioned either. Does this mean
that CDNs like Cloudflare often serve up obsolete Web pages? If so, does
nobody notice (and complain)?

A delay of an hour could have an adverse effect on online commerce,
especially during the busy holiday season.


On Thu, 10 Dec 2020 18:34:36 +0000
"Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net> wrote:

> Ged, Joel, Andrew, Paul:
>
> Ged wrote:
> > As I said earlier to the OP, I've never seen the problem that he's complaining of and I'm beginning to suspect that he's right - that it's the use of the `ScriptedUpdates no` option which is at the root of the problem.
>
> This is correct -- there is no issue getting the latest patch when using scripted updates. The issue is when trying to download the whole CVD. The whole CVD filename is not versioned (always "daily.cvd") which is why the CloudFlare caching issue may result in serving the previous version.
>
> Andrew wrote:
> > Would it be sensible for freshclam to update the file when a newer version is available, even if it is not the newest ?
> > ...
> > To be clearer, say I have version 26011, the DNS says 26013 is current but the newest that freshclam can find on any configured mirror is 26012, it might be better to update to 26012 than wait for 26013.
>
> It should already do this. If you have version 26011 and it says 26013, but only 26012 is available, it should get 26012. If that's not working -- let me know, we'd have a bug to fix.
>
> Joel wrote:
> > I think the way to fix this is, freshclam, if it receives an "I'm behind" error from the PoP, to do a sleep for awhile and then try again. If the second attempt still fails then give the error to the user.
>
> I want to be clear -- the message that was originally reported is not an error message. It's a verbose (a.k.a debug-level) message. If you're running freshclam relatively frequently, then this "wait a while and try again" thing is transparent to you. Disable the `Verbose` option in freshclam.conf and don't worry about it.
>
> -Micah
>
> > -----Original Message-----
> > From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> > G.W. Haywood via clamav-users
> > Sent: Thursday, December 10, 2020 9:21 AM
> > To: Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net>
> > Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
> > Subject: Re: [clamav-users] local server takes time to update clamav db
> >
> > Hi there,
> >
> > On Thu, 10 Dec 2020, Joel Esler (jesler) via clamav-users wrote:
> >
> > >>> I think the way to fix this is, freshclam, if it receives an "I'm
> > >>> behind" error from the PoP, to do a sleep for awhile and then try
> > >>> again. ...
> >
> > Maybe the workaround is simpler than that.
> >
> > The document at
> >
> > https://www.clamav.net/documents/private-local-mirrors
> >
> > tells the reader to set the 'ScriptedUpdates' option to 'no' for _both_ the local
> > mirror _and_ that mirror's clients.
> >
> > I can understand the logic of setting the option to 'no' for clients of the local
> > mirror, because a local mirror won't serve '.cdiff' files and if they ask the local
> > mirror for such a file they'll get a 404.
> >
> > But the local mirror could grab the .cdiff files from the Cloudflare mirrors
> > using freshclam, just as does any client which does _not_ use a local mirror,
> > no?
> >
> > What reason is there for not using 'ScriptedUpdates yes' on the mirror?
> >
> > As I said earlier to the OP, I've never seen the problem that he's complaining
> > of and I'm beginning to suspect that he's right - that it's the use of the
> >
> > ScriptedUpdates no
> >
> > option which is at the root of the problem. (Well, that and the fact that
> > Cloudflare apparently isn't providing the service that Cisco has presumably
> > contracted it to provide - if all that's necessary in order for the Cloudflare PoP
> > to update its copy of the .cvd file is for some random client to request a
> > download of it, then you'd expect that the OP's request would trigger that,
> > and apparently it doesn't).
> >
> > Most freshclam daemons will be configured to make just a few attempts per
> > day to update, and a failure will mean using outdated databases (on a server
> > which by definition is providing service to many clients) until at least the time
> > of the next scheduled update. That and the "try again in an hour or two"
> > suggestion seem to fly in the face of the freshclam man page:
> >
> > --on-error-execute=COMMAND Execute COMMAND if error occurred.
> > Remember, that virus database freshness is the most important thing in
> > anti-virus system. ...
> >
> > I wonder if another workaround might be to use the 'DatabaseMirror' or
> > 'PrivateMirror' options in freshclam.conf to avoid Cloudflare issues.
> >
> > But the real fix must be in the hands of Cloudflare, or perhaps those of
> > Cloudflare's customers (making more fuss about something which, at first
> > sight, could very easily be remedied).
> >
> > --
> >
> > 73,
> > Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Except we purge the cache at time of file upload.

Sent from my ? iPhone

> On Dec 11, 2020, at 10:53, Paul Kosinski via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?"The whole CVD filename is not versioned (always "daily.cvd") which is
> why the CloudFlare caching issue may result in serving the previous
> version."
>
> HTML filenames for Web pages are not versioned either. Does this mean
> that CDNs like Cloudflare often serve up obsolete Web pages? If so, does
> nobody notice (and complain)?
>
> A delay of an hour could have an adverse effect on online commerce,
> especially during the busy holiday season.
>
>
>> On Thu, 10 Dec 2020 18:34:36 +0000
>> "Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net> wrote:
>>
>> Ged, Joel, Andrew, Paul:
>>
>> Ged wrote:
>>> As I said earlier to the OP, I've never seen the problem that he's complaining of and I'm beginning to suspect that he's right - that it's the use of the `ScriptedUpdates no` option which is at the root of the problem.
>>
>> This is correct -- there is no issue getting the latest patch when using scripted updates. The issue is when trying to download the whole CVD. The whole CVD filename is not versioned (always "daily.cvd") which is why the CloudFlare caching issue may result in serving the previous version.
>>
>> Andrew wrote:
>>> Would it be sensible for freshclam to update the file when a newer version is available, even if it is not the newest ?
>>> ...
>>> To be clearer, say I have version 26011, the DNS says 26013 is current but the newest that freshclam can find on any configured mirror is 26012, it might be better to update to 26012 than wait for 26013.
>>
>> It should already do this. If you have version 26011 and it says 26013, but only 26012 is available, it should get 26012. If that's not working -- let me know, we'd have a bug to fix.
>>
>> Joel wrote:
>>> I think the way to fix this is, freshclam, if it receives an "I'm behind" error from the PoP, to do a sleep for awhile and then try again. If the second attempt still fails then give the error to the user.
>>
>> I want to be clear -- the message that was originally reported is not an error message. It's a verbose (a.k.a debug-level) message. If you're running freshclam relatively frequently, then this "wait a while and try again" thing is transparent to you. Disable the `Verbose` option in freshclam.conf and don't worry about it.
>>
>> -Micah
>>
>>> -----Original Message-----
>>> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
>>> G.W. Haywood via clamav-users
>>> Sent: Thursday, December 10, 2020 9:21 AM
>>> To: Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net>
>>> Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
>>> Subject: Re: [clamav-users] local server takes time to update clamav db
>>>
>>> Hi there,
>>>
>>>> On Thu, 10 Dec 2020, Joel Esler (jesler) via clamav-users wrote:
>>>
>>>>>> I think the way to fix this is, freshclam, if it receives an "I'm
>>>>>> behind" error from the PoP, to do a sleep for awhile and then try
>>>>>> again. ...
>>>
>>> Maybe the workaround is simpler than that.
>>>
>>> The document at
>>>
>>> https://www.clamav.net/documents/private-local-mirrors
>>>
>>> tells the reader to set the 'ScriptedUpdates' option to 'no' for _both_ the local
>>> mirror _and_ that mirror's clients.
>>>
>>> I can understand the logic of setting the option to 'no' for clients of the local
>>> mirror, because a local mirror won't serve '.cdiff' files and if they ask the local
>>> mirror for such a file they'll get a 404.
>>>
>>> But the local mirror could grab the .cdiff files from the Cloudflare mirrors
>>> using freshclam, just as does any client which does _not_ use a local mirror,
>>> no?
>>>
>>> What reason is there for not using 'ScriptedUpdates yes' on the mirror?
>>>
>>> As I said earlier to the OP, I've never seen the problem that he's complaining
>>> of and I'm beginning to suspect that he's right - that it's the use of the
>>>
>>> ScriptedUpdates no
>>>
>>> option which is at the root of the problem. (Well, that and the fact that
>>> Cloudflare apparently isn't providing the service that Cisco has presumably
>>> contracted it to provide - if all that's necessary in order for the Cloudflare PoP
>>> to update its copy of the .cvd file is for some random client to request a
>>> download of it, then you'd expect that the OP's request would trigger that,
>>> and apparently it doesn't).
>>>
>>> Most freshclam daemons will be configured to make just a few attempts per
>>> day to update, and a failure will mean using outdated databases (on a server
>>> which by definition is providing service to many clients) until at least the time
>>> of the next scheduled update. That and the "try again in an hour or two"
>>> suggestion seem to fly in the face of the freshclam man page:
>>>
>>> --on-error-execute=COMMAND Execute COMMAND if error occurred.
>>> Remember, that virus database freshness is the most important thing in
>>> anti-virus system. ...
>>>
>>> I wonder if another workaround might be to use the 'DatabaseMirror' or
>>> 'PrivateMirror' options in freshclam.conf to avoid Cloudflare issues.
>>>
>>> But the real fix must be in the hands of Cloudflare, or perhaps those of
>>> Cloudflare's customers (making more fuss about something which, at first
>>> sight, could very easily be remedied).
>>>
>>> --
>>>
>>> 73,
>>> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

On 2020-12-11 08:51, Paul Kosinski via clamav-users wrote:
> "The whole CVD filename is not versioned (always "daily.cvd") which is
> why the CloudFlare caching issue may result in serving the previous
> version."
>
> HTML filenames for Web pages are not versioned either. Does this mean
> that CDNs like Cloudflare often serve up obsolete Web pages? If so, does
> nobody notice (and complain)?
>
> A delay of an hour could have an adverse effect on online commerce,
> especially during the busy holiday season.

By default Cloudflare does not cache HTML. Cloudflare also respects
cache-control headers, which is the normal mechanism used for websites
which want caching, but only to a point.

Cloudflare also has an API to clear the cache (at least by URI, or
everything, and possibly more depending on the particular options
offered by your plan). But in practice clearing the cache is not
completely reliable and seems to be intended for cases where it is
strictly needed and not for every "I updated this file" situation. I
have the impression that this applies when using Cloudflare's tiered
caching, my idle speculation wonders if perhaps this is a timing issue,
where server #1 clears the cache, processes a request for the file which
it obtains from server #2 all before server #2 clears the file from
cache and then processes a request by pulling it from server #1.

From a ClamAV perspective, one solution to solve this would be to call
daily.cvd?version=26013 -- Note that the underlying web server could
ignore the version parameter completely, but this would ensure that each
Cloudflare cache retrieves a fresh version of the file and negates the
need to push a cache clear message at all. If ClamAV's server serves an
outdated version of the file then it would still get cached, but this
would defeat any caching within Cloudflare for new versions as they're
released.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Both of those things are done as well.

Sent from my ? iPhone

> On Dec 13, 2020, at 19:24, Dave Warren via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?On 2020-12-11 08:51, Paul Kosinski via clamav-users wrote:
>> "The whole CVD filename is not versioned (always "daily.cvd") which is
>> why the CloudFlare caching issue may result in serving the previous
>> version."
>> HTML filenames for Web pages are not versioned either. Does this mean
>> that CDNs like Cloudflare often serve up obsolete Web pages? If so, does
>> nobody notice (and complain)?
>> A delay of an hour could have an adverse effect on online commerce,
>> especially during the busy holiday season.
>
> By default Cloudflare does not cache HTML. Cloudflare also respects cache-control headers, which is the normal mechanism used for websites which want caching, but only to a point.
>
> Cloudflare also has an API to clear the cache (at least by URI, or everything, and possibly more depending on the particular options offered by your plan). But in practice clearing the cache is not completely reliable and seems to be intended for cases where it is strictly needed and not for every "I updated this file" situation. I have the impression that this applies when using Cloudflare's tiered caching, my idle speculation wonders if perhaps this is a timing issue, where server #1 clears the cache, processes a request for the file which it obtains from server #2 all before server #2 clears the file from cache and then processes a request by pulling it from server #1.
>
> From a ClamAV perspective, one solution to solve this would be to call daily.cvd?version=26013 -- Note that the underlying web server could ignore the version parameter completely, but this would ensure that each Cloudflare cache retrieves a fresh version of the file and negates the need to push a cache clear message at all. If ClamAV's server serves an outdated version of the file then it would still get cached, but this would defeat any caching within Cloudflare for new versions as they're released.
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

I did a quick grep on the the source code (and compiled output too) of
ClamAV 0.103.0, and I couldn't find any instance of 'CF-Cache-Status'.
Should freshclam (or somebody) be checking this HTTP header line that
Cloudflare returns? The 'STALE' and 'UPDATING' values sound like they
might be particularly relevant.


On Mon, 14 Dec 2020 02:57:48 +0000
"Joel Esler \(jesler\) via clamav-users" <clamav-users@lists.clamav.net> wrote:

> Both of those things are done as well.
>
> Sent from my ? iPhone
>
> > On Dec 13, 2020, at 19:24, Dave Warren via clamav-users <clamav-users@lists.clamav.net> wrote:
> >
> > ?On 2020-12-11 08:51, Paul Kosinski via clamav-users wrote:
> >> "The whole CVD filename is not versioned (always "daily.cvd") which is
> >> why the CloudFlare caching issue may result in serving the previous
> >> version."
> >> HTML filenames for Web pages are not versioned either. Does this mean
> >> that CDNs like Cloudflare often serve up obsolete Web pages? If so, does
> >> nobody notice (and complain)?
> >> A delay of an hour could have an adverse effect on online commerce,
> >> especially during the busy holiday season.
> >
> > By default Cloudflare does not cache HTML. Cloudflare also respects cache-control headers, which is the normal mechanism used for websites which want caching, but only to a point.
> >
> > Cloudflare also has an API to clear the cache (at least by URI, or everything, and possibly more depending on the particular options offered by your plan). But in practice clearing the cache is not completely reliable and seems to be intended for cases where it is strictly needed and not for every "I updated this file" situation. I have the impression that this applies when using Cloudflare's tiered caching, my idle speculation wonders if perhaps this is a timing issue, where server #1 clears the cache, processes a request for the file which it obtains from server #2 all before server #2 clears the file from cache and then processes a request by pulling it from server #1.
> >
> > From a ClamAV perspective, one solution to solve this would be to call daily.cvd?version=26013 -- Note that the underlying web server could ignore the version parameter completely, but this would ensure that each Cloudflare cache retrieves a fresh version of the file and negates the need to push a cache clear message at all. If ClamAV's server serves an outdated version of the file then it would still get cached, but this would defeat any caching within Cloudflare for new versions as they're released.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Okay, so then it seems like 1) ClamAV’s origin server periodically serves an old version of a file after the DNS TXT record is updated, or 2) Cloudflare returns a cached resource from the wrong URL, or 3) Someone is making a request to new ?version URLs before the DNS TXT record is updated (and such would be visible in the origin server’s HTTP request log).

What is the URL format that is used? I don’t see an obvious example in the conf man pages for the fully constructed URL, and I’m not near a full computer to figure it out. I was hoping to throw a few HTTP requests at it and see if the headers give any clues.

I have no way to prove or test #1, but #2 would be a major and fairly obvious issue that would cause an impact to virtually all Cloudflare customers. While not impossible, this seems unlikely.

#3 would certainly be possible, but would be moderately straightforward to identify on the web server hosting the original files — Or could be avoided if the origin web server includes a cache-control: no-cache (or maybe max-age=300) for version numbers greater than the current, while still returning whatever version is actually current, so that the requesting client still gets something valid, but I’m not clear what, if any, smarts are contained on the origin server.

Either way, perhaps “cache-control: max-age=3600, must-revalidate” would make sense so that the problem has the opportunity to clear itself faster than the current 43200 seconds? As long as the origin server supports last-modified and similar, the impact would be relatively minimal in terms of the number of bytes delivered, although the number of requests making it to the origin would increase somewhat, but still well within the capabilities of a modest server.

I’m sure smarter minds than I have looked at this, but it seems like a relatively small set of possibilities, and it just seems unlikely to me that it would go unnoticed if Cloudflare were regularly returning cached content from a different URL.



On Sun, Dec 13, 2020, at 19:57, Joel Esler (jesler) via clamav-users wrote:
> Both of those things are done as well.
>
> Sent from my ? iPhone
>
> > On Dec 13, 2020, at 19:24, Dave Warren via clamav-users <clamav-users@lists.clamav.net> wrote:
> >
> > ?On 2020-12-11 08:51, Paul Kosinski via clamav-users wrote:
> >> "The whole CVD filename is not versioned (always "daily.cvd") which is
> >> why the CloudFlare caching issue may result in serving the previous
> >> version."
> >> HTML filenames for Web pages are not versioned either. Does this mean
> >> that CDNs like Cloudflare often serve up obsolete Web pages? If so, does
> >> nobody notice (and complain)?
> >> A delay of an hour could have an adverse effect on online commerce,
> >> especially during the busy holiday season.
> >
> > By default Cloudflare does not cache HTML. Cloudflare also respects cache-control headers, which is the normal mechanism used for websites which want caching, but only to a point.
> >
> > Cloudflare also has an API to clear the cache (at least by URI, or everything, and possibly more depending on the particular options offered by your plan). But in practice clearing the cache is not completely reliable and seems to be intended for cases where it is strictly needed and not for every "I updated this file" situation. I have the impression that this applies when using Cloudflare's tiered caching, my idle speculation wonders if perhaps this is a timing issue, where server #1 clears the cache, processes a request for the file which it obtains from server #2 all before server #2 clears the file from cache and then processes a request by pulling it from server #1.
> >
> > From a ClamAV perspective, one solution to solve this would be to call daily.cvd?version=26013 -- Note that the underlying web server could ignore the version parameter completely, but this would ensure that each Cloudflare cache retrieves a fresh version of the file and negates the need to push a cache clear message at all. If ClamAV's server serves an outdated version of the file then it would still get cached, but this would defeat any caching within Cloudflare for new versions as they're released.
> >
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
> *Attachments:*
> * smime.p7s