Mailing List Archive: [clamav-users] Problem with private mirror and cld, inc files

[clamav-users] Problem with private mirror and cld, inc files

Jan 26, 2021, 8:46 AM

Post #1 of 8 (657 views)

Hi

I am using Ubuntu 18.04 and recently found out that downloaind clamav files
was blocked by cloudflare. Based on
https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html I
contacted Joel Esler (many thanks for the quick reply) and inform me that
we were downloading too often.

I tried to fix the situation by introducing a private mirror, but I ran
into problems with cld and inc files and systemd conditions for the
clamav-daemon service. I'd like to ask what are the "cld" and "inc" files,
and how to download them in order to meet the systemd condition that
requires them to be on disk?

I did the following steps:
1) Created a private mirror as described in
https://www.clamav.net/documents/private-local-mirrors and mirrored all
"cvd" files.

2) Cloud-init ensures that on boot the files are downloaded from the
private mirror and are always available locally. Tthen cloud-init restarts
the clamav daemon.

3) Configured freshclam with
PrivateMirror https://my-clamav-mirror/clamav
ScriptedUpdates no
Checks 3

4) Now freshclam works fine, however the clamav daemon does not, even if
the cvd are available locally.
$ sudo systemctl status clamav-daemon.service
? clamav-daemon.service - Clam AntiVirus userspace daemon
Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled;
vendor preset: enabled)
Drop-In: /etc/systemd/system/clamav-daemon.service.d
??extend.conf
Active: inactive (dead)
Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min
ago
?? ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
was not met
Docs: man:clamd(8)
man:clamd.conf(5)
https://www.clamav.net/documents/

The systemd unit file has 2 ConditionPathExistsGlob that require the
additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be
available, for the service to start.
$ cat /lib/systemd/system/clamav-daemon.service
[Unit]
Description=Clam AntiVirus userspace daemon
Documentation=man:clamd(8) man:clamd.conf(5)
https://www.clamav.net/documents/
# Check for database existence
ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}

[Service]
ExecStart=/usr/sbin/clamd --foreground=true
# Reload the database
ExecReload=/bin/kill -USR2 $MAINPID
StandardOutput=syslog
TimeoutStartSec=420

[Install]
WantedBy=multi-user.target

5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not
available to download from the main download site

$ curl --head -s https://database.clamav.net/daily.cld | grep HTTP
HTTP/2 404
$ curl --head -s https://database.clamav.net/daily.inc | grep HTTP
HTTP/2 404
$ curl --head -s https://database.clamav.net/main.cld | grep HTTP
HTTP/2 404
$ curl --head -s https://database.clamav.net/main.inc | grep HTTP
HTTP/2 404

6) I ran the suggested clamdownloader.pl (again from
https://www.clamav.net/documents/private-local-mirrors) but I don't see any
$ ls -al /tmp/clam
total 222848
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 26 16:27 .
drwxrwxrwt 93 root root 12288 Jan 26 16:27 ..
-rw-rw-r-- 1 ubuntu ubuntu 296388 Sep 19 2019 bytecode.cvd
-rw-rw-r-- 1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
-rw-rw-r-- 1 ubuntu ubuntu 42 Jan 26 16:27 dns.txt
-rw-rw-r-- 1 ubuntu ubuntu 117859675 Nov 25 2019 main.cvd
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 26 16:27 temp

but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc" files.

Repeating the question in the beginning: I'd like to ask what are the "cld"
and "inc" files, and how to download in my private mirror in order to meet
the systemd condition that requires them to be on disk?

Regards
Vangelis Katsikaros

Re: [clamav-users] Problem with private mirror and cld, inc files [ In reply to ]

clamav-users at lists

Jan 26, 2021, 9:00 AM

Post #2 of 8 (657 views)

Permalink

Why wouldn’t you just download less often? Instead of doing all of this? We publish updates once a day, there’s no need to download the entire package once a second.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com | https://www.snort.org

> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Hi
>
> I am using Ubuntu 18.04 and recently found out that downloaind clamav files was blocked by cloudflare. Based on https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html <https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html> I contacted Joel Esler (many thanks for the quick reply) and inform me that we were downloading too often.
>
> I tried to fix the situation by introducing a private mirror, but I ran into problems with cld and inc files and systemd conditions for the clamav-daemon service. I'd like to ask what are the "cld" and "inc" files, and how to download them in order to meet the systemd condition that requires them to be on disk?
>
> I did the following steps:
> 1) Created a private mirror as described in https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors> and mirrored all "cvd" files.
>
> 2) Cloud-init ensures that on boot the files are downloaded from the private mirror and are always available locally. Tthen cloud-init restarts the clamav daemon.
>
> 3) Configured freshclam with
> PrivateMirror https://my-clamav-mirror/clamav <https://my-clamav-mirror/clamav>
> ScriptedUpdates no
> Checks 3
>
> 4) Now freshclam works fine, however the clamav daemon does not, even if the cvd are available locally.
> $ sudo systemctl status clamav-daemon.service
> ? clamav-daemon.service - Clam AntiVirus userspace daemon
> Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
> Drop-In: /etc/systemd/system/clamav-daemon.service.d
> ??extend.conf
> Active: inactive (dead)
> Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min ago
> ?? ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc} was not met
> Docs: man:clamd(8)
> man:clamd.conf(5)
> https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
>
> The systemd unit file has 2 ConditionPathExistsGlob that require the additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be available, for the service to start.
> $ cat /lib/systemd/system/clamav-daemon.service
> [Unit]
> Description=Clam AntiVirus userspace daemon
> Documentation=man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
> # Check for database existence
> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>
> [Service]
> ExecStart=/usr/sbin/clamd --foreground=true
> # Reload the database
> ExecReload=/bin/kill -USR2 $MAINPID
> StandardOutput=syslog
> TimeoutStartSec=420
>
> [Install]
> WantedBy=multi-user.target
>
>
> 5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not available to download from the main download site
>
> $ curl --head -s https://database.clamav.net/daily.cld <https://database.clamav.net/daily.cld> | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/daily.inc <https://database.clamav.net/daily.inc> | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/main.cld <https://database.clamav.net/main.cld> | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/main.inc <https://database.clamav.net/main.inc> | grep HTTP
> HTTP/2 404
>
>
> 6) I ran the suggested clamdownloader.pl <http://clamdownloader.pl/> (again from https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>) but I don't see any
> $ ls -al /tmp/clam
> total 222848
> drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 26 16:27 .
> drwxrwxrwt 93 root root 12288 Jan 26 16:27 ..
> -rw-rw-r-- 1 ubuntu ubuntu 296388 Sep 19 2019 bytecode.cvd
> -rw-rw-r-- 1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
> -rw-rw-r-- 1 ubuntu ubuntu 42 Jan 26 16:27 dns.txt
> -rw-rw-r-- 1 ubuntu ubuntu 117859675 Nov 25 2019 main.cvd
> drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 26 16:27 temp
>
> but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc" files.
>
> Repeating the question in the beginning: I'd like to ask what are the "cld" and "inc" files, and how to download in my private mirror in order to meet the systemd condition that requires them to be on disk?
>
> Regards
> Vangelis Katsikaros
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

Re: [clamav-users] Problem with private mirror and cld, inc files [ In reply to ]

clamav-users at lists

Jan 26, 2021, 9:17 AM

Post #3 of 8 (657 views)

Permalink

Hi Joel, thanks for the quick response. We already download once every hour
(the default ubuntu 18.04 behavior). However, we are using auto scaling and
we might be running a large number of EC2 instances (a few hundreds), that
could try to download simultaneously. To further scale without issues it
seemed quite easy (and it was easy in practice) to use a private mirror.
The system conditions and the cld & inc files is the only blocker to make
this work :/

On Tue, Jan 26, 2021 at 7:00 PM Joel Esler (jesler) <jesler@cisco.com>
wrote:

> Why wouldn’t you just download less often? Instead of doing all of this?
> We publish updates once a day, there’s no need to download the entire
> package once a second.
>
> --
> Joel Esler
> Manager, Communities Division
> Cisco Talos Intelligence Group
> http://www.talosintelligence.com | https://www.snort.org
>
> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users <
> clamav-users@lists.clamav.net> wrote:
>
> Hi
>
> I am using Ubuntu 18.04 and recently found out that downloaind clamav
> files was blocked by cloudflare. Based on
> https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html I
> contacted Joel Esler (many thanks for the quick reply) and inform me that
> we were downloading too often.
>
> I tried to fix the situation by introducing a private mirror, but I ran
> into problems with cld and inc files and systemd conditions for the
> clamav-daemon service. I'd like to ask what are the "cld" and "inc" files,
> and how to download them in order to meet the systemd condition that
> requires them to be on disk?
>
> I did the following steps:
> 1) Created a private mirror as described in
> https://www.clamav.net/documents/private-local-mirrors and mirrored all
> "cvd" files.
>
> 2) Cloud-init ensures that on boot the files are downloaded from the
> private mirror and are always available locally. Tthen cloud-init restarts
> the clamav daemon.
>
> 3) Configured freshclam with
> PrivateMirror https://my-clamav-mirror/clamav
> ScriptedUpdates no
> Checks 3
>
> 4) Now freshclam works fine, however the clamav daemon does not, even if
> the cvd are available locally.
> $ sudo systemctl status clamav-daemon.service
> ? clamav-daemon.service - Clam AntiVirus userspace daemon
> Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled;
> vendor preset: enabled)
> Drop-In: /etc/systemd/system/clamav-daemon.service.d
> ??extend.conf
> Active: inactive (dead)
> Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min
> ago
> ?? ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
> was not met
> Docs: man:clamd(8)
> man:clamd.conf(5)
> https://www.clamav.net/documents/
>
> The systemd unit file has 2 ConditionPathExistsGlob that require the
> additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be
> available, for the service to start.
> $ cat /lib/systemd/system/clamav-daemon.service
> [Unit]
> Description=Clam AntiVirus userspace daemon
> Documentation=man:clamd(8) man:clamd.conf(5)
> https://www.clamav.net/documents/
> # Check for database existence
> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>
> [Service]
> ExecStart=/usr/sbin/clamd --foreground=true
> # Reload the database
> ExecReload=/bin/kill -USR2 $MAINPID
> StandardOutput=syslog
> TimeoutStartSec=420
>
> [Install]
> WantedBy=multi-user.target
>
>
> 5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not
> available to download from the main download site
>
> $ curl --head -s https://database.clamav.net/daily.cld | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/daily.inc | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/main.cld | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/main.inc | grep HTTP
> HTTP/2 404
>
>
> 6) I ran the suggested clamdownloader.pl (again from
> https://www.clamav.net/documents/private-local-mirrors) but I don't see
> any
> $ ls -al /tmp/clam
> total 222848
> drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 26 16:27 .
> drwxrwxrwt 93 root root 12288 Jan 26 16:27 ..
> -rw-rw-r-- 1 ubuntu ubuntu 296388 Sep 19 2019 bytecode.cvd
> -rw-rw-r-- 1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
> -rw-rw-r-- 1 ubuntu ubuntu 42 Jan 26 16:27 dns.txt
> -rw-rw-r-- 1 ubuntu ubuntu 117859675 Nov 25 2019 main.cvd
> drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 26 16:27 temp
>
> but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc" files.
>
> Repeating the question in the beginning: I'd like to ask what are the
> "cld" and "inc" files, and how to download in my private mirror in order to
> meet the systemd condition that requires them to be on disk?
>
> Regards
> Vangelis Katsikaros
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>

Re: [clamav-users] Problem with private mirror and cld, inc files [ In reply to ]

clamav-users at lists

Jan 26, 2021, 9:20 AM

Post #4 of 8 (657 views)

Permalink

Let me remove the block for your IP and see what we see.

> On Jan 26, 2021, at 12:17 PM, Vangelis Katsikaros <vkatsikaros@gmail.com> wrote:
>
> Hi Joel, thanks for the quick response. We already download once every hour (the default ubuntu 18.04 behavior). However, we are using auto scaling and we might be running a large number of EC2 instances (a few hundreds), that could try to download simultaneously. To further scale without issues it seemed quite easy (and it was easy in practice) to use a private mirror. The system conditions and the cld & inc files is the only blocker to make this work :/
>
> On Tue, Jan 26, 2021 at 7:00 PM Joel Esler (jesler) <jesler@cisco.com <mailto:jesler@cisco.com>> wrote:
> Why wouldn’t you just download less often? Instead of doing all of this? We publish updates once a day, there’s no need to download the entire package once a second.
>
> --
> Joel Esler
> Manager, Communities Division
> Cisco Talos Intelligence Group
> http://www.talosintelligence.com <http://www.talosintelligence.com/> | https://www.snort.org <https://www.snort.org/>
>
>> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>
>> Hi
>>
>> I am using Ubuntu 18.04 and recently found out that downloaind clamav files was blocked by cloudflare. Based on https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html <https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html> I contacted Joel Esler (many thanks for the quick reply) and inform me that we were downloading too often.
>>
>> I tried to fix the situation by introducing a private mirror, but I ran into problems with cld and inc files and systemd conditions for the clamav-daemon service. I'd like to ask what are the "cld" and "inc" files, and how to download them in order to meet the systemd condition that requires them to be on disk?
>>
>> I did the following steps:
>> 1) Created a private mirror as described in https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors> and mirrored all "cvd" files.
>>
>> 2) Cloud-init ensures that on boot the files are downloaded from the private mirror and are always available locally. Tthen cloud-init restarts the clamav daemon.
>>
>> 3) Configured freshclam with
>> PrivateMirror https://my-clamav-mirror/clamav <https://my-clamav-mirror/clamav>
>> ScriptedUpdates no
>> Checks 3
>>
>> 4) Now freshclam works fine, however the clamav daemon does not, even if the cvd are available locally.
>> $ sudo systemctl status clamav-daemon.service
>> ? clamav-daemon.service - Clam AntiVirus userspace daemon
>> Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
>> Drop-In: /etc/systemd/system/clamav-daemon.service.d
>> ??extend.conf
>> Active: inactive (dead)
>> Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min ago
>> ?? ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc} was not met
>> Docs: man:clamd(8)
>> man:clamd.conf(5)
>> https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
>>
>> The systemd unit file has 2 ConditionPathExistsGlob that require the additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be available, for the service to start.
>> $ cat /lib/systemd/system/clamav-daemon.service
>> [Unit]
>> Description=Clam AntiVirus userspace daemon
>> Documentation=man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
>> # Check for database existence
>> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
>> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>>
>> [Service]
>> ExecStart=/usr/sbin/clamd --foreground=true
>> # Reload the database
>> ExecReload=/bin/kill -USR2 $MAINPID
>> StandardOutput=syslog
>> TimeoutStartSec=420
>>
>> [Install]
>> WantedBy=multi-user.target
>>
>>
>> 5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not available to download from the main download site
>>
>> $ curl --head -s https://database.clamav.net/daily.cld <https://database.clamav.net/daily.cld> | grep HTTP
>> HTTP/2 404
>> $ curl --head -s https://database.clamav.net/daily.inc <https://database.clamav.net/daily.inc> | grep HTTP
>> HTTP/2 404
>> $ curl --head -s https://database.clamav.net/main.cld <https://database.clamav.net/main.cld> | grep HTTP
>> HTTP/2 404
>> $ curl --head -s https://database.clamav.net/main.inc <https://database.clamav.net/main.inc> | grep HTTP
>> HTTP/2 404
>>
>>
>> 6) I ran the suggested clamdownloader.pl <http://clamdownloader.pl/> (again from https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>) but I don't see any
>> $ ls -al /tmp/clam
>> total 222848
>> drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 26 16:27 .
>> drwxrwxrwt 93 root root 12288 Jan 26 16:27 ..
>> -rw-rw-r-- 1 ubuntu ubuntu 296388 Sep 19 2019 bytecode.cvd
>> -rw-rw-r-- 1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
>> -rw-rw-r-- 1 ubuntu ubuntu 42 Jan 26 16:27 dns.txt
>> -rw-rw-r-- 1 ubuntu ubuntu 117859675 Nov 25 2019 main.cvd
>> drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 26 16:27 temp
>>
>> but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc" files.
>>
>> Repeating the question in the beginning: I'd like to ask what are the "cld" and "inc" files, and how to download in my private mirror in order to meet the systemd condition that requires them to be on disk?
>>
>> Regards
>> Vangelis Katsikaros
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>
>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>

Re: [clamav-users] Problem with private mirror and cld, inc files [ In reply to ]

clamav-users at lists

Jan 27, 2021, 2:45 AM

Post #5 of 8 (656 views)

Permalink

Thanks for the actions Joel, my IPs are now unblocked.

I'd be interested to continue the discussion on
a) how to find the main & daily .inc and .cld files
b) why in ubuntu 18.04 there is a systemd condition for the clamav-daemon
to run, but the files are not available for download
I am afraid that in the long run my service could again DoS
database.clamav.net and it would be prudent for me to proceed with the
private mirror solution.

Regards
Vangelis

On Tue, Jan 26, 2021 at 7:20 PM Joel Esler (jesler) <jesler@cisco.com>
wrote:

> Let me remove the block for your IP and see what we see.
>
> On Jan 26, 2021, at 12:17 PM, Vangelis Katsikaros <vkatsikaros@gmail.com>
> wrote:
>
> Hi Joel, thanks for the quick response. We already download once every
> hour (the default ubuntu 18.04 behavior). However, we are using auto
> scaling and we might be running a large number of EC2 instances (a few
> hundreds), that could try to download simultaneously. To further scale
> without issues it seemed quite easy (and it was easy in practice) to use a
> private mirror. The system conditions and the cld & inc files is the only
> blocker to make this work :/
>
> On Tue, Jan 26, 2021 at 7:00 PM Joel Esler (jesler) <jesler@cisco.com>
> wrote:
>
>> Why wouldn’t you just download less often? Instead of doing all of
>> this? We publish updates once a day, there’s no need to download the
>> entire package once a second.
>>
>> --
>> Joel Esler
>> Manager, Communities Division
>> Cisco Talos Intelligence Group
>> http://www.talosintelligence.com | https://www.snort.org
>>
>> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users <
>> clamav-users@lists.clamav.net> wrote:
>>
>> Hi
>>
>> I am using Ubuntu 18.04 and recently found out that downloaind clamav
>> files was blocked by cloudflare. Based on
>> https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html I
>> contacted Joel Esler (many thanks for the quick reply) and inform me that
>> we were downloading too often.
>>
>> I tried to fix the situation by introducing a private mirror, but I ran
>> into problems with cld and inc files and systemd conditions for the
>> clamav-daemon service. I'd like to ask what are the "cld" and "inc" files,
>> and how to download them in order to meet the systemd condition that
>> requires them to be on disk?
>>
>> I did the following steps:
>> 1) Created a private mirror as described in
>> https://www.clamav.net/documents/private-local-mirrors and mirrored all
>> "cvd" files.
>>
>> 2) Cloud-init ensures that on boot the files are downloaded from the
>> private mirror and are always available locally. Tthen cloud-init restarts
>> the clamav daemon.
>>
>> 3) Configured freshclam with
>> PrivateMirror https://my-clamav-mirror/clamav
>> ScriptedUpdates no
>> Checks 3
>>
>> 4) Now freshclam works fine, however the clamav daemon does not, even if
>> the cvd are available locally.
>> $ sudo systemctl status clamav-daemon.service
>> ? clamav-daemon.service - Clam AntiVirus userspace daemon
>> Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled;
>> vendor preset: enabled)
>> Drop-In: /etc/systemd/system/clamav-daemon.service.d
>> ??extend.conf
>> Active: inactive (dead)
>> Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min
>> ago
>> ?? ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>> was not met
>> Docs: man:clamd(8)
>> man:clamd.conf(5)
>> https://www.clamav.net/documents/
>>
>> The systemd unit file has 2 ConditionPathExistsGlob that require the
>> additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be
>> available, for the service to start.
>> $ cat /lib/systemd/system/clamav-daemon.service
>> [Unit]
>> Description=Clam AntiVirus userspace daemon
>> Documentation=man:clamd(8) man:clamd.conf(5)
>> https://www.clamav.net/documents/
>> # Check for database existence
>> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
>> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>>
>> [Service]
>> ExecStart=/usr/sbin/clamd --foreground=true
>> # Reload the database
>> ExecReload=/bin/kill -USR2 $MAINPID
>> StandardOutput=syslog
>> TimeoutStartSec=420
>>
>> [Install]
>> WantedBy=multi-user.target
>>
>>
>> 5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not
>> available to download from the main download site
>>
>> $ curl --head -s https://database.clamav.net/daily.cld | grep HTTP
>> HTTP/2 404
>> $ curl --head -s https://database.clamav.net/daily.inc | grep HTTP
>> HTTP/2 404
>> $ curl --head -s https://database.clamav.net/main.cld | grep HTTP
>> HTTP/2 404
>> $ curl --head -s https://database.clamav.net/main.inc | grep HTTP
>> HTTP/2 404
>>
>>
>> 6) I ran the suggested clamdownloader.pl (again from
>> https://www.clamav.net/documents/private-local-mirrors) but I don't see
>> any
>> $ ls -al /tmp/clam
>> total 222848
>> drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 26 16:27 .
>> drwxrwxrwt 93 root root 12288 Jan 26 16:27 ..
>> -rw-rw-r-- 1 ubuntu ubuntu 296388 Sep 19 2019 bytecode.cvd
>> -rw-rw-r-- 1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
>> -rw-rw-r-- 1 ubuntu ubuntu 42 Jan 26 16:27 dns.txt
>> -rw-rw-r-- 1 ubuntu ubuntu 117859675 Nov 25 2019 main.cvd
>> drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 26 16:27 temp
>>
>> but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc"
>> files.
>>
>> Repeating the question in the beginning: I'd like to ask what are the
>> "cld" and "inc" files, and how to download in my private mirror in order to
>> meet the systemd condition that requires them to be on disk?
>>
>> Regards
>> Vangelis Katsikaros
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>

Re: [clamav-users] Problem with private mirror and cld, inc files [ In reply to ]

clamav-users at lists

Jan 27, 2021, 6:59 AM

Post #6 of 8 (656 views)

Permalink

As long as you are using fresh clam, which handles incremental delta updates, and the instances are running correctly (not stuck in a loop or something silly), the best place to get updates is from database.clamav.net <http://database.clamav.net/>

> On Jan 27, 2021, at 5:45 AM, Vangelis Katsikaros via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Thanks for the actions Joel, my IPs are now unblocked.
>
> I'd be interested to continue the discussion on
> a) how to find the main & daily .inc and .cld files
> b) why in ubuntu 18.04 there is a systemd condition for the clamav-daemon to run, but the files are not available for download
> I am afraid that in the long run my service could again DoS database.clamav.net <http://database.clamav.net/> and it would be prudent for me to proceed with the private mirror solution.
>
> Regards
> Vangelis
>
> On Tue, Jan 26, 2021 at 7:20 PM Joel Esler (jesler) <jesler@cisco.com <mailto:jesler@cisco.com>> wrote:
> Let me remove the block for your IP and see what we see.
>
>> On Jan 26, 2021, at 12:17 PM, Vangelis Katsikaros <vkatsikaros@gmail.com <mailto:vkatsikaros@gmail.com>> wrote:
>>
>> Hi Joel, thanks for the quick response. We already download once every hour (the default ubuntu 18.04 behavior). However, we are using auto scaling and we might be running a large number of EC2 instances (a few hundreds), that could try to download simultaneously. To further scale without issues it seemed quite easy (and it was easy in practice) to use a private mirror. The system conditions and the cld & inc files is the only blocker to make this work :/
>>
>> On Tue, Jan 26, 2021 at 7:00 PM Joel Esler (jesler) <jesler@cisco.com <mailto:jesler@cisco.com>> wrote:
>> Why wouldn’t you just download less often? Instead of doing all of this? We publish updates once a day, there’s no need to download the entire package once a second.
>>
>> --
>> Joel Esler
>> Manager, Communities Division
>> Cisco Talos Intelligence Group
>> http://www.talosintelligence.com <http://www.talosintelligence.com/> | https://www.snort.org <https://www.snort.org/>
>>
>>> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>>
>>> Hi
>>>
>>> I am using Ubuntu 18.04 and recently found out that downloaind clamav files was blocked by cloudflare. Based on https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html <https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html> I contacted Joel Esler (many thanks for the quick reply) and inform me that we were downloading too often.
>>>
>>> I tried to fix the situation by introducing a private mirror, but I ran into problems with cld and inc files and systemd conditions for the clamav-daemon service. I'd like to ask what are the "cld" and "inc" files, and how to download them in order to meet the systemd condition that requires them to be on disk?
>>>
>>> I did the following steps:
>>> 1) Created a private mirror as described in https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors> and mirrored all "cvd" files.
>>>
>>> 2) Cloud-init ensures that on boot the files are downloaded from the private mirror and are always available locally. Tthen cloud-init restarts the clamav daemon.
>>>
>>> 3) Configured freshclam with
>>> PrivateMirror https://my-clamav-mirror/clamav <https://my-clamav-mirror/clamav>
>>> ScriptedUpdates no
>>> Checks 3
>>>
>>> 4) Now freshclam works fine, however the clamav daemon does not, even if the cvd are available locally.
>>> $ sudo systemctl status clamav-daemon.service
>>> ? clamav-daemon.service - Clam AntiVirus userspace daemon
>>> Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
>>> Drop-In: /etc/systemd/system/clamav-daemon.service.d
>>> ??extend.conf
>>> Active: inactive (dead)
>>> Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min ago
>>> ?? ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc} was not met
>>> Docs: man:clamd(8)
>>> man:clamd.conf(5)
>>> https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
>>>
>>> The systemd unit file has 2 ConditionPathExistsGlob that require the additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be available, for the service to start.
>>> $ cat /lib/systemd/system/clamav-daemon.service
>>> [Unit]
>>> Description=Clam AntiVirus userspace daemon
>>> Documentation=man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
>>> # Check for database existence
>>> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
>>> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>>>
>>> [Service]
>>> ExecStart=/usr/sbin/clamd --foreground=true
>>> # Reload the database
>>> ExecReload=/bin/kill -USR2 $MAINPID
>>> StandardOutput=syslog
>>> TimeoutStartSec=420
>>>
>>> [Install]
>>> WantedBy=multi-user.target
>>>
>>>
>>> 5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not available to download from the main download site
>>>
>>> $ curl --head -s https://database.clamav.net/daily.cld <https://database.clamav.net/daily.cld> | grep HTTP
>>> HTTP/2 404
>>> $ curl --head -s https://database.clamav.net/daily.inc <https://database.clamav.net/daily.inc> | grep HTTP
>>> HTTP/2 404
>>> $ curl --head -s https://database.clamav.net/main.cld <https://database.clamav.net/main.cld> | grep HTTP
>>> HTTP/2 404
>>> $ curl --head -s https://database.clamav.net/main.inc <https://database.clamav.net/main.inc> | grep HTTP
>>> HTTP/2 404
>>>
>>>
>>> 6) I ran the suggested clamdownloader.pl <http://clamdownloader.pl/> (again from https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>) but I don't see any
>>> $ ls -al /tmp/clam
>>> total 222848
>>> drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 26 16:27 .
>>> drwxrwxrwt 93 root root 12288 Jan 26 16:27 ..
>>> -rw-rw-r-- 1 ubuntu ubuntu 296388 Sep 19 2019 bytecode.cvd
>>> -rw-rw-r-- 1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
>>> -rw-rw-r-- 1 ubuntu ubuntu 42 Jan 26 16:27 dns.txt
>>> -rw-rw-r-- 1 ubuntu ubuntu 117859675 Nov 25 2019 main.cvd
>>> drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 26 16:27 temp
>>>
>>> but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc" files.
>>>
>>> Repeating the question in the beginning: I'd like to ask what are the "cld" and "inc" files, and how to download in my private mirror in order to meet the systemd condition that requires them to be on disk?
>>>
>>> Regards
>>> Vangelis Katsikaros
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>>
>>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

Re: [clamav-users] Problem with private mirror and cld, inc files [ In reply to ]

kdeugau at vianet

Jan 27, 2021, 7:24 AM

Post #7 of 8 (656 views)

Permalink

Vangelis Katsikaros via clamav-users wrote:
> Hi Joel, thanks for the quick response. We already download once every
> hour (the default ubuntu 18.04 behavior). However, we are using auto
> scaling and we might be running a large number of EC2 instances (a few
> hundreds), that could try to download simultaneously. To further scale
> without issues it seemed quite easy (and it was easy in practice) to use
> a private mirror. The system conditions and the cld & inc files is the
> only blocker to make this work :/

It sounds like you need to add a freshclam call as one of the last bits
of spinning up new VM/container instances just before actually
activating one. [.edit: I saw downthread you've supposedly already done
this; clearly something is going wrong close by that action.] Or
include a reference set of signature files and let freshclam run its
updates in the VM/container as normally configured (with PrivateMirror,
you should be able to get away with setting freshclam's update check
interval down to 10-15 minutes - or possibly less).

>> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users
>> <clamav-users@lists.clamav.net
>> <mailto:clamav-users@lists.clamav.net>> wrote:
>>
>> Hi
>>
>> I am using Ubuntu 18.04 and recently found out that downloaind
>> clamav files was blocked by cloudflare. Based on
>> https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html
>> I contacted Joel Esler (many thanks for the quick reply) and
>> inform me that we were downloading too often.

>> I did the following steps:
>> 1) Created a private mirror as described in
>> https://www.clamav.net/documents/private-local-mirrors and
>> mirrored all "cvd" files.
>>
>> 2) Cloud-init ensures that on boot the files are downloaded from
>> the private mirror and are always available locally. Tthen
>> cloud-init restarts the clamav daemon.

Double-check this step. If the systemd condition is failing, or clamd
is failing to find the files, then the files aren't getting where they
need to.

>> The systemd unit file has 2 ConditionPathExistsGlob that require
>> the additional files "main.cld", "main.inc", "daily.cld",
>> "daily.inc" to be available, for the service to start.
>> $ cat /lib/systemd/system/clamav-daemon.service
>> [Unit]
>> Description=Clam AntiVirus userspace daemon
>> Documentation=man:clamd(8) man:clamd.conf(5)
>> https://www.clamav.net/documents/
>> # Check for database existence
>> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
>> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}

Unless I misread, that means "at least one of main.cvd, main.cld, or
main.inc, plus at least one of daily.cvd, daily.cld, or daily.inc"; you
should not need more. If you're using the stock signature files you'll
usually see bytecode.(cvd|cld|inc) as well, but it's not considered
critical.

I'm not sure about the .inc files (sounds a bit familiar, legacy
historic files?). The .cvd files are the compressed version of the .cld
files - for long-running systems using the standard incremental updates
you'll usually end up with .cld files, but I expect in your case you'll
have .cvd files. IIRC at one time you did actually end up with both -
or all three - but checking several instances locally I only see the
.cld files.

TBH, re-reading https://www.clamav.net/documents/private-local-mirrors
it seems to me you'd be better off including a baseline set of .cvd or
.cld files in your instance image (periodically updated to cut down the
time for an instance to finish going live), and using option 1 - using a
proxy server to let freshclam just do its incremental updates on your
VPS/container instances without hitting the public mirror network each time.

-kgd

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Problem with private mirror and cld, inc files [ In reply to ]

clamav-users at lists

Jan 29, 2021, 2:14 AM

Post #8 of 8 (656 views)

Permalink

Hi Kris

On Wed, Jan 27, 2021 at 5:25 PM Kris Deugau <kdeugau@vianet.ca> wrote:

> >> 2) Cloud-init ensures that on boot the files are downloaded from
> >> the private mirror and are always available locally. Tthen
> >> cloud-init restarts the clamav daemon.
>
> Double-check this step. If the systemd condition is failing, or clamd
> is failing to find the files, then the files aren't getting where they
> need to.
>

_facepalm_ Indeed, cloud init and freshclam were supposed to have the same
url, however cloud init had a typo.

* cloud init was not downloading the files
* clamav daemon was not starting due to the systemd condition
* freshclam was downloaidng the files. It could not notify clamav deamon
because it was already stopped
* then I was sshing in the machine, misread the systemd glob (as you
mention below) thinking it was an && not ||. And then I wrongly blamed the
systemd condition for the problem.

Thanks for pointing out what I missed completely.

> >> The systemd unit file has 2 ConditionPathExistsGlob that require
> >> the additional files "main.cld", "main.inc", "daily.cld",
> >> "daily.inc" to be available, for the service to start.
> >> $ cat /lib/systemd/system/clamav-daemon.service
> >> [Unit]
> >> Description=Clam AntiVirus userspace daemon
> >> Documentation=man:clamd(8) man:clamd.conf(5)
> >> https://www.clamav.net/documents/
> >> # Check for database existence
> >> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
> >> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>
> Unless I misread, that means "at least one of main.cvd, main.cld, or
> main.inc, plus at least one of daily.cvd, daily.cld, or daily.inc"; you
> should not need more. If you're using the stock signature files you'll
> usually see bytecode.(cvd|cld|inc) as well, but it's not considered
> critical.
>

Regards
Vangelis

Mailing List Archive

Attached Files:

Attached Files:

Attached Files: