Mailing List Archive

[clamav-users] Downloading CVD files manually..
OK, I've seen the rate limiting post, and that's understandable.

We're running an (old) version of Freshclam every hour, and that's just
started failing with '403 Forbidden' errors. I guess we'll need to
update to a more recent version, which is OK, but not trivial in our
case so may take us a few days to implement properly. (It needs doing,
so I'm not too bothered by having to do it, but it'd be nice to have a
workaround until that can be done).

So, I thought we'd run a private mirror using the instructions here:
https://www.clamav.net/documents/private-local-mirrors as then we
wouldn't have any issues with rate limiting, and would help by reducing
the load on your servers, etc.

Unfortunately, those don't work any more, because "wget
http://database.clamav.net/main.cvd" also fails with a 403 Forbidden
error. I've tried from multiple PCs on multiple networks, and it always
fails.

(Note that we're not getting a 429, so we're not being rate limited,
just blocked totally)

So, your request in your rate limiting post of "2. Consider setting up a
local mirror on your network" is seemingly impossible to do.

Is there any new trick to setting up a local mirror?

(PS - also, on my PC, 'wget' fails, but Chrome can download the updates
absolutely fine, so it's not blocking my IP address, just the use of wget)


--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
Correct. Wget is restricted. I wrote that in my email.

So if you upgrade your version of freshclam/ClamAV, you should be good. Please see our blog post made back in the beginning of February.

Sent from my ? iPhone

> On Mar 6, 2021, at 16:10, Paul Smith via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?OK, I've seen the rate limiting post, and that's understandable.
>
> We're running an (old) version of Freshclam every hour, and that's just started failing with '403 Forbidden' errors. I guess we'll need to update to a more recent version, which is OK, but not trivial in our case so may take us a few days to implement properly. (It needs doing, so I'm not too bothered by having to do it, but it'd be nice to have a workaround until that can be done).
>
> So, I thought we'd run a private mirror using the instructions here: https://www.clamav.net/documents/private-local-mirrors as then we wouldn't have any issues with rate limiting, and would help by reducing the load on your servers, etc.
>
> Unfortunately, those don't work any more, because "wget http://database.clamav.net/main.cvd" also fails with a 403 Forbidden error. I've tried from multiple PCs on multiple networks, and it always fails.
>
> (Note that we're not getting a 429, so we're not being rate limited, just blocked totally)
>
> So, your request in your rate limiting post of "2. Consider setting up a local mirror on your network" is seemingly impossible to do.
>
> Is there any new trick to setting up a local mirror?
>
> (PS - also, on my PC, 'wget' fails, but Chrome can download the updates absolutely fine, so it's not blocking my IP address, just the use of wget)
>
>
> --
>
>
> Paul Smith Computer Services
> Tel: 01484 855800
> Vat No: GB 685 6987 53
>
> Sign up for news & updates at http://www.pscs.co.uk/go/subscribe
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
The immediate crisis that led to Joel restricting the downloads to use freshclam was unexpected.

I'm working on a tool as fast as I'm able to replace the Perl-wget script for those that need to host private mirrors. It will have built-in features to minimize the possibility for abuse.

I'll share it with the list as soon as I'm able.

Best,
Micah
________________________________
From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Paul Smith via clamav-users <clamav-users@lists.clamav.net>
Sent: Saturday, March 6, 2021 1:03:52 PM
To: clamav-users@lists.clamav.net <clamav-users@lists.clamav.net>
Cc: Paul Smith <paul@pscs.co.uk>
Subject: [clamav-users] Downloading CVD files manually..

OK, I've seen the rate limiting post, and that's understandable.

We're running an (old) version of Freshclam every hour, and that's just
started failing with '403 Forbidden' errors. I guess we'll need to
update to a more recent version, which is OK, but not trivial in our
case so may take us a few days to implement properly. (It needs doing,
so I'm not too bothered by having to do it, but it'd be nice to have a
workaround until that can be done).

So, I thought we'd run a private mirror using the instructions here:
https://www.clamav.net/documents/private-local-mirrors as then we
wouldn't have any issues with rate limiting, and would help by reducing
the load on your servers, etc.

Unfortunately, those don't work any more, because "wget
http://database.clamav.net/main.cvd" also fails with a 403 Forbidden
error. I've tried from multiple PCs on multiple networks, and it always
fails.

(Note that we're not getting a 429, so we're not being rate limited,
just blocked totally)

So, your request in your rate limiting post of "2. Consider setting up a
local mirror on your network" is seemingly impossible to do.

Is there any new trick to setting up a local mirror?

(PS - also, on my PC, 'wget' fails, but Chrome can download the updates
absolutely fine, so it's not blocking my IP address, just the use of wget)


--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
On 07/03/2021 00:17, Joel Esler (jesler) wrote:
> Correct. Wget is restricted. I wrote that in my email.
>
> So if you upgrade your version of freshclam/ClamAV, you should be good. Please see our blog post made back in the beginning of February.

Sorry if I sounded a bit harsh, I totally understand the rate limiting etc.

The problem with upgrading is that we use a customised Windows version
of ClamAV, and building ClamAV on Windows is never trivial...

I think I've managed to get Mussels working yesterday after about 4
hours. I'm not sure what I did, but it wouldn't believe that it had a
recipe for clamav_deps which was valid for Windows ("python3 -m mussels
list" returned a blank list, but "python3 -m mussels list -a" showed all
the recipes). Then, suddenly, for no obvious reason, it just started
working properly (just as I'd decided to join the Discord to ask for
help), so I left it building the dependencies overnight

I'm going to give building ClamAV a go today and see how far we get. And
then we've got to test it etc.


The bigger problem was the inability to make a private mirror using the
instructions on the ClamAV site, especially since the announcement said
that we should do just that... If it was possible to make private
mirrors now, then the issue would be much less. Hopefully there'll be a
solution for that soon.

Paul

--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
On 07/03/2021 04:07, Micah Snyder (micasnyd) wrote:
> The immediate crisis that led to Joel restricting the downloads to use
> freshclam was unexpected.
>
> I'm working on a tool as fast as I'm able to replace the Perl-wget
> script for those that need to host private mirrors. It will have
> built-in features to minimize the possibility for abuse.
>
> I'll share it with the list as soon as I'm able.

Thanks for working on this.

In the meantime we're using FreshClam to get the CVDs and are copying
those into the local webserver's DocumentRoot. Not ideal at all because
of the lack of cdiffs, but it seems to work for now.


--
Paul
Paul Smith Computer Services
support@pscs.co.uk - 01484 855800



--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
I'm really happy to hear you're using Mussels, sad to hear it was frustrating at first. Please feel free to document any sharp edges in the github repo so we can try to make it a better experience.
Some good news: we just merged a major update to the clamav cmake build system along with a lot more instructions on how to build & test on Windows.
Please check it out https://github.com/Cisco-Talos/clamav-devel/blob/dev/0.104/INSTALL.cmake.md#windows-builds

Best,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.



> -----Original Message-----
> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> Paul Smith via clamav-users
> Sent: Sunday, March 7, 2021 2:54 AM
> To: Joel Esler (jesler) <jesler@cisco.com>; ClamAV users ML <clamav-
> users@lists.clamav.net>
> Cc: Paul Smith <paul@pscs.co.uk>
> Subject: Re: [clamav-users] Downloading CVD files manually..
>
> On 07/03/2021 00:17, Joel Esler (jesler) wrote:
> > Correct. Wget is restricted. I wrote that in my email.
> >
> > So if you upgrade your version of freshclam/ClamAV, you should be good.
> Please see our blog post made back in the beginning of February.
>
> Sorry if I sounded a bit harsh, I totally understand the rate limiting etc.
>
> The problem with upgrading is that we use a customised Windows version of
> ClamAV, and building ClamAV on Windows is never trivial...
>
> I think I've managed to get Mussels working yesterday after about 4 hours. I'm
> not sure what I did, but it wouldn't believe that it had a recipe for clamav_deps
> which was valid for Windows ("python3 -m mussels list" returned a blank list,
> but "python3 -m mussels list -a" showed all the recipes). Then, suddenly, for no
> obvious reason, it just started working properly (just as I'd decided to join the
> Discord to ask for help), so I left it building the dependencies overnight
>
> I'm going to give building ClamAV a go today and see how far we get. And then
> we've got to test it etc.
>
>
> The bigger problem was the inability to make a private mirror using the
> instructions on the ClamAV site, especially since the announcement said
> that we should do just that... If it was possible to make private
> mirrors now, then the issue would be much less. Hopefully there'll be a
> solution for that soon.
>
> Paul
>
> --
>
>
> Paul Smith Computer Services
> Tel: 01484 855800
> Vat No: GB 685 6987 53
>
> Sign up for news & updates at http://www.pscs.co.uk/go/subscribe
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
On 08/03/2021 05:55, Micah Snyder (micasnyd) wrote:
> I'm really happy to hear you're using Mussels, sad to hear it was frustrating at first. Please feel free to document any sharp edges in the github repo so we can try to make it a better experience.

Thanks. It worked in the end, and wasn't as painful as I'd feared.

The problem with Mussels was weird. I guess it couldn't work out what
platform I was using, as all the recipes were there, just 'not for my
platform'. I had trusted the cookbook and everything.

As I was composing a message to put on the Discord to ask for tips, I
was running through the commands to reproduce the problem, and then
suddenly it worked. So, I have no idea what changed, so I can't really
report a bug or what I did to fix it, because I simply don't know... I
don't know if using a different command prompt messed it up (I use Take
Command by default, not cmd.exe), but now it works OK in Take Command as
well as in cmd.

It would have been useful if 'list -a' (or 'list -a -V') showed whether
cookbooks were trusted or not, and which platform it thought I was
using. 'list -a -V' shows details, but not that.


--
Paul
Paul Smith Computer Services
support@pscs.co.uk - 01484 855800


--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
Dear All,

Thanks already for your feedback!

Find here below the answer I received from QNAP support:

Dear Customer,

Thank you for contacting QNAP support.
The root cause is the size of the update files in the database. Because of
that the automatic updates are failing on the older NAS models. This is due
to the small CPU and memory size.
When you update manually the system get forced to update and will pause
other applications running.
So please try to perform the updates manually one at a time.
I hope this information is at your convenience.


Br,

Tom

Le lun. 8 mars 2021 à 10:14, Paul Smith via clamav-users <
clamav-users@lists.clamav.net> a écrit :

> On 08/03/2021 05:55, Micah Snyder (micasnyd) wrote:
> > I'm really happy to hear you're using Mussels, sad to hear it was
> frustrating at first. Please feel free to document any sharp edges in the
> github repo so we can try to make it a better experience.
>
> Thanks. It worked in the end, and wasn't as painful as I'd feared.
>
> The problem with Mussels was weird. I guess it couldn't work out what
> platform I was using, as all the recipes were there, just 'not for my
> platform'. I had trusted the cookbook and everything.
>
> As I was composing a message to put on the Discord to ask for tips, I
> was running through the commands to reproduce the problem, and then
> suddenly it worked. So, I have no idea what changed, so I can't really
> report a bug or what I did to fix it, because I simply don't know... I
> don't know if using a different command prompt messed it up (I use Take
> Command by default, not cmd.exe), but now it works OK in Take Command as
> well as in cmd.
>
> It would have been useful if 'list -a' (or 'list -a -V') showed whether
> cookbooks were trusted or not, and which platform it thought I was
> using. 'list -a -V' shows details, but not that.
>
>
> --
> Paul
> Paul Smith Computer Services
> support@pscs.co.uk - 01484 855800
>
>
> --
>
>
> Paul Smith Computer Services
> Tel: 01484 855800
> Vat No: GB 685 6987 53
>
> Sign up for news & updates at http://www.pscs.co.uk/go/subscribe
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
Re: [clamav-users] Downloading CVD files manually.. [ In reply to ]
Paul,

I'm not familiar with Take Command. I mostly use powershell in Windows Terminal these days. I am glad to hear it worked in the end.

That's great feedback regarding the "trusted" status. Would you be willing to submit a Github Issue for the project to remind me?

Best,
Micah


> -----Original Message-----
> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> Paul Smith via clamav-users
> Sent: Monday, March 8, 2021 1:08 AM
> To: ClamAV users ML <clamav-users@lists.clamav.net>
> Cc: Paul Smith <paul@pscs.co.uk>
> Subject: Re: [clamav-users] Downloading CVD files manually..
>
> On 08/03/2021 05:55, Micah Snyder (micasnyd) wrote:
> > I'm really happy to hear you're using Mussels, sad to hear it was frustrating
> at first. Please feel free to document any sharp edges in the github repo so
> we can try to make it a better experience.
>
> Thanks. It worked in the end, and wasn't as painful as I'd feared.
>
> The problem with Mussels was weird. I guess it couldn't work out what
> platform I was using, as all the recipes were there, just 'not for my platform'.
> I had trusted the cookbook and everything.
>
> As I was composing a message to put on the Discord to ask for tips, I was
> running through the commands to reproduce the problem, and then
> suddenly it worked. So, I have no idea what changed, so I can't really report a
> bug or what I did to fix it, because I simply don't know... I don't know if using
> a different command prompt messed it up (I use Take Command by default,
> not cmd.exe), but now it works OK in Take Command as well as in cmd.
>
> It would have been useful if 'list -a' (or 'list -a -V') showed whether
> cookbooks were trusted or not, and which platform it thought I was using.
> 'list -a -V' shows details, but not that.
>
>
> --
> Paul
> Paul Smith Computer Services
> support@pscs.co.uk - 01484 855800
>
>
> --
>
>
> Paul Smith Computer Services
> Tel: 01484 855800
> Vat No: GB 685 6987 53
>
> Sign up for news & updates at http://www.pscs.co.uk/go/subscribe
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml