Mailing List Archive

[clamav-users] Unable to download clamav cvd file using google cloud python function
Hi All,

I am getting error http client 403 where I have deployed google cloud function using python to download the daily clamav virus definitions from http://database.clamav.net.

It seems clamav is blocking access to google cloud IPs and denied the requests.

Has anybody faced similar issues and any workarounds available? Can Clamav enable our IPs and allow to download daily virus definition files?

Any help is greatly appreciated !!

Error log from google cloud function.

clamav_definitions_updaterc7jdhxftjt82 Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise raise value File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 149, in view_func function(data, context) File "/workspace/main.py", line 42, in get_latest_clamav_definitions downloaded_file = download_file_to_temp(url) File "/workspace/main.py", line 18, in download_file_to_temp response.raise_for_status() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://database.clamav.net/daily.cvd



Regards,
Ganesh Kachare








C2 General
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
> I am getting error? http client 403 where I have deployed google cloud
> function using python to download the daily clamav virus definitions
> from http://database.clamav.net <http://database.clamav.net>.
>
ClamAV have implemented rate-limiting and restrictions because some
people were downloading updates far too frequently from AWS and Google
Cloud servers.

See: https://lists.clamav.net/pipermail/clamav-users/2021-March/010559.html

You should use FreshClam to download the updates. If you are running a
private mirror, then use this tool to refresh your mirror:
https://github.com/micahsnyder/cvdupdate


--
Paul
Paul Smith Computer Services
support@pscs.co.uk - 01484 855800



--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
Can't believe how many people haven't been read this forum...

-Al-

On Mon, Mar 08, 2021 at 11:23 AM, Joel Esler via clamav-users (jesler) wrote:
> As a result of events documented in places here:
> https://lists.clamav.net/pipermail/clamav-users/2021-March/010577.html <https://lists.clamav.net/pipermail/clamav-users/2021-March/010577.html>
> and
> https://lists.clamav.net/pipermail/clamav-users/2021-March/010543.html <https://lists.clamav.net/pipermail/clamav-users/2021-March/010543.html>
>
> We’ve been forced to take emergency measures to protect the ClamAV environment.
>
> Please Immediately switch to using Freshclam or https://github.com/micahsnyder/cvdupdate <https://github.com/micahsnyder/cvdupdate> to update your AV definitions.
>
> Sorry for the inconvenience, but we are currently in emergency mode and have to make several drastic changes over the several days.
>
> Joel Esler
> Manager, Communities Division
> Cisco Talos Intelligence Group
> http://www.talosintelligence.com <http://www.talosintelligence.com/> | https://www.snort.org <https://www.snort.org/>

On Wed, Mar 10, 2021 at 02:07 AM, Kachare, Ganesh, Vodafone Group (External) via clamav-users wrote:
> Hi All,
>
> I am getting error http client 403 where I have deployed google cloud function using python to download the daily clamav virus definitions from http://database.clamav.net <http://database.clamav.net/>.
>
> It seems clamav is blocking access to google cloud IPs and denied the requests.
>
> Has anybody faced similar issues and any workarounds available? Can Clamav enable our IPs and allow to download daily virus definition files?
>
> Any help is greatly appreciated !!
>
> Error log from google cloud function.
>
> clamav_definitions_updaterc7jdhxftjt82 Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise raise value File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 149, in view_func function(data, context) File "/workspace/main.py", line 42, in get_latest_clamav_definitions downloaded_file = download_file_to_temp(url) File "/workspace/main.py", line 18, in download_file_to_temp response.raise_for_status() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://database.clamav.net/daily.cvd <http://database.clamav.net/daily.cvd>
>
>
>
> Regards,
> Ganesh Kachare
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
On Wednesday 10 March 2021 06:16:55 Al Varnell via clamav-users wrote:

> Can't believe how many people haven't been read this forum...
>
> -Al-

I hate forums Al, all that BS just to get logged in is a pita, and ALL my
email is automated. If you want me to see it, please post it here.

[...]

Take care and stay well, Al. BTW, your posts are one of two folks at
cisco I get from about 70 lists that turn the "Signed by" checker green
in kmail from TDE. Not kde, but TDE. Congratulations for doing it right.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
Hi there,

On Wed, 10 Mar 2021, Kachare, Ganesh, Vodafone Group (External) via clamav-users wrote:

> I am getting error http client 403 ...

I suspect that some people are subscribing to the mailing list when
they belatedly find that the non-preferred way of getting the ClamAV
signatures that they've been using has suddenly stopped working.

Is there not an argument for (at least temporarily) putting something
in the mailing list manager's 'welcome' message which asks them to
read some of Joel's recent messages?

Not only could it prevent a lot of unnecessary noise on the mailing
list, it could also help people to avoid looking dumb - and that's
hardly ever a bad thing.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
I wonder how many "ordinary" users of ClamAV are giving up on using it after getting permanent 403s. I would imagine there are lots of people who don't pursue the issue. They may even tell others that ClamAV is unreliable (which would tarnish its reputation).


On Wed, 10 Mar 2021 11:58:13 +0000 (GMT)
"G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> On Wed, 10 Mar 2021, Kachare, Ganesh, Vodafone Group (External) via clamav-users wrote:
>
> > I am getting error http client 403 ...
>
> I suspect that some people are subscribing to the mailing list when
> they belatedly find that the non-preferred way of getting the ClamAV
> signatures that they've been using has suddenly stopped working.
>
> Is there not an argument for (at least temporarily) putting something
> in the mailing list manager's 'welcome' message which asks them to
> read some of Joel's recent messages?
>
> Not only could it prevent a lot of unnecessary noise on the mailing
> list, it could also help people to avoid looking dumb - and that's
> hardly ever a bad thing.
>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
>On Wed, 10 Mar 2021, Kachare, Ganesh, Vodafone Group (External) via clamav-users wrote:
>
>>I am getting error http client 403 ...

On 10.03.21 11:58, G.W. Haywood via clamav-users wrote:
>I suspect that some people are subscribing to the mailing list when
>they belatedly find that the non-preferred way of getting the ClamAV
>signatures that they've been using has suddenly stopped working.
>
>Is there not an argument for (at least temporarily) putting something
>in the mailing list manager's 'welcome' message which asks them to
>read some of Joel's recent messages?
>
>Not only could it prevent a lot of unnecessary noise on the mailing
>list, it could also help people to avoid looking dumb - and that's
>hardly ever a bad thing.

at this level a notification at the main clamav page may be welcome.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
(R)etry, (A)bort, (C)ancer

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
On 10/03/2021 17:00, Paul Kosinski via clamav-users wrote:
> I wonder how many "ordinary" users of ClamAV are giving up on using it after getting permanent 403s. I would imagine there are lots of people who don't pursue the issue. They may even tell others that ClamAV is unreliable (which would tarnish its reputation).

Indeed. There does seem to be a view from some people here that anyone
using ClamAV should be regularly updating, monitoring this list,
monitoring blogs, etc. Ordinary people just don't do that.

I expect many will just be thinking that the database servers are
broken, and are waiting for them to recover on their own (as they've
done in the past) and they'll eventually go elsewhere.

The change should really be published everywhere possible - at least in
big letters on the ClamAV home page, and possibly including going to
popular computer press, etc.

A blog article (which is actually very hard to find) or announcement
list post (which is even harder to find) which vaguely says that
databases won't be tested on older versions isn't quite the same as a
home page announcement that old versions & wget just won't work any more!

Of course, people have limited rights to complain - it's not like we're
paying for it.

--
Paul


--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
Citeren Paul Smith via clamav-users <clamav-users@lists.clamav.net>:

> Indeed. There does seem to be a view from some people here that
> anyone using ClamAV should be regularly updating, monitoring this
> list, monitoring blogs, etc. Ordinary people just don't do that.

I wonder how many ordinary users are actually *not* using freshclam
for updates. Pretty much every major distribution I know of will setup
ClamAV in a way that updates are handled through freshclam. Unless you
have a *really* fast internet connection, downloading the full .cvd
files will be a lot slower than downloading a .cdiff and applying
that. I think most ordinary users with reasonably recent ClamAV
installations won't even be aware what is happening at the moment.
Having said that, I agree a prominent message about the obsolescence
of ClamAV < 0.100 and the current download limits is desireable.



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
On 10/03/2021 18:42, Arjen de Korte via clamav-users wrote:
>
>> Indeed. There does seem to be a view from some people here that
>> anyone using ClamAV should be regularly updating, monitoring this
>> list, monitoring blogs, etc. Ordinary people just don't do that.
>
> I wonder how many ordinary users are actually *not* using freshclam
> for updates. Pretty much every major distribution I know of will setup
> ClamAV in a way that updates are handled through freshclam.

Yes - but many people won't be using ClamAV from (reasonably up-to-date)
Linux distributions... Many will be using ClamWin, or ClamAV otherwise
installed on Windows, or on a NAS, or whatever. Those could well be
using old versions or unusual installations without necessarily
realising what's going on. All they'll know is that suddenly it's
stopped working.

--
Paul


--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
On Mar 10, 2021, at 12:31 PM, Paul Smith via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

On 10/03/2021 17:00, Paul Kosinski via clamav-users wrote:
I wonder how many "ordinary" users of ClamAV are giving up on using it after getting permanent 403s. I would imagine there are lots of people who don't pursue the issue. They may even tell others that ClamAV is unreliable (which would tarnish its reputation).

Indeed. There does seem to be a view from some people here that anyone using ClamAV should be regularly updating, monitoring this list, monitoring blogs, etc. Ordinary people just don't do that.

I expect many will just be thinking that the database servers are broken, and are waiting for them to recover on their own (as they've done in the past) and they'll eventually go elsewhere.

The change should really be published everywhere possible - at least in big letters on the ClamAV home page, and possibly including going to popular computer press, etc.

A blog article (which is actually very hard to find) or announcement list post (which is even harder to find) which vaguely says that databases won't be tested on older versions isn't quite the same as a home page announcement that old versions & wget just won't work any more!

Of course, people have limited rights to complain - it's not like we're paying for it.

We are going to be writing a couple blog posts in the coming days. I haven’t had the time to sit down and do it.
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
On Mar 10, 2021, at 1:42 PM, Arjen de Korte via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

Citeren Paul Smith via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>:

Indeed. There does seem to be a view from some people here that anyone using ClamAV should be regularly updating, monitoring this list, monitoring blogs, etc. Ordinary people just don't do that.

I wonder how many ordinary users are actually *not* using freshclam for updates. Pretty much every major distribution I know of will setup ClamAV in a way that updates are handled through freshclam. Unless you have a *really* fast internet connection, downloading the full .cvd files will be a lot slower than downloading a .cdiff and applying that. I think most ordinary users with reasonably recent ClamAV installations won't even be aware what is happening at the moment. Having said that, I agree a prominent message about the obsolescence of ClamAV < 0.100 and the current download limits is desireable.

Traffic wise? About 80% of people aren’t using Freshclam or cvdupdate. What that equates to in real person numbers? I have not done that calculation.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com | https://www.snort.org
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
To give everyone a frame of reference. This is what a Cdiff release and download cycle should look like:


[cid:311D041A-A699-48A6-BB74-8523A3927866]

Big influx right in the morning when we publish, and then peaks on the top and bottom of the hour every hour throughout a 24 hour period, (people having a cron job that runs at the top of every hour throughout the day) Theoretically speaking, at the end of 24 hours, the line should go to zero, it never will, because of new installs that download a bunch of cdiffs right in a row and things like that. But I I look between the peaks find people like this:

[cid:B0884332-310A-4C6F-9960-A0A8DB6C2B0D]

100 CDIFFs or so behind, and they download it nearly 2k times in a row? Why? This is not a partial download either. It’s the full file. Stuck cron?

Or this single IP:

[cid:AE797960-535D-44D1-AB4F-7C5823B5BBF2]

Who in the past 24 hours has created 22.17M file downloads all by themselves from a single IP. (The main.cvd btw)

It’s these bad apples that have ruined the basket for everyone. I can’t play wack-a-mole with single IPs or even whole ASNs.

Multiply this one IP above x thousands, and you see the volume I am dealing with. But that graph at the top there is from yesterday, and it’s much better. This is what we are aiming for. We’ve reduced transferred data by 60% by cutting back on abusers.

Like I said, I’ll be writing a blog post about this, but just to show you guys what I am dealing with:

[cid:D66E6145-0352-45EA-8579-5353C85C15F1]

In the past 72 hours, this is what our event graphs look like. Big drop offs and increases are attributed to the constant adjustment I am doing to find the right balance.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com | https://www.snort.org

On Mar 10, 2021, at 3:30 PM, Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:



On Mar 10, 2021, at 12:31 PM, Paul Smith via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

On 10/03/2021 17:00, Paul Kosinski via clamav-users wrote:
I wonder how many "ordinary" users of ClamAV are giving up on using it after getting permanent 403s. I would imagine there are lots of people who don't pursue the issue. They may even tell others that ClamAV is unreliable (which would tarnish its reputation).

Indeed. There does seem to be a view from some people here that anyone using ClamAV should be regularly updating, monitoring this list, monitoring blogs, etc. Ordinary people just don't do that.

I expect many will just be thinking that the database servers are broken, and are waiting for them to recover on their own (as they've done in the past) and they'll eventually go elsewhere.

The change should really be published everywhere possible - at least in big letters on the ClamAV home page, and possibly including going to popular computer press, etc.

A blog article (which is actually very hard to find) or announcement list post (which is even harder to find) which vaguely says that databases won't be tested on older versions isn't quite the same as a home page announcement that old versions & wget just won't work any more!

Of course, people have limited rights to complain - it's not like we're paying for it.

We are going to be writing a couple blog posts in the coming days. I haven’t had the time to sit down and do it.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
"I can’t play wack-a-mole with single IPs or even whole ASNs."

Does Cloudflare have the iptables hashlimit filter (or the equivalent) available?



On Wed, 10 Mar 2021 22:29:41 +0000
"Joel Esler \(jesler\) via clamav-users" <clamav-users@lists.clamav.net> wrote:

> To give everyone a frame of reference. This is what a Cdiff release and download cycle should look like:
>
>
> [cid:311D041A-A699-48A6-BB74-8523A3927866]
>
> Big influx right in the morning when we publish, and then peaks on the top and bottom of the hour every hour throughout a 24 hour period, (people having a cron job that runs at the top of every hour throughout the day) Theoretically speaking, at the end of 24 hours, the line should go to zero, it never will, because of new installs that download a bunch of cdiffs right in a row and things like that. But I I look between the peaks find people like this:
>
> [cid:B0884332-310A-4C6F-9960-A0A8DB6C2B0D]
>
> 100 CDIFFs or so behind, and they download it nearly 2k times in a row? Why? This is not a partial download either. It’s the full file. Stuck cron?
>
> Or this single IP:
>
> [cid:AE797960-535D-44D1-AB4F-7C5823B5BBF2]
>
> Who in the past 24 hours has created 22.17M file downloads all by themselves from a single IP. (The main.cvd btw)
>
> It’s these bad apples that have ruined the basket for everyone. I can’t play wack-a-mole with single IPs or even whole ASNs.
>
> Multiply this one IP above x thousands, and you see the volume I am dealing with. But that graph at the top there is from yesterday, and it’s much better. This is what we are aiming for. We’ve reduced transferred data by 60% by cutting back on abusers.
>
> Like I said, I’ll be writing a blog post about this, but just to show you guys what I am dealing with:
>
> [cid:D66E6145-0352-45EA-8579-5353C85C15F1]
>
> In the past 72 hours, this is what our event graphs look like. Big drop offs and increases are attributed to the constant adjustment I am doing to find the right balance.
>
> --
> Joel Esler
> Manager, Communities Division
> Cisco Talos Intelligence Group
> http://www.talosintelligence.com | https://www.snort.org
>
> On Mar 10, 2021, at 3:30 PM, Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
>
>
>
> On Mar 10, 2021, at 12:31 PM, Paul Smith via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
>
> On 10/03/2021 17:00, Paul Kosinski via clamav-users wrote:
> I wonder how many "ordinary" users of ClamAV are giving up on using it after getting permanent 403s. I would imagine there are lots of people who don't pursue the issue. They may even tell others that ClamAV is unreliable (which would tarnish its reputation).
>
> Indeed. There does seem to be a view from some people here that anyone using ClamAV should be regularly updating, monitoring this list, monitoring blogs, etc. Ordinary people just don't do that.
>
> I expect many will just be thinking that the database servers are broken, and are waiting for them to recover on their own (as they've done in the past) and they'll eventually go elsewhere.
>
> The change should really be published everywhere possible - at least in big letters on the ClamAV home page, and possibly including going to popular computer press, etc.
>
> A blog article (which is actually very hard to find) or announcement list post (which is even harder to find) which vaguely says that databases won't be tested on older versions isn't quite the same as a home page announcement that old versions & wget just won't work any more!
>
> Of course, people have limited rights to complain - it's not like we're paying for it.
>
> We are going to be writing a couple blog posts in the coming days. I haven’t had the time to sit down and do it.
>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
On 10/03/2021 22:29, Joel Esler (jesler) via clamav-users wrote:
> 100 CDIFFs or so behind, and they download it nearly 2k times in a
> row?  Why?  This is not a partial download either.  It’s the full
> file.  Stuck cron?
>
>
> Who in the past 24 hours has created 22.17M file downloads /all by
> themselves/ from a single IP. (The main.cvd btw)

You *may* be forgetting NAT.

Eg, it's possible the first one is a network of a few thousand computers
going through a NAT firewall where each of them has had an old daily.cvd
copied onto them in an internal release cycle or something, so each of
the computers on that network is trying to download a backlog of CDIFFs.
(Or maybe another problem stopping the updates has been discovered and
fixed, or something)

I'm not saying it is, but it may be. If you are only analysing by IP
address, NAT will innocently cause strange results.

--
Paul



--


Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
On Thu, 11 Mar 2021, Paul Smith via clamav-users wrote:

> On 10/03/2021 22:29, Joel Esler (jesler) via clamav-users wrote:
>> 100 CDIFFs or so behind, and they download it nearly 2k times in a row?
>>  Why?  This is not a partial download either.  It’s the full file.  Stuck
>> cron?
>>
>>
>> Who in the past 24 hours has created 22.17M file downloads /all by
>> themselves/ from a single IP. (The main.cvd btw)
>
> You *may* be forgetting NAT.
>
> Eg, it's possible the first one is a network of a few thousand computers
> going through a NAT firewall where each of them has had an old daily.cvd
> copied onto them in an internal release cycle or something, so each of the
> computers on that network is trying to download a backlog of CDIFFs. (Or
> maybe another problem stopping the updates has been discovered and fixed, or
> something)
>
> I'm not saying it is, but it may be. If you are only analysing by IP address,
> NAT will innocently cause strange results.

I'm thinking short-lived virtual machines that install clamav on first boot.
I guess the thinking is that "Since the cvd files are large and change
"frequently"* it isn't worth installing them in the image;
the running image can download the current versions ..."
*Daily isn't that frequent, but they think it is.
Of course a short-lived vm should be using an external clamd server,
if it needs AV at all.

If this *is* the problem, freshclam isn't the solution :-(
You will need either to persuade the owners to think about how and why
they are attempting to run clamav, or perhaps persuade the suppliers
of the container images not to include a local clam service.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
Citeren Paul Smith via clamav-users <clamav-users@lists.clamav.net>:

> You *may* be forgetting NAT.
>
> Eg, it's possible the first one is a network of a few thousand
> computers going through a NAT firewall where each of them has had an
> old daily.cvd copied onto them in an internal release cycle or
> something, so each of the computers on that network is trying to
> download a backlog of CDIFFs. (Or maybe another problem stopping the
> updates has been discovered and fixed, or something)

In that case, the organisation behind that NAT should provide a local
mirror. There is no excuse for running thousands of systems on a
single IP (if that is even possible) and not use a local mirror.

> I'm not saying it is, but it may be. If you are only analysing by IP
> address, NAT will innocently cause strange results.

There is nothing innocent about the above scenario, it's either
negliance or incompetence.




_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
Hi there,

On Thu, 11 Mar 2021, Paul Smith via clamav-users wrote:
> On 10/03/2021 22:29, Joel Esler (jesler) via clamav-users wrote:
>>
>> ... in the past 24 hours has created 22.17M file downloads
>> /all by themselves/ from a single IP. (The main.cvd btw)
>
> ... internal release cycle or something ... or something
> ... NAT will innocently cause strange results.

The (for want of a better word) history of the Internet is littered
with parallels to the Tacoma Narrows Bridge incident, or as those of
us in the engineering professions often say [****].

Large networks can seem to take on a life and character all of their
own, but in the end it's all susceptible to reason. Every IP address
should have a working abuse reporting address which can be found by a
'whois' query. For example for clamav.net:

$ whois `dig +short clamav.net` | grep -i abuse
OrgAbuseHandle: TALOS-ARIN
OrgAbuseName: Talos Operations
OrgAbusePhone: +1-727-540-3152
OrgAbuseEmail: talos.ops@cisco.com
OrgAbuseRef: https://rdap.arin.net/registry/entity/TALOS-ARIN
$

It _should_ be trivial to report the abuse to the address given by the
whois query and that should get the abuse stopped fairly promptly. If
it doesn't, then it's not a working abuse reporting address. Large
sections of the Internet address space either don't have working abuse
addresses, or their operators are in league with criminals and make a
token response which is ineffective, or they're just plain incompetent
and do nothing that's effective. To me that all means 'not working'.
If an IP doesn't have a working abuse reporting address, in my view
prima facie there's a case that it should be permanently firewalled.

Joel, have you tried reporting to abuse addresses at least for some of
the worst offenders? Do you have a large body of low-grade offenders
which make you feel you don't want to go to the office in the morning?
Like many system administrators I also have that tee-shirt.

Less than 5% of the mail that my mail systems see is genuine. More
than 95% is in some way abusive. It's almost overwhelming, and it's
impractical to deal with it all manually, so over the last few years
I've developed an automatic abuse reporting system (of which clamd is
an integral part) which not only sends reports to the abuse addresses
from 'whois', but also uses other ways to find them, and, depending on
the kind of abuse, can report to ClamAV, Sanesecurity, Securiteinfo,
and for example abuse clearing houses run by various government and
law enforcement agencies for what that's worth. Of course it blocks
the abusive messages too - that's almost a side-effect. I tend to use
TEMPFAIL rather than REJECT and/or firewall - it's configurable - so
exceedingly spammy providers like Gm@il and M1cro$oft use up more of
their resources but the option simply to firewall the IP is available.

Unfortunately, automatic systems have sometimes had a reputation for
making the problem worse, not better [****]. It's important to avoid
that, which I think I've managed. Very little of what I see is what
you would call malware, and even less is automatically identified as
such, so only about 1% of reports go to the ClamaAV signature team at
present but at least it gives automatic feedback. Perhaps the guys at
Sanesecurity and Securiteinfo can chip in with an opinion? You get a
sizeable fraction of the reports and any feedback that you can give me
will be very valuable. It seems not easy to get. It's been a lot of
work, and there's a lot left to do, but it's been worth it to be able
to return serve thousands of times every day with little extra effort.

Joel, I'm sure it wouldn't be hard to adapt the ideas to other systems
if you'd be interested in exploring that.

[***] Roughly translated, "I never thought of that".

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
Yup. and that’s why people are getting 429’s

Sent from my ? iPhone

> On Mar 10, 2021, at 23:01, Paul Kosinski <clamav-users@iment.com> wrote:
>
> ?"I can’t play wack-a-mole with single IPs or even whole ASNs."
>
> Does Cloudflare have the iptables hashlimit filter (or the equivalent) available?
>
>
>
>> On Wed, 10 Mar 2021 22:29:41 +0000
>> "Joel Esler \(jesler\) via clamav-users" <clamav-users@lists.clamav.net> wrote:
>>
>> To give everyone a frame of reference. This is what a Cdiff release and download cycle should look like:
>>
>>
>> [cid:311D041A-A699-48A6-BB74-8523A3927866]
>>
>> Big influx right in the morning when we publish, and then peaks on the top and bottom of the hour every hour throughout a 24 hour period, (people having a cron job that runs at the top of every hour throughout the day) Theoretically speaking, at the end of 24 hours, the line should go to zero, it never will, because of new installs that download a bunch of cdiffs right in a row and things like that. But I I look between the peaks find people like this:
>>
>> [cid:B0884332-310A-4C6F-9960-A0A8DB6C2B0D]
>>
>> 100 CDIFFs or so behind, and they download it nearly 2k times in a row? Why? This is not a partial download either. It’s the full file. Stuck cron?
>>
>> Or this single IP:
>>
>> [cid:AE797960-535D-44D1-AB4F-7C5823B5BBF2]
>>
>> Who in the past 24 hours has created 22.17M file downloads all by themselves from a single IP. (The main.cvd btw)
>>
>> It’s these bad apples that have ruined the basket for everyone. I can’t play wack-a-mole with single IPs or even whole ASNs.
>>
>> Multiply this one IP above x thousands, and you see the volume I am dealing with. But that graph at the top there is from yesterday, and it’s much better. This is what we are aiming for. We’ve reduced transferred data by 60% by cutting back on abusers.
>>
>> Like I said, I’ll be writing a blog post about this, but just to show you guys what I am dealing with:
>>
>> [cid:D66E6145-0352-45EA-8579-5353C85C15F1]
>>
>> In the past 72 hours, this is what our event graphs look like. Big drop offs and increases are attributed to the constant adjustment I am doing to find the right balance.
>>
>> --
>> Joel Esler
>> Manager, Communities Division
>> Cisco Talos Intelligence Group
>> http://www.talosintelligence.com | https://www.snort.org
>>
>> On Mar 10, 2021, at 3:30 PM, Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
>>
>>
>>
>> On Mar 10, 2021, at 12:31 PM, Paul Smith via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
>>
>> On 10/03/2021 17:00, Paul Kosinski via clamav-users wrote:
>> I wonder how many "ordinary" users of ClamAV are giving up on using it after getting permanent 403s. I would imagine there are lots of people who don't pursue the issue. They may even tell others that ClamAV is unreliable (which would tarnish its reputation).
>>
>> Indeed. There does seem to be a view from some people here that anyone using ClamAV should be regularly updating, monitoring this list, monitoring blogs, etc. Ordinary people just don't do that.
>>
>> I expect many will just be thinking that the database servers are broken, and are waiting for them to recover on their own (as they've done in the past) and they'll eventually go elsewhere.
>>
>> The change should really be published everywhere possible - at least in big letters on the ClamAV home page, and possibly including going to popular computer press, etc.
>>
>> A blog article (which is actually very hard to find) or announcement list post (which is even harder to find) which vaguely says that databases won't be tested on older versions isn't quite the same as a home page announcement that old versions & wget just won't work any more!
>>
>> Of course, people have limited rights to complain - it's not like we're paying for it.
>>
>> We are going to be writing a couple blog posts in the coming days. I haven’t had the time to sit down and do it.
>>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Unable to download clamav cvd file using google cloud python function [ In reply to ]
There are some ideas that we’re thinking of that mitigate and handle the issues. But all of the ideas require code change, some of the ideas we’re actually investigating with Cloudflare directly to see if we can actually invent a feature.

ALL of the ideas require the future restriction to either Freshclam or other authorized tools that interact with those restrictions.

The days of just scripting python or wget or curl or something are gone, and we have to immediately start moving to metered and careful downloads. Like I said, this INCLUDES changes we have to make ourselves, so there’s work on all sides right now.

On Mar 11, 2021, at 7:20 AM, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

Hi there,

On Thu, 11 Mar 2021, Paul Smith via clamav-users wrote:
On 10/03/2021 22:29, Joel Esler (jesler) via clamav-users wrote:
... in the past 24 hours has created 22.17M file downloads
/all by themselves/ from a single IP. (The main.cvd btw)

... internal release cycle or something ... or something
... NAT will innocently cause strange results.

The (for want of a better word) history of the Internet is littered
with parallels to the Tacoma Narrows Bridge incident, or as those of
us in the engineering professions often say [****].

Large networks can seem to take on a life and character all of their
own, but in the end it's all susceptible to reason. Every IP address
should have a working abuse reporting address which can be found by a
'whois' query. For example for clamav.net<http://clamav.net>:

$ whois `dig +short clamav.net<http://clamav.net>` | grep -i abuse
OrgAbuseHandle: TALOS-ARIN
OrgAbuseName: Talos Operations
OrgAbusePhone: +1-727-540-3152 OrgAbuseEmail: talos.ops@cisco.com<mailto:talos.ops@cisco.com>
OrgAbuseRef: https://rdap.arin.net/registry/entity/TALOS-ARIN
$

It _should_ be trivial to report the abuse to the address given by the
whois query and that should get the abuse stopped fairly promptly. If
it doesn't, then it's not a working abuse reporting address. Large
sections of the Internet address space either don't have working abuse
addresses, or their operators are in league with criminals and make a
token response which is ineffective, or they're just plain incompetent
and do nothing that's effective. To me that all means 'not working'.
If an IP doesn't have a working abuse reporting address, in my view
prima facie there's a case that it should be permanently firewalled.

Joel, have you tried reporting to abuse addresses at least for some of
the worst offenders? Do you have a large body of low-grade offenders
which make you feel you don't want to go to the office in the morning?
Like many system administrators I also have that tee-shirt.

Less than 5% of the mail that my mail systems see is genuine. More
than 95% is in some way abusive. It's almost overwhelming, and it's
impractical to deal with it all manually, so over the last few years
I've developed an automatic abuse reporting system (of which clamd is
an integral part) which not only sends reports to the abuse addresses
from 'whois', but also uses other ways to find them, and, depending on
the kind of abuse, can report to ClamAV, Sanesecurity, Securiteinfo,
and for example abuse clearing houses run by various government and
law enforcement agencies for what that's worth. Of course it blocks
the abusive messages too - that's almost a side-effect. I tend to use
TEMPFAIL rather than REJECT and/or firewall - it's configurable - so
exceedingly spammy providers like Gm@il and M1cro$oft use up more of
their resources but the option simply to firewall the IP is available.

Unfortunately, automatic systems have sometimes had a reputation for
making the problem worse, not better [****]. It's important to avoid
that, which I think I've managed. Very little of what I see is what
you would call malware, and even less is automatically identified as
such, so only about 1% of reports go to the ClamaAV signature team at
present but at least it gives automatic feedback. Perhaps the guys at
Sanesecurity and Securiteinfo can chip in with an opinion? You get a
sizeable fraction of the reports and any feedback that you can give me
will be very valuable. It seems not easy to get. It's been a lot of
work, and there's a lot left to do, but it's been worth it to be able
to return serve thousands of times every day with little extra effort.

Joel, I'm sure it wouldn't be hard to adapt the ideas to other systems
if you'd be interested in exploring that.

[***] Roughly translated, "I never thought of that".

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml