Mailing List Archive

[clamav-users] Not able to use curl to download the cvd files successfully
Dear Clamav support,

May I know why I am not able to use curl to download the cvd files successfully? The cvd files show error code 1020. Thank you.

[cid:06787b26-eacf-44dc-8a05-dc207794583b]
Re: [clamav-users] Not able to use curl to download the cvd files successfully [ In reply to ]
Hi there,

On Mon, 8 Mar 2021, Lo Nelson via clamav-users wrote:

> May I know why I am not able to use curl to download the cvd files
> successfully? The cvd files show error code 1020. Thank you.

https://marc.info/?l=clamav-users&r=1&b=202103&w=2

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Not able to use curl to download the cvd files successfully [ In reply to ]
On 08.03.21 13:48, Lo Nelson via clamav-users wrote:
>May I know why I am not able to use curl to download the cvd files successfully? The cvd files show error code 1020. Thank you.

because of ongoing abuse of mirror servers by those who used curl to
download cvd files. use freshclam instead.

If you run private mirror, there's new tool for you:

https://pypi.org/project/cvdupdate/0.1.0/
https://github.com/micahsnyder/cvdupdate



--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
My mind is like a steel trap - rusty and illegal in 37 states.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Not able to use curl to download the cvd files successfully [ In reply to ]
> From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Lo Nelson via clamav-users <clamav-users@lists.clamav.net>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
> Date: Monday, March 8, 2021 at 8:49 AM
> To: "clamav-users@lists.clamav.net" <clamav-users@lists.clamav.net>
> Cc: Lo Nelson <nelo_1990@hotmail.com>
> Subject: [EXTERNAL] [clamav-users] Not able to use curl to download the cvd files successfully
>
> Dear Clamav support,
>
> May I know why I am not able to use curl to download the cvd files successfully? The cvd files show error code 1020. Thank you.

This is Cloudfare "protecting" the ClamAV website. You can bypass it by sending a fake user agent string, like this:

curl -A "Mozilla/5.0" http://database.clamav.net/daily.cvd --output daily.cvd

or using wget:

wget --user-agent "Mozilla/5.0" http://database.clamav.net/daily.cvd''



Todd A. Aiken

Systems Analyst & Administrator

ITS Department

BISHOP'S UNIVERSITY

2600 College Street

Sherbrooke, Quebec

CANADA J1M 1Z7

--------

"What's going on around here?" - RS

Having a technology issue?

Visit http://octopus.ubishops.ca to place a ticket directly into our ITS work order system.

This is the best way to get your requests to ITS and provide more detailed information for our analysts and technicians.
Re: [clamav-users] Not able to use curl to download the cvd files successfully [ In reply to ]
On Mar 8, 2021, at 9:36 AM, Todd Aiken <todd.aiken@ubishops.ca<mailto:todd.aiken@ubishops.ca>> wrote:

> From: clamav-users <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net>> on behalf of Lo Nelson via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Date: Monday, March 8, 2021 at 8:49 AM
> To: "clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Cc: Lo Nelson <nelo_1990@hotmail.com<mailto:nelo_1990@hotmail.com>>
> Subject: [EXTERNAL] [clamav-users] Not able to use curl to download the cvd files successfully
>
> Dear Clamav support,
>
> May I know why I am not able to use curl to download the cvd files successfully? The cvd files show error code 1020. Thank you.

This is Cloudfare "protecting" the ClamAV website. You can bypass it by sending a fake user agent string, like this:

curl -A "Mozilla/5.0" http://database.clamav.net/daily.cvd --output daily.cvd

or using wget:

wget --user-agent "Mozilla/5.0" http://database.clamav.net/daily.cvd''


No! Don’t “bypass” it.

And “protecting” does not need to be in quotes, it’s quite literally what we are doing. And people doing the above are the problem.

As I said in countless other emails, either use Freshclam or https://github.com/micahsnyder/cvdupdate. The more people that do the above will force us to take drastic measures.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com | https://www.snort.org
Re: [clamav-users] Not able to use curl to download the cvd files successfully [ In reply to ]
Hi there,

On Mon, 8 Mar 2021, Todd Aiken wrote:

> > ... May I know why I am not able to use curl to download the cvd
> > files successfully? The cvd files show error code 1020. ...
>
> This is Cloudfare "protecting" the ClamAV website. You can bypass
> it by sending a fake user agent string, like this:
>
> curl -A "Mozilla/5.0" http://database.clamav.net/daily.cvd --output daily.cvd
> or using wget:
> wget --user-agent "Mozilla/5.0" http://database.clamav.net/daily.cvd''
>
> Todd A. Aiken
> Systems Analyst & Administrator
> ITS Department
> BISHOP'S UNIVERSITY
> 2600 College Street
> Sherbrooke, Quebec
> CANADA J1M 1Z7

I don't understand why you put "protecting" in quotes, because that's
exactly what's happening. There are IPs downloading the entire main
and daily databases several times per second, which can only represent
either malice or breathtaking stupidity. See Joel's post here

https://marc.info/?l=clamav-users&m=161516064110943&w=2

and try not to be stupid.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Not able to use curl to download the cvd files successfully [ In reply to ]
> From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of "Joel Esler (jesler) via clamav-users" <clamav-users@lists.clamav.net>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
> Date: Monday, March 8, 2021 at 9:47 AM
> To: ClamAV users ML <clamav-users@lists.clamav.net>
> Cc: "Joel Esler (jesler)" <jesler@cisco.com>
> Subject: [EXTERNAL] Re: [clamav-users] Not able to use curl to download the cvd files successfully

> No! Don’t “bypass” it.
>
> And “protecting” does not need to be in quotes, it’s quite literally what we are doing. And people doing the above are the problem.
>
> As I said in countless other emails, either use Freshclam or https://github.com/micahsnyder/cvdupdate. The more people that do the above will force us to take drastic
> measures

Here's the reason I bypassed it.

I had a very old machine that I needed to do a scan on. I had to boot the machine with a recovery CD which was a very basic version of Linux. I compiled a statically linked version of ClamAV on another machine and transferred it to the problem machine, but needed to transfer two additional libraries (libpcre2 and libltdl I believe) before clamscan would run. Trying to get freshclam was a pain because it required all sorts of extra libraries, so rather than fetch them one at a time and transfer them, I decided to download main.cvd, daily.cvd, and bytecode.cvd myself. No Python on the machine, so I couldn't use the cvdupdate script. So I figured out that changing the User Agent string would allow me to use wget to download the files, and that's what I did.

If you want to protect your site, I completely understand, but do so by limiting or rate limiting the amount of transfers that happen from IP addresses to the database sites. There is nothing stopping people from abusing downloading full copies of these files using a real browser with some sort of automated download plugin, especially when you provide links to these files on your download page. Blocking valid transfer applications like wget from downloading legitimately just because they don't send a browser as a user agent is a dumb way of protection.

As well, if you don't want people using stuff like wget or curl to download these files, why do you specifically tell them to do so in your own Troubleshooting FAQ? A quote from the page https://www.clamav.net/documents/troubleshooting-faq: "Try to download daily.cvd with curl, wget, or lynx from the same machine that is running freshclam."

I am not being stupid as G.W. Haywood claimed, I was just trying to solve a problem that I had, and that other legitimate, responsible people might have in the future.



Todd A. Aiken

Systems Analyst & Administrator

ITS Department

BISHOP'S UNIVERSITY

2600 College Street

Sherbrooke, Quebec

CANADA J1M 1Z7

--------

"What's going on around here?" - RS

Having a technology issue?

Visit http://octopus.ubishops.ca to place a ticket directly into our ITS work order system.

This is the best way to get your requests to ITS and provide more detailed information for our analysts and technicians.
Re: [clamav-users] Not able to use curl to download the cvd files successfully [ In reply to ]
On Mar 8, 2021, at 11:30 AM, Todd Aiken <todd.aiken@ubishops.ca<mailto:todd.aiken@ubishops.ca>> wrote:

> From: clamav-users <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net>> on behalf of "Joel Esler (jesler) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Date: Monday, March 8, 2021 at 9:47 AM
> To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Cc: "Joel Esler (jesler)" <jesler@cisco.com<mailto:jesler@cisco.com>>
> Subject: [EXTERNAL] Re: [clamav-users] Not able to use curl to download the cvd files successfully

> No! Don’t “bypass” it.
>
> And “protecting” does not need to be in quotes, it’s quite literally what we are doing. And people doing the above are the problem.
>
> As I said in countless other emails, either use Freshclam or https://github.com/micahsnyder/cvdupdate. The more people that do the above will force us to take drastic
> measures

Here's the reason I bypassed it.

I had a very old machine that I needed to do a scan on. I had to boot the machine with a recovery CD which was a very basic version of Linux. I compiled a statically linked version of ClamAV on another machine and transferred it to the problem machine, but needed to transfer two additional libraries (libpcre2 and libltdl I believe) before clamscan would run. Trying to get freshclam was a pain because it required all sorts of extra libraries, so rather than fetch them one at a time and transfer them, I decided to download main.cvd, daily.cvd, and bytecode.cvd myself. No Python on the machine, so I couldn't use the cvdupdate script. So I figured out that changing the User Agent string would allow me to use wget to download the files, and that's what I did.

If you want to protect your site, I completely understand, but do so by limiting or rate limiting the amount of transfers that happen from IP addresses to the database sites. There is nothing stopping people from abusing downloading full copies of these files using a real browser with some sort of automated download plugin, especially when you provide links to these files on your download page. Blocking valid transfer applications like wget from downloading legitimately just because they don't send a browser as a user agent is a dumb way of protection.

As well, if you don't want people using stuff like wget or curl to download these files, why do you specifically tell them to do so in your own Troubleshooting FAQ? A quote from the page https://www.clamav.net/documents/troubleshooting-faq: "Try to download daily.cvd with curl, wget, or lynx from the same machine that is running freshclam."

I am not being stupid as G.W. Haywood claimed, I was just trying to solve a problem that I had, and that other legitimate, responsible people might have in the future.



Yup. We’re in emergency mode, and we’ll be fixing a lot of the documentation point to better solutions.