Mailing List Archive

[clamav-users] Update on rate limits and downloading
Overall — we’re doing much better.

We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve made significant progress.

However, we still have over 700 individual systems downloading the full daily.cvd over 200x a day. (This should be once a day, if that.)

If you are not using 0.103.2 and it’s accompanying FreshClam to download these updates, and when you do create a NEW FreshClam.conf file and move your settings to that. We’re going to have to start blocking these atrocious abusers, as the rate limits are hurting everyone else at this point.

Please help us, stay diligent, keep going keep upgrading. Upgrade to 0.103.2, and keep your mirrors.dat file around, this file contains a snapshot of where you are in your update progression so that the next time that FreshClam run, it can start where it left off.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
https://www.talosintelligence.com | https://www.snort.org | https://www.clamav.net
Re: [clamav-users] Update on rate limits and downloading [ In reply to ]
Joel Esler jesler via clamav-users wrote:
> Overall — we’re doing much better.
>
> We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve made
> significant progress.
>
> /However, /we still have over 700 individual systems downloading the
> full daily.cvd over 200x a day. (This should be once a day, /if that/.)
>
> If you are not using 0.103.2 and it’s accompanying FreshClam to download
> these updates, and when you do create a NEW FreshClam.conf file and move
> your settings to that.  We’re going to have to start blocking these
> atrocious abusers, as the rate limits are hurting everyone else at this
> point.

I'm new to installing ClamAV, so there may be something I haven't done
quite right here. A couple of weeks ago, I installed ClamAV 0.103.2
from the Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon,
clamav-docs, clamtk and libclamunrar9 packages).

By default, FreshClam seems to use too short a download timeout and
retry too frequently, triggering the rate limiting. After installing,
the FreshClam service would repeatedly attempt to download the daily.cvd
file, time out after 30 seconds, and wait 5 seconds before trying again.
After a few attempts, it then gets blocked by the CDN (if that's what
"you are on cool-down" in the log means?) for 4 hours. By the time I'd
realised this was happening following the initial install, I was already
blocked.

Perhaps this might, if left in a default configuration, be seen to
attempt to download daily.cvd over 100 times a day, but without ever
actually getting the whole file. From what I'd seen here and in
documentation / FAQs, I thought FreshClam was supposed to avoid retrying
so frequently that it triggers the rate limiting?

I don't know if the default configuration is provided by ClamAV or the
Ubuntu packaging (either way, it seems FreshClam shouldn't just keep
retrying so quickly?) In my case, freshclam.conf originally had
"ReceiveTimeout 30". Increasing it to 60 wasn't enough. I then went to
600, which was successful. Somewhere in between would probably have
been fine, but incrementing more gradually would have been a long
process, having to wait at least 4 hours between attempts (particularly
as restarting FreshClam after setting a new timeout seems to get blocked
for a further 4 hours - not just the remainder of the original block).

In case it's of any use (and if this list allows it), I've attached my
freshclam.log from those initial attempts.

All seems to be working OK now, but posting here in case the information
is useful.

> Please help us, stay diligent, keep going keep upgrading.  Upgrade to
> 0.103.2, and keep your mirrors.dat file around, this file contains a
> snapshot of where you are in your update progression so that the next
> time that FreshClam run, it can start where it left off.

Interesting you should mention mirrors.dat... Aside from the downloads
timing out, there are also some errors in my freshclam.log about not
being able to create mirrors.dat. That's a bit odd, since the
/var/lib/clamav/ directory is owned and writeable by the correct user,
but the mirrors.dat file within it is owned by root. Deleting that file
and restarting the freshclam service, the mirrors.dat file gets
recreated, again owned by root. That error hasn't appeared in the logs
since, although mirrors.dat is still dated 25th April, so I'm not sure
if there's still a problem with that.

--
Mark.
Re: [clamav-users] Update on rate limits and downloading [ In reply to ]
Alright, I'll try this. I've been using your Product for quite sometime without problems.


On Thursday, May 6, 2021, 02:09:12 PM CDT, Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net> wrote:

Overall — we’re doing much better.  
We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve made significant progress.
However, we still have over 700 individual systems downloading the full daily.cvd over 200x a day. (This should be once a day,if that.)
If you are not using 0.103.2 and it’s accompanying FreshClam to download these updates, and when you do create a NEW FreshClam.conf file and move your settings to that.  We’re going to have to start blocking these atrocious abusers, as the rate limits are hurting everyone else at this point.
Please help us, stay diligent, keep going keep upgrading.  Upgrade to 0.103.2, and keep your mirrors.dat file around, this file contains a snapshot of where you are in your update progression so that the next time that FreshClam run, it can start where it left off.
-- Joel EslerManager, Communities DivisionCisco Talos Intelligence Grouphttps://www.talosintelligence.com |https://www.snort.org | https://www.clamav.net 
_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Update on rate limits and downloading [ In reply to ]
Hi Mark,

I'm not sure how you got a config with the default set to "ReceiveTimeout 30". I just tested with ubunte 20.04 a moment ago and a fresh `apt install clamav` (0.103.2). The config I found in /etc/clamav/freshclam.conf has "ReceiveTimeout 0" which means it is disabled. ClamAV's built-in default (if you don't specify) is also "0". So I'm not really sure what went wrong for you.

So we do have a minor problem with the mirrors.dat in 0.103.2. It will be owned by root instead of by the "clamav" user if you run "sudo freshclam --daemon". Then if you try running freshclam a different way, you may run into permissions issues. We'll have to fix this in the next patch version ?.

-Micah

> -----Original Message-----
> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> clamav.mbourne@spamgourmet.com
> Sent: Thursday, May 6, 2021 1:12 PM
> To: clamav-users@lists.clamav.net
> Subject: Re: [clamav-users] Update on rate limits and downloading
>
> Joel Esler jesler via clamav-users wrote:
> > Overall — we’re doing much better.
> >
> > We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve
> > made significant progress.
> >
> > /However, /we still have over 700 individual systems downloading the
> > full daily.cvd over 200x a day. (This should be once a day, /if
> > that/.)
> >
> > If you are not using 0.103.2 and it’s accompanying FreshClam to
> > download these updates, and when you do create a NEW FreshClam.conf
> > file and move your settings to that.  We’re going to have to start
> > blocking these atrocious abusers, as the rate limits are hurting
> > everyone else at this point.
>
> I'm new to installing ClamAV, so there may be something I haven't done
> quite right here. A couple of weeks ago, I installed ClamAV 0.103.2 from the
> Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, clamav-
> docs, clamtk and libclamunrar9 packages).
>
> By default, FreshClam seems to use too short a download timeout and retry
> too frequently, triggering the rate limiting. After installing, the FreshClam
> service would repeatedly attempt to download the daily.cvd file, time out
> after 30 seconds, and wait 5 seconds before trying again.
> After a few attempts, it then gets blocked by the CDN (if that's what "you are
> on cool-down" in the log means?) for 4 hours. By the time I'd realised this
> was happening following the initial install, I was already blocked.
>
> Perhaps this might, if left in a default configuration, be seen to attempt to
> download daily.cvd over 100 times a day, but without ever actually getting
> the whole file. From what I'd seen here and in documentation / FAQs, I
> thought FreshClam was supposed to avoid retrying so frequently that it
> triggers the rate limiting?
>
> I don't know if the default configuration is provided by ClamAV or the
> Ubuntu packaging (either way, it seems FreshClam shouldn't just keep
> retrying so quickly?) In my case, freshclam.conf originally had
> "ReceiveTimeout 30". Increasing it to 60 wasn't enough. I then went to 600,
> which was successful. Somewhere in between would probably have been
> fine, but incrementing more gradually would have been a long process,
> having to wait at least 4 hours between attempts (particularly as restarting
> FreshClam after setting a new timeout seems to get blocked for a further 4
> hours - not just the remainder of the original block).
>
> In case it's of any use (and if this list allows it), I've attached my freshclam.log
> from those initial attempts.
>
> All seems to be working OK now, but posting here in case the information is
> useful.
>
> > Please help us, stay diligent, keep going keep upgrading.  Upgrade to
> > 0.103.2, and keep your mirrors.dat file around, this file contains a
> > snapshot of where you are in your update progression so that the next
> > time that FreshClam run, it can start where it left off.
>
> Interesting you should mention mirrors.dat... Aside from the downloads
> timing out, there are also some errors in my freshclam.log about not being
> able to create mirrors.dat. That's a bit odd, since the /var/lib/clamav/
> directory is owned and writeable by the correct user, but the mirrors.dat file
> within it is owned by root. Deleting that file and restarting the freshclam
> service, the mirrors.dat file gets recreated, again owned by root. That error
> hasn't appeared in the logs since, although mirrors.dat is still dated 25th April,
> so I'm not sure if there's still a problem with that.
>
> --
> Mark.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Update on rate limits and downloading [ In reply to ]
Hi Micah,

Thanks for the info. It looks like the timeout is an Ubuntu packaging
issue. The post-install scripts for the Ubuntu 16.04 and 18.04
clamav-freshclam 0.103.2 packages create a freshclam.conf with
"ReceiveTimeout=30", while the Ubuntu 20.04 package sets
"ReceiveTimeout=0". I hadn't thought there would be a difference
between the packages for different versions of Ubuntu, since they're all
ClamAV/FreshClam version 0.103.2. I've raised a bug on launchpad
<https://bugs.launchpad.net/ubuntu/+source/clamav/+bug/1927777> to
suggest at least updating the 18.04 package (and the 16.04 one if it's
still getting updates now that 16.04 itself is end-of-life).

I'm still not sure whether FreshClam should be holding off for a bit
longer than 5 seconds between attempts, to avoid triggering the rate
limiting? The closest thing I can see in the configuration is
"MaxAttempts 5". Although from the log, it looks like it tries 5 times
with 5 seconds between attempts, says "Giving up on
https://database.clamav.net...", but then immediately starts trying
again and triggers the rate limiting after a few more attempts.

I haven't run freshclam manually at all, it's only ever been run by the
clamav-freshclam systemd service installed by the package. It doesn't
look like the service configuration specifies a user, so it's presumably
starting as root, but freshclam is then dropping privileges to the
"clamav" user after starting (freshclam.conf includes "DatabaseOwner
clamav"). It looks like it might be creating the file as root before
dropping privileges, and then trying to update it later - probably
hitting the same condition as you mention when running it via sudo and
then in other ways.

Thanks,
Mark.


Micah Snyder micasnyd via clamav-users wrote:
> Hi Mark,
>
> I'm not sure how you got a config with the default set to "ReceiveTimeout 30". I just tested with ubunte 20.04 a moment ago and a fresh `apt install clamav` (0.103.2). The config I found in /etc/clamav/freshclam.conf has "ReceiveTimeout 0" which means it is disabled. ClamAV's built-in default (if you don't specify) is also "0". So I'm not really sure what went wrong for you.
>
> So we do have a minor problem with the mirrors.dat in 0.103.2. It will be owned by root instead of by the "clamav" user if you run "sudo freshclam --daemon". Then if you try running freshclam a different way, you may run into permissions issues. We'll have to fix this in the next patch version ?.
>
> -Micah
>
>> -----Original Message-----
>> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
>> clamav.mbourne@spamgourmet.com
>> Sent: Thursday, May 6, 2021 1:12 PM
>> To: clamav-users@lists.clamav.net
>> Subject: Re: [clamav-users] Update on rate limits and downloading
>>
>> Joel Esler jesler via clamav-users wrote:
>>> Overall — we’re doing much better.
>>>
>>> We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve
>>> made significant progress.
>>>
>>> /However, /we still have over 700 individual systems downloading the
>>> full daily.cvd over 200x a day. (This should be once a day, /if
>>> that/.)
>>>
>>> If you are not using 0.103.2 and it’s accompanying FreshClam to
>>> download these updates, and when you do create a NEW FreshClam.conf
>>> file and move your settings to that.  We’re going to have to start
>>> blocking these atrocious abusers, as the rate limits are hurting
>>> everyone else at this point.
>>
>> I'm new to installing ClamAV, so there may be something I haven't done
>> quite right here. A couple of weeks ago, I installed ClamAV 0.103.2 from the
>> Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, clamav-
>> docs, clamtk and libclamunrar9 packages).
>>
>> By default, FreshClam seems to use too short a download timeout and retry
>> too frequently, triggering the rate limiting. After installing, the FreshClam
>> service would repeatedly attempt to download the daily.cvd file, time out
>> after 30 seconds, and wait 5 seconds before trying again.
>> After a few attempts, it then gets blocked by the CDN (if that's what "you are
>> on cool-down" in the log means?) for 4 hours. By the time I'd realised this
>> was happening following the initial install, I was already blocked.
>>
>> Perhaps this might, if left in a default configuration, be seen to attempt to
>> download daily.cvd over 100 times a day, but without ever actually getting
>> the whole file. From what I'd seen here and in documentation / FAQs, I
>> thought FreshClam was supposed to avoid retrying so frequently that it
>> triggers the rate limiting?
>>
>> I don't know if the default configuration is provided by ClamAV or the
>> Ubuntu packaging (either way, it seems FreshClam shouldn't just keep
>> retrying so quickly?) In my case, freshclam.conf originally had
>> "ReceiveTimeout 30". Increasing it to 60 wasn't enough. I then went to 600,
>> which was successful. Somewhere in between would probably have been
>> fine, but incrementing more gradually would have been a long process,
>> having to wait at least 4 hours between attempts (particularly as restarting
>> FreshClam after setting a new timeout seems to get blocked for a further 4
>> hours - not just the remainder of the original block).
>>
>> In case it's of any use (and if this list allows it), I've attached my freshclam.log
>> from those initial attempts.
>>
>> All seems to be working OK now, but posting here in case the information is
>> useful.
>>
>>> Please help us, stay diligent, keep going keep upgrading.  Upgrade to
>>> 0.103.2, and keep your mirrors.dat file around, this file contains a
>>> snapshot of where you are in your update progression so that the next
>>> time that FreshClam run, it can start where it left off.
>>
>> Interesting you should mention mirrors.dat... Aside from the downloads
>> timing out, there are also some errors in my freshclam.log about not being
>> able to create mirrors.dat. That's a bit odd, since the /var/lib/clamav/
>> directory is owned and writeable by the correct user, but the mirrors.dat file
>> within it is owned by root. Deleting that file and restarting the freshclam
>> service, the mirrors.dat file gets recreated, again owned by root. That error
>> hasn't appeared in the logs since, although mirrors.dat is still dated 25th April,
>> so I'm not sure if there's still a problem with that.
>>
>> --
>> Mark.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Update on rate limits and downloading [ In reply to ]
Mark,

Thanks for tracking down the freshclam.conf issue and submitting the bug report.

With regards to the 5 second between attempts, I'm not sure either. It would seem reasonable if there was some sort of network glitch, but if it's a persistent issue like the receive timeout for slower connections, then yeah 5 seconds doesn't make much sense. I'm not really sure what to say. We could reduce the number or retries attempts as well, but in the end the config change to "ReceiveTimeout=0" should resolve the issue and no one should have to retry. I'm inclined to leave freshclam as-is.

Regarding the mirrors.dat ownership issue: You're probably right. It probably tries to update mirrors.dat later on after it has switched to run as "clamav" and then fails. :-(
Well, we'll get this fixed in the next patch release. Sorry about the trouble.

-Micah

> -----Original Message-----
> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> clamav.mbourne@spamgourmet.com
> Sent: Friday, May 7, 2021 1:33 PM
> To: clamav-users@lists.clamav.net
> Subject: Re: [clamav-users] Update on rate limits and downloading
>
> Hi Micah,
>
> Thanks for the info. It looks like the timeout is an Ubuntu packaging issue. The
> post-install scripts for the Ubuntu 16.04 and 18.04 clamav-freshclam 0.103.2
> packages create a freshclam.conf with "ReceiveTimeout=30", while the Ubuntu
> 20.04 package sets "ReceiveTimeout=0". I hadn't thought there would be a
> difference between the packages for different versions of Ubuntu, since they're
> all ClamAV/FreshClam version 0.103.2. I've raised a bug on launchpad
> <https://bugs.launchpad.net/ubuntu/+source/clamav/+bug/1927777> to
> suggest at least updating the 18.04 package (and the 16.04 one if it's still
> getting updates now that 16.04 itself is end-of-life).
>
> I'm still not sure whether FreshClam should be holding off for a bit longer than
> 5 seconds between attempts, to avoid triggering the rate limiting? The closest
> thing I can see in the configuration is "MaxAttempts 5". Although from the log,
> it looks like it tries 5 times with 5 seconds between attempts, says "Giving up
> on https://database.clamav.net...", but then immediately starts trying again
> and triggers the rate limiting after a few more attempts.
>
> I haven't run freshclam manually at all, it's only ever been run by the clamav-
> freshclam systemd service installed by the package. It doesn't look like the
> service configuration specifies a user, so it's presumably starting as root, but
> freshclam is then dropping privileges to the "clamav" user after starting
> (freshclam.conf includes "DatabaseOwner clamav"). It looks like it might be
> creating the file as root before dropping privileges, and then trying to update it
> later - probably hitting the same condition as you mention when running it via
> sudo and then in other ways.
>
> Thanks,
> Mark.
>
>
> Micah Snyder micasnyd via clamav-users wrote:
> > Hi Mark,
> >
> > I'm not sure how you got a config with the default set to "ReceiveTimeout
> 30". I just tested with ubunte 20.04 a moment ago and a fresh `apt install
> clamav` (0.103.2). The config I found in /etc/clamav/freshclam.conf has
> "ReceiveTimeout 0" which means it is disabled. ClamAV's built-in default (if you
> don't specify) is also "0". So I'm not really sure what went wrong for you.
> >
> > So we do have a minor problem with the mirrors.dat in 0.103.2. It will be
> owned by root instead of by the "clamav" user if you run "sudo freshclam --
> daemon". Then if you try running freshclam a different way, you may run into
> permissions issues. We'll have to fix this in the next patch version ?.
> >
> > -Micah
> >
> >> -----Original Message-----
> >> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
> >> clamav.mbourne@spamgourmet.com
> >> Sent: Thursday, May 6, 2021 1:12 PM
> >> To: clamav-users@lists.clamav.net
> >> Subject: Re: [clamav-users] Update on rate limits and downloading
> >>
> >> Joel Esler jesler via clamav-users wrote:
> >>> Overall — we’re doing much better.
> >>>
> >>> We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve
> >>> made significant progress.
> >>>
> >>> /However, /we still have over 700 individual systems downloading the
> >>> full daily.cvd over 200x a day. (This should be once a day, /if
> >>> that/.)
> >>>
> >>> If you are not using 0.103.2 and it’s accompanying FreshClam to
> >>> download these updates, and when you do create a NEW FreshClam.conf
> >>> file and move your settings to that.  We’re going to have to start
> >>> blocking these atrocious abusers, as the rate limits are hurting
> >>> everyone else at this point.
> >>
> >> I'm new to installing ClamAV, so there may be something I haven't done
> >> quite right here. A couple of weeks ago, I installed ClamAV 0.103.2 from the
> >> Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, clamav-
> >> docs, clamtk and libclamunrar9 packages).
> >>
> >> By default, FreshClam seems to use too short a download timeout and retry
> >> too frequently, triggering the rate limiting. After installing, the FreshClam
> >> service would repeatedly attempt to download the daily.cvd file, time out
> >> after 30 seconds, and wait 5 seconds before trying again.
> >> After a few attempts, it then gets blocked by the CDN (if that's what "you
> are
> >> on cool-down" in the log means?) for 4 hours. By the time I'd realised this
> >> was happening following the initial install, I was already blocked.
> >>
> >> Perhaps this might, if left in a default configuration, be seen to attempt to
> >> download daily.cvd over 100 times a day, but without ever actually getting
> >> the whole file. From what I'd seen here and in documentation / FAQs, I
> >> thought FreshClam was supposed to avoid retrying so frequently that it
> >> triggers the rate limiting?
> >>
> >> I don't know if the default configuration is provided by ClamAV or the
> >> Ubuntu packaging (either way, it seems FreshClam shouldn't just keep
> >> retrying so quickly?) In my case, freshclam.conf originally had
> >> "ReceiveTimeout 30". Increasing it to 60 wasn't enough. I then went to
> 600,
> >> which was successful. Somewhere in between would probably have been
> >> fine, but incrementing more gradually would have been a long process,
> >> having to wait at least 4 hours between attempts (particularly as restarting
> >> FreshClam after setting a new timeout seems to get blocked for a further 4
> >> hours - not just the remainder of the original block).
> >>
> >> In case it's of any use (and if this list allows it), I've attached my
> freshclam.log
> >> from those initial attempts.
> >>
> >> All seems to be working OK now, but posting here in case the information is
> >> useful.
> >>
> >>> Please help us, stay diligent, keep going keep upgrading.  Upgrade to
> >>> 0.103.2, and keep your mirrors.dat file around, this file contains a
> >>> snapshot of where you are in your update progression so that the next
> >>> time that FreshClam run, it can start where it left off.
> >>
> >> Interesting you should mention mirrors.dat... Aside from the downloads
> >> timing out, there are also some errors in my freshclam.log about not being
> >> able to create mirrors.dat. That's a bit odd, since the /var/lib/clamav/
> >> directory is owned and writeable by the correct user, but the mirrors.dat file
> >> within it is owned by root. Deleting that file and restarting the freshclam
> >> service, the mirrors.dat file gets recreated, again owned by root. That error
> >> hasn't appeared in the logs since, although mirrors.dat is still dated 25th
> April,
> >> so I'm not sure if there's still a problem with that.
> >>
> >> --
> >> Mark.
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
> >
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml