Hi Micah,
Thanks for the info. It looks like the timeout is an Ubuntu packaging
issue. The post-install scripts for the Ubuntu 16.04 and 18.04
clamav-freshclam 0.103.2 packages create a freshclam.conf with
"ReceiveTimeout=30", while the Ubuntu 20.04 package sets
"ReceiveTimeout=0". I hadn't thought there would be a difference
between the packages for different versions of Ubuntu, since they're all
ClamAV/FreshClam version 0.103.2. I've raised a bug on launchpad
<
https://bugs.launchpad.net/ubuntu/+source/clamav/+bug/1927777> to
suggest at least updating the 18.04 package (and the 16.04 one if it's
still getting updates now that 16.04 itself is end-of-life).
I'm still not sure whether FreshClam should be holding off for a bit
longer than 5 seconds between attempts, to avoid triggering the rate
limiting? The closest thing I can see in the configuration is
"MaxAttempts 5". Although from the log, it looks like it tries 5 times
with 5 seconds between attempts, says "Giving up on
https://database.clamav.net...", but then immediately starts trying
again and triggers the rate limiting after a few more attempts.
I haven't run freshclam manually at all, it's only ever been run by the
clamav-freshclam systemd service installed by the package. It doesn't
look like the service configuration specifies a user, so it's presumably
starting as root, but freshclam is then dropping privileges to the
"clamav" user after starting (freshclam.conf includes "DatabaseOwner
clamav"). It looks like it might be creating the file as root before
dropping privileges, and then trying to update it later - probably
hitting the same condition as you mention when running it via sudo and
then in other ways.
Thanks,
Mark.
Micah Snyder micasnyd via clamav-users wrote:
> Hi Mark,
>
> I'm not sure how you got a config with the default set to "ReceiveTimeout 30". I just tested with ubunte 20.04 a moment ago and a fresh `apt install clamav` (0.103.2). The config I found in /etc/clamav/freshclam.conf has "ReceiveTimeout 0" which means it is disabled. ClamAV's built-in default (if you don't specify) is also "0". So I'm not really sure what went wrong for you.
>
> So we do have a minor problem with the mirrors.dat in 0.103.2. It will be owned by root instead of by the "clamav" user if you run "sudo freshclam --daemon". Then if you try running freshclam a different way, you may run into permissions issues. We'll have to fix this in the next patch version ?.
>
> -Micah
>
>> -----Original Message-----
>> From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of
>> clamav.mbourne@spamgourmet.com
>> Sent: Thursday, May 6, 2021 1:12 PM
>> To: clamav-users@lists.clamav.net
>> Subject: Re: [clamav-users] Update on rate limits and downloading
>>
>> Joel Esler jesler via clamav-users wrote:
>>> Overall — we’re doing much better.
>>>
>>> We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve
>>> made significant progress.
>>>
>>> /However, /we still have over 700 individual systems downloading the
>>> full daily.cvd over 200x a day. (This should be once a day, /if
>>> that/.)
>>>
>>> If you are not using 0.103.2 and it’s accompanying FreshClam to
>>> download these updates, and when you do create a NEW FreshClam.conf
>>> file and move your settings to that. We’re going to have to start
>>> blocking these atrocious abusers, as the rate limits are hurting
>>> everyone else at this point.
>>
>> I'm new to installing ClamAV, so there may be something I haven't done
>> quite right here. A couple of weeks ago, I installed ClamAV 0.103.2 from the
>> Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, clamav-
>> docs, clamtk and libclamunrar9 packages).
>>
>> By default, FreshClam seems to use too short a download timeout and retry
>> too frequently, triggering the rate limiting. After installing, the FreshClam
>> service would repeatedly attempt to download the daily.cvd file, time out
>> after 30 seconds, and wait 5 seconds before trying again.
>> After a few attempts, it then gets blocked by the CDN (if that's what "you are
>> on cool-down" in the log means?) for 4 hours. By the time I'd realised this
>> was happening following the initial install, I was already blocked.
>>
>> Perhaps this might, if left in a default configuration, be seen to attempt to
>> download daily.cvd over 100 times a day, but without ever actually getting
>> the whole file. From what I'd seen here and in documentation / FAQs, I
>> thought FreshClam was supposed to avoid retrying so frequently that it
>> triggers the rate limiting?
>>
>> I don't know if the default configuration is provided by ClamAV or the
>> Ubuntu packaging (either way, it seems FreshClam shouldn't just keep
>> retrying so quickly?) In my case, freshclam.conf originally had
>> "ReceiveTimeout 30". Increasing it to 60 wasn't enough. I then went to 600,
>> which was successful. Somewhere in between would probably have been
>> fine, but incrementing more gradually would have been a long process,
>> having to wait at least 4 hours between attempts (particularly as restarting
>> FreshClam after setting a new timeout seems to get blocked for a further 4
>> hours - not just the remainder of the original block).
>>
>> In case it's of any use (and if this list allows it), I've attached my freshclam.log
>> from those initial attempts.
>>
>> All seems to be working OK now, but posting here in case the information is
>> useful.
>>
>>> Please help us, stay diligent, keep going keep upgrading. Upgrade to
>>> 0.103.2, and keep your mirrors.dat file around, this file contains a
>>> snapshot of where you are in your update progression so that the next
>>> time that FreshClam run, it can start where it left off.
>>
>> Interesting you should mention mirrors.dat... Aside from the downloads
>> timing out, there are also some errors in my freshclam.log about not being
>> able to create mirrors.dat. That's a bit odd, since the /var/lib/clamav/
>> directory is owned and writeable by the correct user, but the mirrors.dat file
>> within it is owned by root. Deleting that file and restarting the freshclam
>> service, the mirrors.dat file gets recreated, again owned by root. That error
>> hasn't appeared in the logs since, although mirrors.dat is still dated 25th April,
>> so I'm not sure if there's still a problem with that.
>>
>> --
>> Mark.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml