Mailing List Archive

[clamav-users] ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends
https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html<https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html?m=1>

Freshclam, cdiffs and bandwidth are your friends
During a recent review of file downloads from our ClamAV CDN network, we've noticed hundreds of IPs that seem to be downloading the daily.cvd and the main.cvd thousands of times a day.

There are about a dozen IPs that are downloading those to files more than 40,000 times a day. This is causing us to transfer about 250TB of data a day. We would encourage any users still doing this to cease as soon as possible. Not only does it waste our bandwidth — as we have much more efficient ways of downloading the updates — but it also wastes your bandwidth, as well.

Freshclam has the ability to download partial files of updates (called cdiffs). Which are smaller, more incremental updates to the database. This allows users, and us, to manage our downloads in a much more efficient manner. We often receive the complaint, "I have to download the daily.cvd and main.cvd with Python and move the updates to an off-internet system." That's fine — it's a use case we support. However, you can do the same with freshclam and the small cdiffs.

Furthermore, we also only release updates once a day. Reducing the number of updates you check for (and, subsequently, download we assume through a crontab or periodic job of some type) would also alleviate this issue.

We will be constantly monitoring this in hopes that people migrate to using freshclam. Over-abusers (for instance, the top 10 IPs that are downloading main.cvd 40,000 times a day), will be immediately blocked. Further abusers may also be blocked, without notice.

To mitigate, please complete the following tasks:

1. Use Freshclam instead of Python or whatever downloading script you have cron'd.
2. Reduce the checks to once or twice a day.

Thank you for helping keep the ClamAV network healthy.

Any questions, please see us over on the ClamAV-Users list.


Sent from my ? iPhone
Re: [clamav-users] ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends [ In reply to ]
"...we also only release updates once a day."

Are there *never* any urgent virus updates released in between? In
other words, is it always useless to check the TXT record more often?



On Mon, 27 Jul 2020 22:09:31 +0000
"Joel Esler \(jesler\) via clamav-users" <clamav-users@lists.clamav.net> wrote:

> https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html<https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html?m=1>
>
> Freshclam, cdiffs and bandwidth are your friends
> During a recent review of file downloads from our ClamAV CDN network, we've noticed hundreds of IPs that seem to be downloading the daily.cvd and the main.cvd thousands of times a day.
>
> There are about a dozen IPs that are downloading those to files more than 40,000 times a day. This is causing us to transfer about 250TB of data a day. We would encourage any users still doing this to cease as soon as possible. Not only does it waste our bandwidth — as we have much more efficient ways of downloading the updates — but it also wastes your bandwidth, as well.
>
> Freshclam has the ability to download partial files of updates (called cdiffs). Which are smaller, more incremental updates to the database. This allows users, and us, to manage our downloads in a much more efficient manner. We often receive the complaint, "I have to download the daily.cvd and main.cvd with Python and move the updates to an off-internet system." That's fine — it's a use case we support. However, you can do the same with freshclam and the small cdiffs.
>
> Furthermore, we also only release updates once a day. Reducing the number of updates you check for (and, subsequently, download we assume through a crontab or periodic job of some type) would also alleviate this issue.
>
> We will be constantly monitoring this in hopes that people migrate to using freshclam. Over-abusers (for instance, the top 10 IPs that are downloading main.cvd 40,000 times a day), will be immediately blocked. Further abusers may also be blocked, without notice.
>
> To mitigate, please complete the following tasks:
>
> 1. Use Freshclam instead of Python or whatever downloading script you have cron'd.
> 2. Reduce the checks to once or twice a day.
>
> Thank you for helping keep the ClamAV network healthy.
>
> Any questions, please see us over on the ClamAV-Users list.
>
>
> Sent from my ? iPhone

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends [ In reply to ]
Feel free to check the TXT record once an hour or whenever you want. Checking the TXT record will tell you if there is a diff to download, for sure, and then you can go download that diff.

The problem isn’t that, the problem is downloading the ENTIRE main.cvd and daily.cvd once a minute, every minute (or in some cases, several times in the same minute)

Sent from my ? iPad

> On Jul 28, 2020, at 19:02, Paul Kosinski via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?"...we also only release updates once a day."
>
> Are there *never* any urgent virus updates released in between? In
> other words, is it always useless to check the TXT record more often?
>
>
>
>> On Mon, 27 Jul 2020 22:09:31 +0000
>> "Joel Esler \(jesler\) via clamav-users" <clamav-users@lists.clamav.net> wrote:
>>
>> https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html<https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html?m=1>
>>
>> Freshclam, cdiffs and bandwidth are your friends
>> During a recent review of file downloads from our ClamAV CDN network, we've noticed hundreds of IPs that seem to be downloading the daily.cvd and the main.cvd thousands of times a day.
>>
>> There are about a dozen IPs that are downloading those to files more than 40,000 times a day. This is causing us to transfer about 250TB of data a day. We would encourage any users still doing this to cease as soon as possible. Not only does it waste our bandwidth — as we have much more efficient ways of downloading the updates — but it also wastes your bandwidth, as well.
>>
>> Freshclam has the ability to download partial files of updates (called cdiffs). Which are smaller, more incremental updates to the database. This allows users, and us, to manage our downloads in a much more efficient manner. We often receive the complaint, "I have to download the daily.cvd and main.cvd with Python and move the updates to an off-internet system." That's fine — it's a use case we support. However, you can do the same with freshclam and the small cdiffs.
>>
>> Furthermore, we also only release updates once a day. Reducing the number of updates you check for (and, subsequently, download we assume through a crontab or periodic job of some type) would also alleviate this issue.
>>
>> We will be constantly monitoring this in hopes that people migrate to using freshclam. Over-abusers (for instance, the top 10 IPs that are downloading main.cvd 40,000 times a day), will be immediately blocked. Further abusers may also be blocked, without notice.
>>
>> To mitigate, please complete the following tasks:
>>
>> 1. Use Freshclam instead of Python or whatever downloading script you have cron'd.
>> 2. Reduce the checks to once or twice a day.
>>
>> Thank you for helping keep the ClamAV network healthy.
>>
>> Any questions, please see us over on the ClamAV-Users list.
>>
>>
>> Sent from my ? iPhone
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
Re: [clamav-users] ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends [ In reply to ]
On Tue, Jul 28, 2020 at 16:01 PM, Paul Kosinski via clamav-users wrote:
> Are there *never* any urgent virus updates released in between?


I may have missed it, but I had to go back to the end of January 2019 to find an occurrence of more than one update on a given day.

-Al-
Re: [clamav-users] ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends [ In reply to ]
You may want to subscribe to the mailing list
clamav-virusdb@lists.clamav.net
for a changelog of the virus db. Indeed this list only sends one mail
per day.


On 29.07.20 01:01, Paul Kosinski via clamav-users wrote:
> "...we also only release updates once a day."
>
> Are there *never* any urgent virus updates released in between? In
> other words, is it always useless to check the TXT record more often?
>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] [ext] Re: ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends [ In reply to ]
* Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>:
> "...we also only release updates once a day."
>
> Are there *never* any urgent virus updates released in between? In
> other words, is it always useless to check the TXT record more often?

I was wondering about this wording as well!

But then I checked:

Mon Jul 20 17:00:17 2020 -> daily.cld updated (version: 25879, sigs: 3519456, f-level: 63, builder: raynman)
Tue Jul 21 17:14:19 2020 -> daily.cld updated (version: 25880, sigs: 3548222, f-level: 63, builder: raynman)
Wed Jul 22 17:14:33 2020 -> daily.cld updated (version: 25881, sigs: 3573651, f-level: 63, builder: raynman)
Thu Jul 23 17:14:47 2020 -> daily.cld updated (version: 25882, sigs: 3584533, f-level: 63, builder: raynman)
Fri Jul 24 17:15:02 2020 -> daily.cld updated (version: 25883, sigs: 3609907, f-level: 63, builder: raynman)
Sat Jul 25 17:15:18 2020 -> daily.cld updated (version: 25884, sigs: 3663341, f-level: 63, builder: raynman)
Sun Jul 26 17:00:15 2020 -> daily.cld updated (version: 25885, sigs: 3668554, f-level: 63, builder: raynman)
Mon Jul 27 18:00:38 2020 -> daily.cld updated (version: 25886, sigs: 3678125, f-level: 63, builder: raynman)
Tue Jul 28 18:00:53 2020 -> daily.cld updated (version: 25887, sigs: 3681654, f-level: 63, builder: raynman)

Ralf Hildebrandt
Charité - Universitätsmedizin Berlin
Geschäftsbereich IT | Abteilung Netzwerk

Campus Benjamin Franklin (CBF)
Haus I | 1. OG | Raum 105
Hindenburgdamm 30 | D-12203 Berlin

Tel. +49 30 450 570 155
ralf.hildebrandt@charite.de
https://www.charite.de

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] [ext] Re: ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends [ In reply to ]
As I said, checking the DNS TXT entry is fine. Checking that every hour is fine, (just in case we push something immediate). Downloading the cdiffs is fine. Downloading the entire CVD files constantly is not fine.

Sent from my ? iPad

> On Jul 29, 2020, at 04:37, Ralf Hildebrandt via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?* Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>:
>> "...we also only release updates once a day."
>>
>> Are there *never* any urgent virus updates released in between? In
>> other words, is it always useless to check the TXT record more often?
>
> I was wondering about this wording as well!
>
> But then I checked:
>
> Mon Jul 20 17:00:17 2020 -> daily.cld updated (version: 25879, sigs: 3519456, f-level: 63, builder: raynman)
> Tue Jul 21 17:14:19 2020 -> daily.cld updated (version: 25880, sigs: 3548222, f-level: 63, builder: raynman)
> Wed Jul 22 17:14:33 2020 -> daily.cld updated (version: 25881, sigs: 3573651, f-level: 63, builder: raynman)
> Thu Jul 23 17:14:47 2020 -> daily.cld updated (version: 25882, sigs: 3584533, f-level: 63, builder: raynman)
> Fri Jul 24 17:15:02 2020 -> daily.cld updated (version: 25883, sigs: 3609907, f-level: 63, builder: raynman)
> Sat Jul 25 17:15:18 2020 -> daily.cld updated (version: 25884, sigs: 3663341, f-level: 63, builder: raynman)
> Sun Jul 26 17:00:15 2020 -> daily.cld updated (version: 25885, sigs: 3668554, f-level: 63, builder: raynman)
> Mon Jul 27 18:00:38 2020 -> daily.cld updated (version: 25886, sigs: 3678125, f-level: 63, builder: raynman)
> Tue Jul 28 18:00:53 2020 -> daily.cld updated (version: 25887, sigs: 3681654, f-level: 63, builder: raynman)
>
> Ralf Hildebrandt
> Charité - Universitätsmedizin Berlin
> Geschäftsbereich IT | Abteilung Netzwerk
>
> Campus Benjamin Franklin (CBF)
> Haus I | 1. OG | Raum 105
> Hindenburgdamm 30 | D-12203 Berlin
>
> Tel. +49 30 450 570 155
> ralf.hildebrandt@charite.de
> https://www.charite.de
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml