Mailing List Archive

> Can you please tell me the H/W and S/W Specification
> of the Private local Mirror Server as a best practice for CVD?!

https://www.clamav.net/documents/private-local-mirrors

It's going to depend on how many clients you will be serving...
10 vs 10,000 is a huge difference in hardware requirements.

Realistically though, no matter which route you take, it is just
clients downloading static content at various intervals, which is not
very CPU intensive. You shouldn't need anything *that* powerful to
serve the files.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

I'm interested as to why people want to do private mirrors? Other than to save bandwidth going to "the internet"?

> On Jul 30, 2019, at 9:40 AM, J.R. via clamav-users <clamav-users@lists.clamav.net> wrote:
>
>> Can you please tell me the H/W and S/W Specification
>> of the Private local Mirror Server as a best practice for CVD?!
>
> https://www.clamav.net/documents/private-local-mirrors
>
> It's going to depend on how many clients you will be serving...
> 10 vs 10,000 is a huge difference in hardware requirements.
>
> Realistically though, no matter which route you take, it is just
> clients downloading static content at various intervals, which is not
> very CPU intensive. You shouldn't need anything *that* powerful to
> serve the files.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

Control. Is it really necessary to go over basic IT management practises here?

On Tue, Jul 30, 2019 at 05:13:50PM +0000, Joel Esler (jesler) via clamav-users wrote:
> I'm interested as to why people want to do private mirrors? Other than to save bandwidth going to "the internet"?
>
> > On Jul 30, 2019, at 9:40 AM, J.R. via clamav-users <clamav-users@lists.clamav.net> wrote:
> >
> >> Can you please tell me the H/W and S/W Specification
> >> of the Private local Mirror Server as a best practice for CVD?!
> >
> > https://www.clamav.net/documents/private-local-mirrors
> >
> > It's going to depend on how many clients you will be serving...
> > 10 vs 10,000 is a huge difference in hardware requirements.
> >
> > Realistically though, no matter which route you take, it is just
> > clients downloading static content at various intervals, which is not
> > very CPU intensive. You shouldn't need anything *that* powerful to
> > serve the files.
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>



>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

I'd have to agree. Bandwidth is the least of the concern. Control is
paramount.

On Tue, Jul 30, 2019 at 7:26 AM Henrik K <hege@hege.li> wrote:

>
> Control. Is it really necessary to go over basic IT management practises
> here?
>
> On Tue, Jul 30, 2019 at 05:13:50PM +0000, Joel Esler (jesler) via
> clamav-users wrote:
> > I'm interested as to why people want to do private mirrors? Other than
> to save bandwidth going to "the internet"?
> >
> > > On Jul 30, 2019, at 9:40 AM, J.R. via clamav-users <
> clamav-users@lists.clamav.net> wrote:
> > >
> > >> Can you please tell me the H/W and S/W Specification
> > >> of the Private local Mirror Server as a best practice for CVD?!
> > >
> > > https://www.clamav.net/documents/private-local-mirrors
> > >
> > > It's going to depend on how many clients you will be serving...
> > > 10 vs 10,000 is a huge difference in hardware requirements.
> > >
> > > Realistically though, no matter which route you take, it is just
> > > clients downloading static content at various intervals, which is not
> > > very CPU intensive. You shouldn't need anything *that* powerful to
> > > serve the files.
> > >
> > > _______________________________________________
> > >
> > > clamav-users mailing list
> > > clamav-users@lists.clamav.net
> > > https://lists.clamav.net/mailman/listinfo/clamav-users
> > >
> > >
> > > Help us build a comprehensive ClamAV guide:
> > > https://github.com/vrtadmin/clamav-faq
> > >
> > > http://www.clamav.net/contact.html#ml
> >
>
>
>
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

Originally, when we were pulling ClamAV updates over a slowish DSL
line, I wanted to save bandwidth, and a Private Mirror looked ideal.
Then, when we started using the much faster cable link, I just kept it,
since it worked, and it also kept machines like our file server isolated
from the Internet.

Then, when we had trouble with Cloudflare's BOS server often being out
of sync (for CVDs) with the DNS TXT record, I removed it. Now, I am
dismayed that I have to give our file server a bit of Internet access so
that it can directly download the CDIFFs.


On Tue, 30 Jul 2019 17:13:50 +0000
"Joel Esler \(jesler\) via clamav-users"
<clamav-users@lists.clamav.net> wrote:

> I'm interested as to why people want to do private mirrors? Other
> than to save bandwidth going to "the internet"?
>
> > On Jul 30, 2019, at 9:40 AM, J.R. via clamav-users
> > <clamav-users@lists.clamav.net> wrote:
> >
> >> Can you please tell me the H/W and S/W Specification
> >> of the Private local Mirror Server as a best practice for CVD?!
> >
> > https://www.clamav.net/documents/private-local-mirrors
> >
> > It's going to depend on how many clients you will be serving...
> > 10 vs 10,000 is a huge difference in hardware requirements.
> >
> > Realistically though, no matter which route you take, it is just
> > clients downloading static content at various intervals, which is
> > not very CPU intensive. You shouldn't need anything *that* powerful
> > to serve the files.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Before retiring I had a requirement to place an AV tool on all our Unix systems,
most of which did not have direct internet access. They were distributed across
several subnets, as well. A single local mirror was able to handle the load, and
our load on the ClamAV mirror farm was not impacted. The mirror was a VM, and
most of the memory was used as file system cache as there was nothing else
running on the box. It was very effective and provided a single point of logging
for the updates.

A similar requirement a couple years earlier was solved by integrating a local
mirror with CFEngine to push signatures to the systems on a schedule that
ensured redundant systems were not all reloading signatures at the same time.

dp

On 7/30/19 10:13 AM, Joel Esler (jesler) via clamav-users wrote:
> I'm interested as to why people want to do private mirrors? Other than to save bandwidth going to "the internet"?
>


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Part I needed:

> On Jul 30, 2019, at 1:25 PM, Henrik K <hege@hege.li> wrote:
>
> Control.


Part I didn't need:


> Is it really necessary to go over basic IT management practises here?

Thanks Dennis.

More than anything I was simply curious as to why people do this, each for their own reasons.

Sent from my ? iPad

> On Jul 30, 2019, at 14:49, Dennis Peterson <dennispe@inetnw.com> wrote:
>
> ?Before retiring I had a requirement to place an AV tool on all our Unix systems, most of which did not have direct internet access. They were distributed across several subnets, as well. A single local mirror was able to handle the load, and our load on the ClamAV mirror farm was not impacted. The mirror was a VM, and most of the memory was used as file system cache as there was nothing else running on the box. It was very effective and provided a single point of logging for the updates.
>
> A similar requirement a couple years earlier was solved by integrating a local mirror with CFEngine to push signatures to the systems on a schedule that ensured redundant systems were not all reloading signatures at the same time.
>
> dp
>
>> On 7/30/19 10:13 AM, Joel Esler (jesler) via clamav-users wrote:
>> I'm interested as to why people want to do private mirrors? Other than to save bandwidth going to "the internet"?
>>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

> Then, when we had trouble with Cloudflare's BOS server often being out
> of sync (for CVDs) with the DNS TXT record, I removed it. Now, I am
> dismayed that I have to give our file server a bit of Internet access so
> that it can directly download the CDIFFs.

I remember issue where some proxy was caching stale copies of daily.cvd...

If you don't want to let your file server access the internet
directly, did you ever try setting up a proxy server (and configure
freshclam to use it)? That would solve the direct access dilemma, and
also cache cdiff (and other) files locally to save bandwidth.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> On Jul 31, 2019, at 9:52 AM, J.R. via clamav-users <clamav-users@lists.clamav.net> wrote:
>
>> Then, when we had trouble with Cloudflare's BOS server often being out
>> of sync (for CVDs) with the DNS TXT record, I removed it. Now, I am
>> dismayed that I have to give our file server a bit of Internet access so
>> that it can directly download the CDIFFs.
>
> I remember issue where some proxy was caching stale copies of daily.cvd...
>
> If you don't want to let your file server access the internet
> directly, did you ever try setting up a proxy server (and configure
> freshclam to use it)? That would solve the direct access dilemma, and
> also cache cdiff (and other) files locally to save bandwidth.


I think that's the intended purpose of the local private mirror in this case.

The only problem with the local mirrors, from our point of view are a couple things:

1. I don't know how many users we have
2. Out of those users, what versions they are running.

Etc.

On Wed, Jul 31, 2019 at 02:49:33PM +0000, Joel Esler (jesler) via clamav-users wrote:
>
> The only problem with the local mirrors, from our point of view are a couple things:
>
> 1. I don't know how many users we have

Would not private mirror users be usually a single organization, so in
practise a single "user"? Why do you need to know how many servers they
have?

Private mirror users would be very much minority anyway looking at the big
picture.

> 2. Out of those users, what versions they are running.

Assuming competent admin, they all run the same version.

I don't see how you can find these "problem" in the least.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> On Jul 31, 2019, at 11:04 AM, Henrik K <hege@hege.li> wrote:
>
> On Wed, Jul 31, 2019 at 02:49:33PM +0000, Joel Esler (jesler) via clamav-users wrote:
>>
>> The only problem with the local mirrors, from our point of view are a couple things:
>>
>> 1. I don't know how many users we have
>
> Would not private mirror users be usually a single organization, so in
> practise a single "user"? Why do you need to know how many servers they
> have?

You know how often I get asked how many users we have?

A lot.

>
> Private mirror users would be very much minority anyway looking at the big
> picture.
>
>> 2. Out of those users, what versions they are running.
>
> Assuming competent admin, they all run the same version.


Your assumptions are proven wrong, by looking at the statistics of the users we have today.

On Wed, Jul 31, 2019 at 03:33:59PM +0000, Joel Esler (jesler) via clamav-users wrote:
>
> Would not private mirror users be usually a single organization, so in
> practise a single "user"? Why do you need to know how many servers they
> have?
>
>
> You know how often I get asked how many users we have?
>
> A lot.

There's some difference in talking about users or number of installations.

> Private mirror users would be very much minority anyway looking at the big
> picture.
>
>
> 2. Out of those users, what versions they are running.
>
>
> Assuming competent admin, they all run the same version.
>
>
>
> Your assumptions are proven wrong, by looking at the statistics of the users we
> have today.

How do you see statistics from users behind a private mirror? My answer to
2 really referred to those installations (yes I should have said that
instead of "users").

You do see the private mirror instance freshclam statistics, so that's what
you have to go with. Just assume some of those queries have hundreds of
servers behind them.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

The problem with our using a Web proxy is that it too cached stale CVDs
if it was using the BOS Cloudflare server. That is, the DNS TXT record
reported a new CVD, but the proxy couldn't deliver it. I considered
using a proxy on our offsite domain host (which happened not to use
BOS), but that seemed to be too complicated, considering our small LAN.

What I finally did was have each machine on our LAN point freshclam
directly at Cloudflare, *but* I modified our firewall to let the file
server access *only* the two Cloudflare IP addresses at port 80 and
nothing else. (Why do computers always need so many special case
workarounds?)


On Wed, 31 Jul 2019 08:52:49 -0500
"J.R. via clamav-users" <clamav-users@lists.clamav.net> wrote:

> > Then, when we had trouble with Cloudflare's BOS server often being
> > out of sync (for CVDs) with the DNS TXT record, I removed it. Now,
> > I am dismayed that I have to give our file server a bit of Internet
> > access so that it can directly download the CDIFFs.
>
> I remember issue where some proxy was caching stale copies of
> daily.cvd...
>
> If you don't want to let your file server access the internet
> directly, did you ever try setting up a proxy server (and configure
> freshclam to use it)? That would solve the direct access dilemma, and
> also cache cdiff (and other) files locally to save bandwidth.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> I think that's the intended purpose of the local private mirror in this case.
>

I realize that, but I believe in that person's case back the he was
doing a basic web server to re-distributed the full .cvd files (which
is what were getting stale). Whereas doing a proxy server (like squid)
would be more transparent and fetch the .cdiff files, which are always
unique each time there is an update.

> The only problem with the local mirrors, from our point of view are a couple things:
>
> 1. I don't know how many users we have
> 2. Out of those users, what versions they are running.

I vaguely remember a discussion a while back about ClamAV's anonymous
statistics got removed some time ago? Was there any plan to
re-implement? I think 3 different choices for the end-user would be
all you would need:

1. Don't Participate
2. Send anonymous version info.
3. Send #2 + daily viruses caught.

If you started getting feedback on which viruses were the most
frequent, then you could start publishing live statistics!

I would think you could do some basic calculations based on the
cloudflare data on how many clients are grabbing the updates, and also
use those IPs to determine country usage.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Inline below:

> On Aug 1, 2019, at 11:33 PM, J.R. via clamav-users <clamav-users@lists.clamav.net> wrote:
>
>> I think that's the intended purpose of the local private mirror in this case.
>>
>
> I realize that, but I believe in that person's case back the he was
> doing a basic web server to re-distributed the full .cvd files (which
> is what were getting stale). Whereas doing a proxy server (like squid)
> would be more transparent and fetch the .cdiff files, which are always
> unique each time there is an update.
>
>> The only problem with the local mirrors, from our point of view are a couple things:
>>
>> 1. I don't know how many users we have
>> 2. Out of those users, what versions they are running.
>
> I vaguely remember a discussion a while back about ClamAV's anonymous
> statistics got removed some time ago?

Technically, the server is still up, and there are lots of people reporting stats to it, but they are very legacy customers that have probably forgotten to upgrade their ClamAV installations (since getting a new user account has been disabled for about 4 years).

> Was there any plan to
> re-implement?

Yes. Micah and I have discussed this recently in fact. Ideally this would be coupled with a user portal on ClamAV.net to be able to display statistics and other information. Happy to collect ideas here on what you all would see as useful. We don't have a timeline on implementation of any of this, since it involves some work on my team's side (API, Website, etc) and then some work on the ClamAV team's (of which Micah is a member) side to implement that API and what statistics to report to that API. But to be honest, the ClamAV team has some higher priorities right now.


> I think 3 different choices for the end-user would be
> all you would need:
>
> 1. Don't Participate
> 2. Send anonymous version info.
> 3. Send #2 + daily viruses caught.

We are thinking something similar, but more in depth. But definitely different levels of participation.

>
> If you started getting feedback on which viruses were the most
> frequent, then you could start publishing live statistics!

Like I said, we have that information now, but from legacy customers, and I am not sure how useful that is since they aren't using the latest signatures.

>
> I would think you could do some basic calculations based on the
> cloudflare data on how many clients are grabbing the updates, and also
> use those IPs to determine country usage.

Yes. Can do that now. Things we can see now, and in the interest of transparency, here's some numbers:

Countries of usage Nearly every country on earth uses ClamAV. Highest countries of usage are United States, Germany, and Taiwan. In that order.
Speed of update (from the time we publish an update, to how long it takes people to download said update). A lot people download it immediately, judging by the fact that immediately after we publish a daily we push about 5TB of traffic within an hour.
How many unique IPs download the updates (around 14M)
How much data is downloaded per day -- about 44 TB
Version of ClamAV user x amount of times they check for updates: I just selected the top 10 here. BTW -- that "0.92.1-exp" is one person.