Mailing List Archive

Manually copy and use local filesystem as DownloadMirror/PrivateMirror
Hi All,

I needed some clarifications in configuring clamav on our client machines.

We have several client machines and the client machines we have cannot contact the official clamav server to fetch the cvd and cdiff files. And hosting a private server and setting is up as a DownloadMirror is also not possible in our case since we have many clients, and we'll have to setup and maintain a server in network of each of these clients.

However, we provide these client machines with an update periodically (once in a quarter as of now) Thereby, I was considering the possibility of pushing the virus definition files as part of a client machine update. I can have a machine in my local network where I can download the cvd and cdiff files as part of cvdupdate and then push these to the client machines as part of the update. I had a few questions related to these, would really appreciate some help

1) If I place the cvd files and cdiff file in a temporary location within the machine, is it possible to use that location in local filesystem as DownloadMirrror/PrivateMirror so that freshclam can merge the cvd and cdiff files (or any other way to do this, to avoid having several cdiff files). I couldn't find any info on this in the documentation.

2) If I place the cvd files and consequent cdiff files in /var/lib/clamav, will clamd consider only the cvd files, or would it consider the cdiffs as well? (If I can't use freshclam on local filesystem)

3) Is there any better way to approach this? I know that having a quarterly update of virus definitions leave the machines at risk. The clients can keep the cvds updated if they want to. But I expect a lot of the customers to not keep the cvds updated and was thinking of a best possible way to address them. I am also aware of the 90 days limit on the cdiffs available. So, if this approach doesn't make sense for quarterly cycle, I can think of pushing them each month.

Thanks a lot,
Anish
Re: Manually copy and use local filesystem as DownloadMirror/PrivateMirror [ In reply to ]
On 17/05/2021 23:24, ANISH SHETTY via clamav-users wrote:
> Hi All,
>
> I needed some clarifications in configuring clamav on our client machines.
>
> We have several client machines and the client machines we have cannot
> contact the official clamav server to fetch the cvd and cdiff files. And
> hosting a private server and setting is up as a DownloadMirror is also
> not possible in our case since we have many clients, and we'll have to
> setup and maintain a server in network of each of these clients.
>
Why not set up a single proxy accessible by all the clients?

> However, we provide these client machines with an update periodically
> (once in a quarter as of now)
Okay, this is a bit of a "Why bother?" You'll be so out of date that it
seems hardly worth it.

[.SNIP the rest, someone who knows things better can have a go]

Cheers,
Gary B-)

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Manually copy and use local filesystem as DownloadMirror/PrivateMirror [ In reply to ]
Anish,

What sort of scanning are you doing on these client machines ?
Which databases are you using with ClamAV ?
What data is stored on these clients ?
What operating system(s) are they running ?
I ask since the way some of us run ClamAV there is
little benefit on running it on each client machine.

On Mon, 17 May 2021, ANISH SHETTY via clamav-users wrote:

> Hi All,
>
> I needed some clarifications in configuring clamav on our client machines.
>
> We have several client machines and the client machines we have cannot contact the official clamav server to fetch the cvd and cdiff files. And hosting a private server and setting is up as a DownloadMirror is also not possible in our case since we have many clients, and we'll have to setup and maintain a server in network of each of these clients.

I believe that the download mirror can be on a different network as
long as the client can see and read it, so you may not need as many
servers as you think.

> However, we provide these client machines with an update periodically (once in a quarter as of now) Thereby, I was considering the possibility of pushing the virus definition files as part of a client machine update. I can have a machine in my local network where I can download the cvd and cdiff files as part of cvdupdate and then push these to the client machines as part of the update. I had a few questions related to these, would really appreciate some help
>
> 1) If I place the cvd files and cdiff file in a temporary location within the machine, is it possible to use that location in local filesystem as DownloadMirrror/PrivateMirror so that freshclam can merge the cvd and cdiff files (or any other way to do this, to avoid having several cdiff files). I couldn't find any info on this in the documentation.
>
> 2) If I place the cvd files and consequent cdiff files in /var/lib/clamav, will clamd consider only the cvd files, or would it consider the cdiffs as well? (If I can't use freshclam on local filesystem)
>
> 3) Is there any better way to approach this? I know that having a quarterly update of virus definitions leave the machines at risk. The clients can keep the cvds updated if they want to. But I expect a lot of the customers to not keep the cvds updated and was thinking of a best possible way to address them. I am also aware of the 90 days limit on the cdiffs available. So, if this approach doesn't make sense for quarterly cycle, I can think of pushing them each month.

Clam people:
if the machines are rebooted (not just hibernated) daily,
could the .cld (probably not .cvd) files be mounted from a network share
(kept updated by running freshclam on the server),
rather than each client running freshclam ?

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Manually copy and use local filesystem as DownloadMirror/PrivateMirror [ In reply to ]
Hi Andrew. Thank you for reply.

The machines are running SLES12 OS. We don't have much data in the machines. It runs a webserver and has a few configuration details in it. The clients aren't expected to install any new softwares other than the ones we have already installed on it or copy any files to the machine. So yes, it may not be that beneficial to setup ClamAV in these machines. But we need to have an antivirus solution in place to meet some compliance requirements mandated by the government and the clients should be able to run the scans if they need to. I am not planning to set up any scans by default.


________________________________
From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Andrew C Aitchison via clamav-users <clamav-users@lists.clamav.net>
Sent: 17 May 2021 20:13
To: ANISH SHETTY via clamav-users <clamav-users@lists.clamav.net>
Cc: Andrew C Aitchison <clamav@aitchison.me.uk>
Subject: Re: [clamav-users] Manually copy and use local filesystem as DownloadMirror/PrivateMirror


Anish,

What sort of scanning are you doing on these client machines ?
Which databases are you using with ClamAV ?
What data is stored on these clients ?
What operating system(s) are they running ?
I ask since the way some of us run ClamAV there is
little benefit on running it on each client machine.

On Mon, 17 May 2021, ANISH SHETTY via clamav-users wrote:

> Hi All,
>
> I needed some clarifications in configuring clamav on our client machines.
>
> We have several client machines and the client machines we have cannot contact the official clamav server to fetch the cvd and cdiff files. And hosting a private server and setting is up as a DownloadMirror is also not possible in our case since we have many clients, and we'll have to setup and maintain a server in network of each of these clients.

I believe that the download mirror can be on a different network as
long as the client can see and read it, so you may not need as many
servers as you think.

> However, we provide these client machines with an update periodically (once in a quarter as of now) Thereby, I was considering the possibility of pushing the virus definition files as part of a client machine update. I can have a machine in my local network where I can download the cvd and cdiff files as part of cvdupdate and then push these to the client machines as part of the update. I had a few questions related to these, would really appreciate some help
>
> 1) If I place the cvd files and cdiff file in a temporary location within the machine, is it possible to use that location in local filesystem as DownloadMirrror/PrivateMirror so that freshclam can merge the cvd and cdiff files (or any other way to do this, to avoid having several cdiff files). I couldn't find any info on this in the documentation.
>
> 2) If I place the cvd files and consequent cdiff files in /var/lib/clamav, will clamd consider only the cvd files, or would it consider the cdiffs as well? (If I can't use freshclam on local filesystem)
>
> 3) Is there any better way to approach this? I know that having a quarterly update of virus definitions leave the machines at risk. The clients can keep the cvds updated if they want to. But I expect a lot of the customers to not keep the cvds updated and was thinking of a best possible way to address them. I am also aware of the 90 days limit on the cdiffs available. So, if this approach doesn't make sense for quarterly cycle, I can think of pushing them each month.

Clam people:
if the machines are rebooted (not just hibernated) daily,
could the .cld (probably not .cvd) files be mounted from a network share
(kept updated by running freshclam on the server),
rather than each client running freshclam ?

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Manually copy and use local filesystem as DownloadMirror/PrivateMirror [ In reply to ]
Hi there,

On Mon, 17 May 2021, ANISH SHETTY via clamav-users wrote:

> The machines are running SLES12 ... we need to have an antivirus
> solution in place to meet some compliance requirements mandated by
> the government ...

I used to work for our government (the UK's nuclear power programme,
mostly on security, but that's not important). The site I worked at
was out in the boondocks (in case it blew up) and was home to about
5,000 people. There was a sports club, financed by the government,
just outside the perimiter fence. I went to that club for ten years.
A guy - a plumber by trade - a big fellow, who used to throw us all
around at the judo sessions, said to us out of the blue one day,

"An elephant is a greyhound, built to government specifications."

That made us fall about laughing.

I feel your pain.

My 'virusdb' mail box contains the daily feed of mail messages from
the ClamAV virus DB updates. As you see below in the last three weeks
there have on average been more than 350 new virus signatures per day.
This is quite apart from the typically more than a dozen but perhaps
as many as 50 signatures which might daily be dropped.

$ grep 'New Sigs' ~/mail/lists/virusdb | tail -n 20
New Sigs: 762
New Sigs: 283
New Sigs: 244
New Sigs: 119
New Sigs: 325
New Sigs: 197
New Sigs: 367
New Sigs: 432
New Sigs: 453
New Sigs: 406
New Sigs: 525
New Sigs: 235
New Sigs: 249
New Sigs: 401
New Sigs: 628
New Sigs: 95
New Sigs: 172
New Sigs: 69
New Sigs: 221
New Sigs: 853
New Sigs: 372

> ... if this approach doesn't make sense for quarterly cycle, I can
> think of pushing them each month.

Apart from just complying with some crackpot regulations, I think
you're wasting your time. You may risk giving yourself (and perhaps
ClamAV) a bad reputation with your clients. Based on the numbers
above, even if you update every month you can expect to be missing
over ten thousand signatures after 30 days, and you'll have quite a
few which are known to be suspect - some of which could be false
positives - which clients will be stuck with for a month, and which
may even be more trouble to you than the signatures you don't have.
At least there's the option of maintaining your own 'ignore' lists.

Having said all that

$ grep ' \* ' ./mail/lists/virusdb | tail -n 10000 | sed -e 's/\..*//;' | sort | uniq -c | sort -n
1 * Andr
1 * Lnk
1 * Rtf
1 * Swf
1 * Vbs
2 * Img
2 * Osx
2 * Ps1
3 * Doc
4 * Ole2
4 * Xls
16 * PUA
19 * Archive
35 * Pdf
36 * Multios
63 * Unix
68 * Email
98 * Txt
765 * Html
8878 * Win

as you can see the vast majority of virus signatures are for Windows
threats, to which your SLES machines are immune. That doesn't mean
that they couldn't be compromised and then used to attack machines
which are not immune.

If you can keep a local copy of the database up to date and you have
direct (write) access to the client machines there must be dozens of
ways to keep them updated from a local copy. For example you could
schedule a task on each client to update its own temporary copy from
your master, then replace the working copy with the temporary copy on
the client in some way that makes the operation atomic. Without more
information about the connectivity issues your clients face I can't
really offer more than hand-waving suggestions like that, but just
from the point of view of network traffic I would urge you to look
into ways of making freshclam do something for you rather than trying
to re-invent any wheels. Perhaps you could have a mirror in each
client network which takes its data from a further mirror which you
maintain in your network. Presumably if the clients are running Web
servers on SLES, one (or more) of the client machines in each client
network could also run a mirror for the local network?

Have you looked at anything like 'Puppet'?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Manually copy and use local filesystem as DownloadMirror/PrivateMirror [ In reply to ]
Hey Ged, Thank you for reply.


The network connectivity issue is indeed the main blocker for me. Perhaps I should have been clearer on that. For now, the update operation performed by clients is the only time when I can manage what data goes into the machines. I can't setup a cron or such alternatives. I could use the web server on the same machine as a server for freshclam (since I guess I need a webserver and can't do it from local filesystem). But like a lot a few people have already mentioned, the signatures will be out of date. And the false positives were something I hadn't considered (thanks for that ). Given all this, I'm not sure if it's worth the effort. I'll see if I can think of any other approaches where the client machines can access a server which is kept up to date.


Anish

________________________________
From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of G.W. Haywood via clamav-users <clamav-users@lists.clamav.net>
Sent: 17 May 2021 23:06
To: ANISH SHETTY via clamav-users <clamav-users@lists.clamav.net>
Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
Subject: Re: [clamav-users] Manually copy and use local filesystem as DownloadMirror/PrivateMirror

Hi there,

On Mon, 17 May 2021, ANISH SHETTY via clamav-users wrote:

> The machines are running SLES12 ... we need to have an antivirus
> solution in place to meet some compliance requirements mandated by
> the government ...

I used to work for our government (the UK's nuclear power programme,
mostly on security, but that's not important). The site I worked at
was out in the boondocks (in case it blew up) and was home to about
5,000 people. There was a sports club, financed by the government,
just outside the perimiter fence. I went to that club for ten years.
A guy - a plumber by trade - a big fellow, who used to throw us all
around at the judo sessions, said to us out of the blue one day,

"An elephant is a greyhound, built to government specifications."

That made us fall about laughing.

I feel your pain.

My 'virusdb' mail box contains the daily feed of mail messages from
the ClamAV virus DB updates. As you see below in the last three weeks
there have on average been more than 350 new virus signatures per day.
This is quite apart from the typically more than a dozen but perhaps
as many as 50 signatures which might daily be dropped.

$ grep 'New Sigs' ~/mail/lists/virusdb | tail -n 20
New Sigs: 762
New Sigs: 283
New Sigs: 244
New Sigs: 119
New Sigs: 325
New Sigs: 197
New Sigs: 367
New Sigs: 432
New Sigs: 453
New Sigs: 406
New Sigs: 525
New Sigs: 235
New Sigs: 249
New Sigs: 401
New Sigs: 628
New Sigs: 95
New Sigs: 172
New Sigs: 69
New Sigs: 221
New Sigs: 853
New Sigs: 372

> ... if this approach doesn't make sense for quarterly cycle, I can
> think of pushing them each month.

Apart from just complying with some crackpot regulations, I think
you're wasting your time. You may risk giving yourself (and perhaps
ClamAV) a bad reputation with your clients. Based on the numbers
above, even if you update every month you can expect to be missing
over ten thousand signatures after 30 days, and you'll have quite a
few which are known to be suspect - some of which could be false
positives - which clients will be stuck with for a month, and which
may even be more trouble to you than the signatures you don't have.
At least there's the option of maintaining your own 'ignore' lists.

Having said all that

$ grep ' \* ' ./mail/lists/virusdb | tail -n 10000 | sed -e 's/\..*//;' | sort | uniq -c | sort -n
1 * Andr
1 * Lnk
1 * Rtf
1 * Swf
1 * Vbs
2 * Img
2 * Osx
2 * Ps1
3 * Doc
4 * Ole2
4 * Xls
16 * PUA
19 * Archive
35 * Pdf
36 * Multios
63 * Unix
68 * Email
98 * Txt
765 * Html
8878 * Win

as you can see the vast majority of virus signatures are for Windows
threats, to which your SLES machines are immune. That doesn't mean
that they couldn't be compromised and then used to attack machines
which are not immune.

If you can keep a local copy of the database up to date and you have
direct (write) access to the client machines there must be dozens of
ways to keep them updated from a local copy. For example you could
schedule a task on each client to update its own temporary copy from
your master, then replace the working copy with the temporary copy on
the client in some way that makes the operation atomic. Without more
information about the connectivity issues your clients face I can't
really offer more than hand-waving suggestions like that, but just
from the point of view of network traffic I would urge you to look
into ways of making freshclam do something for you rather than trying
to re-invent any wheels. Perhaps you could have a mirror in each
client network which takes its data from a further mirror which you
maintain in your network. Presumably if the clients are running Web
servers on SLES, one (or more) of the client machines in each client
network could also run a mirror for the local network?

Have you looked at anything like 'Puppet'?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Manually copy and use local filesystem as DownloadMirror/PrivateMirror [ In reply to ]
Hi there,

On Tue, 18 May 2021, ANISH SHETTY via clamav-users wrote:

> ... the update operation performed by clients is the only time when
> I can manage what data goes into the machines. I can't setup a cron
> or such alternatives. I could use the web server on the same machine
> as a server for freshclam (since I guess I need a webserver and
> can't do it from local filesystem).

Yes, freshclam only obtains the signature data via Web servers, and
cannot usefully access the local filesystem as an alternative. You
can in the case of a multi-homed machine specify which interface is to
be used for the downloads. In case it's also an issue for you, the
current state of the database is held in DNS records. That means that
freshclam should also have access to a nameserver, so that it can make
the DNS queries to get the information which it needs in order to know
if the signature databases are up to date. Although we call them the
'signature databases' they are in fact just ordinary files. Some are
compressed (and signed), but you can uncompress them to plain, flat,
text files which you can display with almost any pager or text editor
(and which I occasionally do to investigate signature issues).

If I understand correctly, the clients disable the network connection
most of the time, and enable it only every three months to do some
sort of update operation; it might be possible to get them to do this
once per month, is that correct? Is the update operation to be purely
for the ClamAV databases or is it also for some kind of maintenance of
other software and/or data?

> Given all this, I'm not sure if it's worth the effort. I'll see if
> I can think of any other approaches where the client machines can
> access a server which is kept up to date.

It does not matter what the Web server is - it could be a proxy like
Squid for example. You could update the files which Squid serves in
whatever way you choose, and of course prevent it from accessing any
data other than your signature databases. I do not know enough about
the restrictions in your networks to know if that might help.

It does not matter to ClamAV (that is, to the scanners - clamdscan,
clamscan and clamd) how the signature files are kept up to date. But
it matters to the infrastructure how the downloads are performed, as
there are abuse protections in place which will probably be activated
if freshclam (and it must be a fairly up to date version of freshclam)
is not used. That would mean that the IP address trying to download
the signatures will be blocked by the infrastructure provider and you
would need to ask for it to be unblocked after rectifying any issues.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Manually copy and use local filesystem as DownloadMirror/PrivateMirror [ In reply to ]
Hi Ged,

> If I understand correctly, the clients disable the network connection
> most of the time, and enable it only every three months to do some
> sort of update operation; it might be possible to get them to do this
> once per month, is that correct? Is the update operation to be purely
> for the ClamAV databases or is it also for some kind of maintenance of
> other software and/or data?

Yes, this is how it is right now. The machines mostly use the only intranet. They connect to our severs when they must perform an update. We perform maintenance of other software, and I can fit in the updates of the virus definitions here.

Thanks a lot in clearing up my doubts related to freshclam. I am considering setting up a webserver now . I will have a discussion with my higher ups with all the inputs I've got here and see if our clients would be okay with this.

Anish.

________________________________
From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of G.W. Haywood via clamav-users <clamav-users@lists.clamav.net>
Sent: 18 May 2021 14:16
To: ANISH SHETTY via clamav-users <clamav-users@lists.clamav.net>
Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
Subject: Re: [clamav-users] Manually copy and use local filesystem as DownloadMirror/PrivateMirror

Hi there,

On Tue, 18 May 2021, ANISH SHETTY via clamav-users wrote:

> ... the update operation performed by clients is the only time when
> I can manage what data goes into the machines. I can't setup a cron
> or such alternatives. I could use the web server on the same machine
> as a server for freshclam (since I guess I need a webserver and
> can't do it from local filesystem).

Yes, freshclam only obtains the signature data via Web servers, and
cannot usefully access the local filesystem as an alternative. You
can in the case of a multi-homed machine specify which interface is to
be used for the downloads. In case it's also an issue for you, the
current state of the database is held in DNS records. That means that
freshclam should also have access to a nameserver, so that it can make
the DNS queries to get the information which it needs in order to know
if the signature databases are up to date. Although we call them the
'signature databases' they are in fact just ordinary files. Some are
compressed (and signed), but you can uncompress them to plain, flat,
text files which you can display with almost any pager or text editor
(and which I occasionally do to investigate signature issues).

If I understand correctly, the clients disable the network connection
most of the time, and enable it only every three months to do some
sort of update operation; it might be possible to get them to do this
once per month, is that correct? Is the update operation to be purely
for the ClamAV databases or is it also for some kind of maintenance of
other software and/or data?

> Given all this, I'm not sure if it's worth the effort. I'll see if
> I can think of any other approaches where the client machines can
> access a server which is kept up to date.

It does not matter what the Web server is - it could be a proxy like
Squid for example. You could update the files which Squid serves in
whatever way you choose, and of course prevent it from accessing any
data other than your signature databases. I do not know enough about
the restrictions in your networks to know if that might help.

It does not matter to ClamAV (that is, to the scanners - clamdscan,
clamscan and clamd) how the signature files are kept up to date. But
it matters to the infrastructure how the downloads are performed, as
there are abuse protections in place which will probably be activated
if freshclam (and it must be a fairly up to date version of freshclam)
is not used. That would mean that the IP address trying to download
the signatures will be blocked by the infrastructure provider and you
would need to ask for it to be unblocked after rectifying any issues.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml