Mailing List Archive

Hi there,

On Wed, 30 Sep 2020, Dave Sill via clamav-users wrote:

> The clamscan man page says:
>
> --disable-cache
> Disable caching and cache checks for hash sums of scanned files.
>
> I've looked high and low via google, strace, looking at source code, conducting tests,
> and I see no sign of caching done by clamscan. Is this on the to-do list?

Perhaps the output below will help. In it, I've added only a couple
of blank lines for clarity. No other changes at all. Look carefully.

8<----------------------------------------------------------------------
$ ls -l clamav-0.103.0-rc2.tar.gz clamav-0.103.0-rc2-copy.tar.gz
-rw-r--r-- 1 ged ged 13356212 Sep 30 13:14 clamav-0.103.0-rc2-copy.tar.gz
-rw-r--r-- 1 ged ged 13356212 Sep 3 10:14 clamav-0.103.0-rc2.tar.gz

$ clamscan clamav-0.103.0-rc2.tar.gz
/home/ged/clamav-0.103.0-rc2.tar.gz: OK

----------- SCAN SUMMARY -----------
Known viruses: 11312903
Engine version: 0.103.0-rc2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 13.56 MB
Data read: 12.73 MB (ratio 1.06:1)
Time: 88.628 sec (1 m 28 s)
Start Date: 2020:09:30 13:17:25
End Date: 2020:09:30 13:18:54

$ clamscan clamav-0.103.0-rc2.tar.gz clamav-0.103.0-rc2-copy.tar.gz
/home/ged/clamav-0.103.0-rc2.tar.gz: OK
/home/ged/clamav-0.103.0-rc2-copy.tar.gz: OK

----------- SCAN SUMMARY -----------
Known viruses: 11312903
Engine version: 0.103.0-rc2
Scanned directories: 0
Scanned files: 2
Infected files: 0
Data scanned: 13.56 MB
Data read: 25.47 MB (ratio 0.53:1)
Time: 88.867 sec (1 m 28 s)
Start Date: 2020:09:30 13:19:43
End Date: 2020:09:30 13:21:12
8<----------------------------------------------------------------------

In the second scan, how did clamscan manage to do what it claims to
have done in the time that it did it?

> We'd like to regularly scan systems but the overhead of scanning the same files
> repeatedly is a bit much. A per-system cache would be good. Even better would be a
> centralized, local to our network, service that clamscan could check & update.

For further enlightenment, on one of your systems try doing something
similar to what I did above but using 'clamdscan'. Consider using a
central clamd server for all your scanning needs.

> I suppose that would be pretty easy to add on top.
>
> Anyone done anything like that?

I doubt anyone is doing that. I'm sure it isn't necessary, as it's
already taken care of by both clamscan and clamd. Perhaps if you can
be a bit more forthcoming about your use case(s) we may be able to
help reduce scan times. One of the best ways of doing that is not to
scan so much junk so often.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Wed, 30 Sep 2020, Dave Sill via clamav-users wrote:

> "G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net> wrote:
>>
>> In the second scan, how did clamscan manage to do what it claims to
>> have done in the time that it did it?
>
> OK, you could have just said that the cache is internal to each invocation
> of clamscan, but that helps.
>
>> For further enlightenment, on one of your systems try doing something
>> similar to what I did above but using 'clamdscan'.
>
> The problem with clamdscan is that it runs into permissions since it's
> not running as root.
>
>> Consider using a
>> central clamd server for all your scanning needs.
>
> How would that work? Clamd only scans files on the system on which it's
> running.

No. clamD scans data passed to it by clamdscan, usually over a socket or
pipe. As a special case,
clamdscan --fdpass filename
passes an open file handle (the man page suggests that that is not
technically accurate) to clamd which means clamd can scan any file which
clamdscan can read, avoiding the running as root problem. --fdpass only
works over local (unix) sockets, not network (tcp) sockets.


>> I doubt anyone is doing that. I'm sure it isn't necessary, as it's
>> already taken care of by both clamscan and clamd. Perhaps if you can
>> be a bit more forthcoming about your use case(s) we may be able to
>> help reduce scan times. One of the best ways of doing that is not to
>> scan so much junk so often.
>
> We've got about 3000 Linux systems that we'd like to periodically scan,
> primarily to ensure that they're not being used to redistribute
> Windows malware. We'd like to scan all of the local file systems for
> completeness. Any attempt to skip "junk" will potentially skip malware,
> and hand crafting scans for each system is not an option.
>
> Skipping multiple copies of the same file won't really help because
> the duplication is across systems, and because every file will be
> rescanned every time clamscan is run.
>
> We could do a full scan on the first run and then weekly scans of files
> modified in the past week. That's kludgy but may be the best we can do.

That does mean that any malware which is missed in the first run
will not be detected in subsequent runs.

3000 machines per week, gives you about 3.36 minutes for each machine to
send all its local data to the scanning machine.
Instead I would run a local, mirror, repository of the database
and use freshclam on each machine to keep its database in sync with your
mirror, then run clamd and a clamdscan cron? script on each machine.
I would also look at on-access scanning.
Scanning files as they are used might mean more or less work
than scanning every file every week.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Andrew C Aitchison via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> No. clamD scans data passed to it by clamdscan, usually over a socket or
> pipe.

Ah... I missed INSTREAM in the clamd man page. Locally, though, surely
SCAN/CONTSCAN/etc, are nuch more efficient. And remotely, sending the
entire contents of the system over the net isn't practical at scale.

> That does mean that any malware which is missed in the first run
> will not be detected in subsequent runs.

True. I suppose we'd want to do monthly full scans.

> 3000 machines per week, gives you about 3.36 minutes for each machine to
> send all its local data to the scanning machine.
> Instead I would run a local, mirror, repository of the database
> and use freshclam on each machine to keep its database in sync with your
> mirror, then run clamd and a clamdscan cron? script on each machine.

We've already got a local mirror. Is there a way to get clamd/clamdscan
to work without permission problems beside running clamd as root? Does
--fdpass get around that?

> I would also look at on-access scanning.

I tried it but got permission errors on anything not world-accessible.
I suspect the overall performance hit would be too high.

> Scanning files as they are used might mean more or less work
> than scanning every file every week.

Except full dumps are going to cause everything to be scanned.

-Dave

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

"G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net> wrote:
>
> There are ways around that, even if you don't want to run clamdscan
> (and clamd) as root - which I'd entirely understand.

Is --fdpass one of them? And --stream? Any others?

> >We've got about 3000 Linux systems that we'd like to periodically scan,
> >primarily to ensure that they're not being used to redistribute
> >Windows malware.
>
> A good use case, perhaps quite a tall order with a single clamd server
> but maybe doable if you can (a) limit what needs to be scanned and (b)
> define 'periodically' in terms of days (at least) and not hours.

We could use multiple clamd servers. And periodically in terms of days
would be OK.

> >Any attempt to skip "junk" will potentially skip malware, and hand
> >crafting scans for each system is not an option.
>
> That seems more like a management problem to me than a technical one.

The nature of our environment precludes actively managing all of the
Linux systems here, unfortunately.

> >Skipping multiple copies of the same file won't really help because
> >the duplication is across systems, and because every file will be
> >rescanned every time clamscan is run.
>
> That's not true of clamdscan.

Hmm...that's promising. I'll give it a try.

> And you probably won't know what's been modified in the past week unless
> you install Tripwire or something like that...

mtime would be sufficient for our purposes.

-Dave

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Wed, 30 Sep 2020, Dave Sill via clamav-users wrote:

> "G.W. Haywood via clamav-users" <clamav-users@lists.clamav.net> wrote:
>>
>> There are ways around that, even if you don't want to run clamdscan
>> (and clamd) as root - which I'd entirely understand.
>
> Is --fdpass one of them? And --stream? Any others?

No. I'd be thinking out-of-the-box, such as (dual) booting out of
office hours with a tiny OS just for ClamAV, maybe mount partitions
read-only; or you could have a process which does have the necessary
permissions temporarily copy the files that you want to scan to some
sandbox area; that kind of thing. It would need some effort, and in
your case it sounds like something like Puppet might help.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml