Mailing List Archive: [clamav-users] Heuristics.Limits.Exceeded FOUND

[clamav-users] Heuristics.Limits.Exceeded FOUND

Apr 3, 2020, 1:02 PM

Post #1 of 16 (1585 views)

I am puzzled (and dismayed) by the following behavior of ClamAV. When I
scan some archive files, I often get "Heuristics.Limits.Exceeded FOUND".
This makes me wonder about ClamAV's utility in protecting our systems
against malware.

I'm not talking about archive files downloaded from questionable
sources, or incredibly large files (like DVD ISO files). I'm talking
about files like the latest 64-bit Linux Firefox ESR binary file (65
MB) downloaded from "http://ftp.mozilla.org/pub/firefox/releases/".

Although this file passed its GPG signature verification, and its
SHA512 checksum test, I still like to use ClamAV for additional
checking (since even reputable distribution sites have been known to
have been compromised).

However, applying clamscan to this file (which was slightly renamed by
my download script to be more readable) results in the following output:

clamscan --alert-exceeds-max=yes --max-scantime=999 --max-scansize=4090M --max-filesize=4090M --max-files=30000 --max-recursion=30 --pcre-match-limit=999999999 --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2

firefox-68.6.1-esr-64.tar.bz2: Heuristics.Limits.Exceeded FOUND

This is in spite of the extra options I added to relax the "Limits".

So I decided that since the Firefox tar.bz2 file passed its GPG and
SHA512 tests, I would expand the into a temp directory and clamscan
the result. (Which files are those used directly to run Firefox):

clamscan --alert-exceeds-max=yes --max-scantime=999 --max-scansize=4090M --max-filesize=4090M --max-files=30000 --max-recursion=30 --pcre-match-limit=999999999 --pcre-max-filesize=999999999 -r firefox | grep -v ' OK'
firefox/removed-files: Empty file
firefox/chrome.manifest: Empty file
firefox/browser/chrome.manifest: Empty file
firefox/browser/omni.ja: Heuristics.Limits.Exceeded FOUND
firefox/omni.ja: Heuristics.Limits.Exceeded FOUND

The 'empty' files are Mozilla's doing, but the fact that the ".ja"
files (which are "Mozilla archive" files -- basically jar/zip), were
flagged bothered me. A lot.

So what should I do to have confidence in ClamAV's utility in such
situations?

The version of clamscan I am using is 0.102.1 (built from source), and
I only started using the "--alert-exceeds-max=yes" option recently.

Thanks,
Paul Kosinski

P.S. I hope everybody is staying healthy.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 3, 2020, 1:36 PM

Post #2 of 16 (1585 views)

Permalink

Citeren Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>:

> I am puzzled (and dismayed) by the following behavior of ClamAV. When I
> scan some archive files, I often get "Heuristics.Limits.Exceeded FOUND".
> This makes me wonder about ClamAV's utility in protecting our systems
> against malware.
>
> I'm not talking about archive files downloaded from questionable
> sources, or incredibly large files (like DVD ISO files). I'm talking
> about files like the latest 64-bit Linux Firefox ESR binary file (65
> MB) downloaded from "http://ftp.mozilla.org/pub/firefox/releases/".
>
> Although this file passed its GPG signature verification, and its
> SHA512 checksum test, I still like to use ClamAV for additional
> checking (since even reputable distribution sites have been known to
> have been compromised).
>
>
> However, applying clamscan to this file (which was slightly renamed by
> my download script to be more readable) results in the following output:
>
> clamscan --alert-exceeds-max=yes --max-scantime=999
> --max-scansize=4090M --max-filesize=4090M --max-files=30000
> --max-recursion=30 --pcre-match-limit=999999999
> --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2
>
> firefox-68.6.1-esr-64.tar.bz2: Heuristics.Limits.Exceeded FOUND
>
> This is in spite of the extra options I added to relax the "Limits".
>
>
> So I decided that since the Firefox tar.bz2 file passed its GPG and
> SHA512 tests, I would expand the into a temp directory and clamscan
> the result. (Which files are those used directly to run Firefox):
>
> clamscan --alert-exceeds-max=yes --max-scantime=999
> --max-scansize=4090M --max-filesize=4090M --max-files=30000
> --max-recursion=30 --pcre-match-limit=999999999
> --pcre-max-filesize=999999999 -r firefox | grep -v ' OK'
> firefox/removed-files: Empty file
> firefox/chrome.manifest: Empty file
> firefox/browser/chrome.manifest: Empty file
> firefox/browser/omni.ja: Heuristics.Limits.Exceeded FOUND
> firefox/omni.ja: Heuristics.Limits.Exceeded FOUND
>
> The 'empty' files are Mozilla's doing, but the fact that the ".ja"
> files (which are "Mozilla archive" files -- basically jar/zip), were
> flagged bothered me. A lot.
>
>
> So what should I do to have confidence in ClamAV's utility in such
> situations?

Before writing this whole rant, you have not considered checking which
of the options might have triggered this? You've reduced the
--max-scantime from the default 120 seconds to under 1 second and
still wonder why this breaks? Really?

> The version of clamscan I am using is 0.102.1 (built from source), and
> I only started using the "--alert-exceeds-max=yes" option recently.
>
> Thanks,
> Paul Kosinski
>
> P.S. I hope everybody is staying healthy.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

kdeugau at vianet

Apr 3, 2020, 2:17 PM

Post #3 of 16 (1585 views)

Permalink

Arjen de Korte via clamav-users wrote:
> Citeren Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>:

>> However, applying clamscan to this file (which was slightly renamed by
>> my download script to be more readable) results in the following output:
>>
>> clamscan --alert-exceeds-max=yes --max-scantime=999
>> --max-scansize=4090M --max-filesize=4090M --max-files=30000
>> --max-recursion=30 --pcre-match-limit=999999999
>> --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2
>>

> Before writing this whole rant, you have not considered checking which
> of the options might have triggered this? You've reduced the
> --max-scantime from the default 120 seconds to under 1 second and still
> wonder why this breaks? Really?

That option seems to be missing from the man page entirely:

$ dpkg -l clamav
ii clamav 0.102.1+dfsg-0+deb10u2 amd64 [...]
$ zgrep scantime /usr/share/man/man1/clamscan.1.gz
$

and does not specify units in the --help text:

$ clamscan --help
[...]
--max-scantime=#n Scan time longer than this
will be skipped and assumed clean
[...]

Absent any documentation, I would reasonably assume this to be in
seconds, not milliseconds.

I have no idea if you're wrong about this being the cause, but without
diving into the source, Paul's use of that option looks entirely
reasonable to me.

-kgd

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 3, 2020, 2:22 PM

Post #4 of 16 (1585 views)

Permalink

On 04.04.2020 00:17, Kris Deugau wrote:
> Arjen de Korte via clamav-users wrote:
>> Citeren Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>:
>
>>> However, applying clamscan to this file (which was slightly renamed by
>>> my download script to be more readable) results in the following
>>> output:
>>>
>>> clamscan --alert-exceeds-max=yes --max-scantime=999
>>> --max-scansize=4090M --max-filesize=4090M --max-files=30000
>>> --max-recursion=30 --pcre-match-limit=999999999
>>> --pcre-max-filesize=999999999    firefox-68.6.1-esr-64.tar.bz2
>>>
>
>> Before writing this whole rant, you have not considered checking
>> which of the options might have triggered this? You've reduced the
>> --max-scantime from the default 120 seconds to under 1 second and
>> still wonder why this breaks? Really?
>
> That option seems to be missing from the man page entirely:
>
> $ dpkg -l clamav
> ii clamav         0.102.1+dfsg-0+deb10u2          amd64 [...]
> $ zgrep scantime /usr/share/man/man1/clamscan.1.gz
> $
>
>
> and does not specify units in the --help text:
>
> $ clamscan --help
> [...]
>     --max-scantime=#n                    Scan time longer than this
> will be skipped and assumed clean
> [...]
>
> Absent any documentation, I would reasonably assume this to be in
> seconds, not milliseconds.
>
> I have no idea if you're wrong about this being the cause, but without
> diving into the source, Paul's use of that option looks entirely
> reasonable to me.
>
> -kgd

https://blog.clamav.net/2019/08/clamav-01014-security-patch-release-has.html

It is indeed a rather obscure option and missing from man pages.

Good luck,
Reio

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 3, 2020, 2:30 PM

Post #5 of 16 (1585 views)

Permalink

Citeren Kris Deugau <kdeugau@vianet.ca>:

> Arjen de Korte via clamav-users wrote:
>> Citeren Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>:
>
>>> However, applying clamscan to this file (which was slightly renamed by
>>> my download script to be more readable) results in the following output:
>>>
>>> clamscan --alert-exceeds-max=yes --max-scantime=999
>>> --max-scansize=4090M --max-filesize=4090M --max-files=30000
>>> --max-recursion=30 --pcre-match-limit=999999999
>>> --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2
>>>
>
>> Before writing this whole rant, you have not considered checking
>> which of the options might have triggered this? You've reduced the
>> --max-scantime from the default 120 seconds to under 1 second and
>> still wonder why this breaks? Really?
>
> That option seems to be missing from the man page entirely:
>
> $ dpkg -l clamav
> ii clamav 0.102.1+dfsg-0+deb10u2 amd64 [...]
> $ zgrep scantime /usr/share/man/man1/clamscan.1.gz
> $
>
>
> and does not specify units in the --help text:
>
> $ clamscan --help
> [...]
> --max-scantime=#n Scan time longer than this
> will be skipped and assumed clean
> [...]
>
> Absent any documentation, I would reasonably assume this to be in
> seconds, not milliseconds.
>
> I have no idea if you're wrong about this being the cause, but
> without diving into the source, Paul's use of that option looks
> entirely reasonable to me.

If one is overriding a default value by providing it on the
commandline, you should know what you're doing. Guessing is never a
good idea, especially if (like here) the documentation is lacking. It
was noted in the list of notable changes in 0.102.0 (see
https://blog.clamav.net/2019/10/clamav-01020-has-been-released.html)
which Paul must have read, otherwise he would not have known of the
existence of this parameter.

> -kgd
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 3, 2020, 3:37 PM

Post #6 of 16 (1585 views)

Permalink

Apologies for the confusion. We actually merged a fix to the man page, help output for this option a couple days ago:
https://github.com/Cisco-Talos/clamav-devel/commit/6e17eb5e97dbb0529dea47e15fd35b5fa79565de#diff-6426d2040de12e907d62da86807e7ca8

-Micah

?On 4/3/20, 5:18 PM, "clamav-users on behalf of Kris Deugau" <clamav-users-bounces@lists.clamav.net on behalf of kdeugau@vianet.ca> wrote:

Arjen de Korte via clamav-users wrote:
> Citeren Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>:

>> However, applying clamscan to this file (which was slightly renamed by
>> my download script to be more readable) results in the following output:
>>
>> clamscan --alert-exceeds-max=yes --max-scantime=999
>> --max-scansize=4090M --max-filesize=4090M --max-files=30000
>> --max-recursion=30 --pcre-match-limit=999999999
>> --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2
>>

> Before writing this whole rant, you have not considered checking which
> of the options might have triggered this? You've reduced the
> --max-scantime from the default 120 seconds to under 1 second and still
> wonder why this breaks? Really?

That option seems to be missing from the man page entirely:

$ dpkg -l clamav
ii clamav 0.102.1+dfsg-0+deb10u2 amd64 [...]
$ zgrep scantime /usr/share/man/man1/clamscan.1.gz
$

and does not specify units in the --help text:

$ clamscan --help
[...]
--max-scantime=#n Scan time longer than this
will be skipped and assumed clean
[...]

Absent any documentation, I would reasonably assume this to be in
seconds, not milliseconds.

I have no idea if you're wrong about this being the cause, but without
diving into the source, Paul's use of that option looks entirely
reasonable to me.

-kgd

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 3, 2020, 6:16 PM

Post #7 of 16 (1584 views)

Permalink

The --max-scantime" option apparently was the culprit. I had set it to
999 to ensure it *wouldn't* times out. I never imagined that the time
was in milliseconds, since "--help" didn't say so, and the clamscan
*command* needs on the order of 100,000 msecs even to start. (So why
specify max scan time in units of msecs then?)

The millisecs hypothesis is "proved" (small sample, though) by the fact
that when I changed the command to say "--max-scantime=999999", the
scan finished normally and reported the file clean (as I would expect,
the file having come from a well regarded source). To wit:

clamscan --alert-exceeds-max=yes --max-scantime=999999 --max-scansize=4090M --max-filesize=4090M --max-files=30000 --max-recursion=30 --pcre-match-limit=999999999 --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2

firefox-68.6.1-esr-64.tar.bz2: OK

----------- SCAN SUMMARY -----------
Known viruses: 6797620
Engine version: 0.102.1
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 622.26 MB
Data read: 62.06 MB (ratio 10.03:1)
Time: 138.749 sec (2 m 18 s)

P.S. It would be helpful if ClamAV reported exactly *which* Heuristic
Limit was exceeded (which would be quite easy, I suspect).

------------------------

On Sat, 4 Apr 2020 00:22:12 +0300
Reio Remma via clamav-users <clamav-users@lists.clamav.net> wrote:

> On 04.04.2020 00:17, Kris Deugau wrote:
> > Arjen de Korte via clamav-users wrote:
> >> Citeren Paul Kosinski via clamav-users
> >> <clamav-users@lists.clamav.net>:
> >
> >>> However, applying clamscan to this file (which was slightly
> >>> renamed by my download script to be more readable) results in the
> >>> following output:
> >>>
> >>> clamscan --alert-exceeds-max=yes --max-scantime=999
> >>> --max-scansize=4090M --max-filesize=4090M --max-files=30000
> >>> --max-recursion=30 --pcre-match-limit=999999999
> >>> --pcre-max-filesize=999999999    firefox-68.6.1-esr-64.tar.bz2
> >>>
> >
> >> Before writing this whole rant, you have not considered checking
> >> which of the options might have triggered this? You've reduced the
> >> --max-scantime from the default 120 seconds to under 1 second and
> >> still wonder why this breaks? Really?
> >
> > That option seems to be missing from the man page entirely:
> >
> > $ dpkg -l clamav
> > ii clamav         0.102.1+dfsg-0+deb10u2          amd64 [...]
> > $ zgrep scantime /usr/share/man/man1/clamscan.1.gz
> > $
> >
> >
> > and does not specify units in the --help text:
> >
> > $ clamscan --help
> > [...]
> >     --max-scantime=#n                    Scan time longer than this
> > will be skipped and assumed clean
> > [...]
> >
> > Absent any documentation, I would reasonably assume this to be in
> > seconds, not milliseconds.
> >
> > I have no idea if you're wrong about this being the cause, but
> > without diving into the source, Paul's use of that option looks
> > entirely reasonable to me.
> >
> > -kgd
>
> https://blog.clamav.net/2019/08/clamav-01014-security-patch-release-has.html
>
> It is indeed a rather obscure option and missing from man pages.
>
> Good luck,
> Reio

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 3, 2020, 10:47 PM

Post #8 of 16 (1576 views)

Permalink

"If one is overriding a default value by providing it on the command
line, you should know what you're doing. Guessing is never a good idea,
especially if (like here) the documentation is lacking."

"It was noted in the list of notable changes in 0.102.0 ... which Paul
*must* have read, otherwise he would *not* have known of the existence
of this parameter". Really?

Does issuing "clamscan --help", and reading its output of 700 words on
103 lines (according to wc), including one line about "--max-scantime",
constitute guessing? Who knew?

P.S. Up until 0.102.0, direct use of the clamscan command worked well
for files like the Firefox download. Starting with 0.102.0, clamscan
started giving Heuristic Limit errors. Since there was no indication as
to *which* Limit was hit, I read the "--help" to see what to do.

On Fri, 03 Apr 2020 23:30:57 +0200
Arjen de Korte via clamav-users <clamav-users@lists.clamav.net> wrote:

> Citeren Kris Deugau <kdeugau@vianet.ca>:
>
> > Arjen de Korte via clamav-users wrote:
> >> Citeren Paul Kosinski via clamav-users
> >> <clamav-users@lists.clamav.net>:
> >
> >>> However, applying clamscan to this file (which was slightly
> >>> renamed by my download script to be more readable) results in the
> >>> following output:
> >>>
> >>> clamscan --alert-exceeds-max=yes --max-scantime=999
> >>> --max-scansize=4090M --max-filesize=4090M --max-files=30000
> >>> --max-recursion=30 --pcre-match-limit=999999999
> >>> --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2
> >>>
> >
> >> Before writing this whole rant, you have not considered checking
> >> which of the options might have triggered this? You've reduced
> >> the --max-scantime from the default 120 seconds to under 1 second
> >> and still wonder why this breaks? Really?
> >
> > That option seems to be missing from the man page entirely:
> >
> > $ dpkg -l clamav
> > ii clamav 0.102.1+dfsg-0+deb10u2 amd64 [...]
> > $ zgrep scantime /usr/share/man/man1/clamscan.1.gz
> > $
> >
> >
> > and does not specify units in the --help text:
> >
> > $ clamscan --help
> > [...]
> > --max-scantime=#n Scan time longer than
> > this will be skipped and assumed clean
> > [...]
> >
> > Absent any documentation, I would reasonably assume this to be in
> > seconds, not milliseconds.
> >
> > I have no idea if you're wrong about this being the cause, but
> > without diving into the source, Paul's use of that option looks
> > entirely reasonable to me.
>
> If one is overriding a default value by providing it on the
> commandline, you should know what you're doing. Guessing is never a
> good idea, especially if (like here) the documentation is lacking.
> It was noted in the list of notable changes in 0.102.0 (see
> https://blog.clamav.net/2019/10/clamav-01020-has-been-released.html)
> which Paul must have read, otherwise he would not have known of the
> existence of this parameter.
>
> > -kgd

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 6, 2020, 8:23 AM

Post #9 of 16 (1570 views)

Permalink

Paul,

Are you seeing many files that take longer than 2 minutes to scan? We thought the default scan time limit was already quite high at 2 minutes.

-Micah

?On 4/4/20, 1:47 AM, "clamav-users on behalf of Paul Kosinski via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

"If one is overriding a default value by providing it on the command
line, you should know what you're doing. Guessing is never a good idea,
especially if (like here) the documentation is lacking."

"It was noted in the list of notable changes in 0.102.0 ... which Paul
*must* have read, otherwise he would *not* have known of the existence
of this parameter". Really?

Does issuing "clamscan --help", and reading its output of 700 words on
103 lines (according to wc), including one line about "--max-scantime",
constitute guessing? Who knew?

P.S. Up until 0.102.0, direct use of the clamscan command worked well
for files like the Firefox download. Starting with 0.102.0, clamscan
started giving Heuristic Limit errors. Since there was no indication as
to *which* Limit was hit, I read the "--help" to see what to do.

On Fri, 03 Apr 2020 23:30:57 +0200
Arjen de Korte via clamav-users <clamav-users@lists.clamav.net> wrote:

> Citeren Kris Deugau <kdeugau@vianet.ca>:
>
> > Arjen de Korte via clamav-users wrote:
> >> Citeren Paul Kosinski via clamav-users
> >> <clamav-users@lists.clamav.net>:
> >
> >>> However, applying clamscan to this file (which was slightly
> >>> renamed by my download script to be more readable) results in the
> >>> following output:
> >>>
> >>> clamscan --alert-exceeds-max=yes --max-scantime=999
> >>> --max-scansize=4090M --max-filesize=4090M --max-files=30000
> >>> --max-recursion=30 --pcre-match-limit=999999999
> >>> --pcre-max-filesize=999999999 firefox-68.6.1-esr-64.tar.bz2
> >>>
> >
> >> Before writing this whole rant, you have not considered checking
> >> which of the options might have triggered this? You've reduced
> >> the --max-scantime from the default 120 seconds to under 1 second
> >> and still wonder why this breaks? Really?
> >
> > That option seems to be missing from the man page entirely:
> >
> > $ dpkg -l clamav
> > ii clamav 0.102.1+dfsg-0+deb10u2 amd64 [...]
> > $ zgrep scantime /usr/share/man/man1/clamscan.1.gz
> > $
> >
> >
> > and does not specify units in the --help text:
> >
> > $ clamscan --help
> > [...]
> > --max-scantime=#n Scan time longer than
> > this will be skipped and assumed clean
> > [...]
> >
> > Absent any documentation, I would reasonably assume this to be in
> > seconds, not milliseconds.
> >
> > I have no idea if you're wrong about this being the cause, but
> > without diving into the source, Paul's use of that option looks
> > entirely reasonable to me.
>
> If one is overriding a default value by providing it on the
> commandline, you should know what you're doing. Guessing is never a
> good idea, especially if (like here) the documentation is lacking.
> It was noted in the list of notable changes in 0.102.0 (see
> https://blog.clamav.net/2019/10/clamav-01020-has-been-released.html)
> which Paul must have read, otherwise he would not have known of the
> existence of this parameter.
>
> > -kgd

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 6, 2020, 12:29 PM

Post #10 of 16 (1569 views)

Permalink

Micah,

It *does* take more than 120 secs for the clamscan command to fully
scan the 62 MB Firefox installation file (.tar.bz2). Trying the scan
with the default clamscan limits results in 62 MB "Data read" but
*zero* "Data scanned"!

Since I previously had run afoul of file size limits, I had written a
wrapper script that set all the "--max-*" limits to values that should
not cause any unnecessary failures. The problem I ran into with 0.102.x
was that the "--help" info for the clamscan command's "--max-scantime"
was incomplete.

I had set the "--max-scantime" limit to 999, assuming it was seconds.
It never occurred to me that it would be milliseconds, especially
since the clamscan command can't even load the DB in under a second.
(Milliseconds would be reasonable for clamd usage, I suppose.)

When somebody pointed out that the max scan time was really in msecs, I
updated my wrapper script and everything worked nicely, like 0.101.x.

Now, scanning the big Firefox installation file takes well over 120
secs real time, to wit (expanding the wrapper):

time clamscan
--alert-exceeds-max=yes --max-scantime=999999 --max-scansize=4090M --max-filesize=4090M --max-files=30000
--max-recursion=30 --pcre-match-limit=999999999 --pcre-max-filesize=999999999
firefox-68.6.1-esr-64.tar.bz2

firefox-68.6.1-esr-64.tar.bz2: OK

----------- SCAN SUMMARY -----------
Known viruses: 6797620
Engine version: 0.102.1
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 622.26 MB
Data read: 62.06 MB (ratio 10.03:1)
Time: 140.191 sec (2 m 20 s)

real 2m20.219s
user 2m17.212s
sys 0m2.820s

Paul

P.S. This is on an "Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz" with 32 GB RAM.

On Mon, 6 Apr 2020 15:23:42 +0000
"Micah Snyder (micasnyd)" <micasnyd@cisco.com> wrote:

> Paul,
>
> Are you seeing many files that take longer than 2 minutes to scan?
> We thought the default scan time limit was already quite high at 2
> minutes.
>
> -Micah
>
> ?On 4/4/20, 1:47 AM, "clamav-users on behalf of Paul Kosinski via
> clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of
> clamav-users@lists.clamav.net> wrote:
>
> "If one is overriding a default value by providing it on the
> command line, you should know what you're doing. Guessing is never a
> good idea, especially if (like here) the documentation is lacking."
>
> "It was noted in the list of notable changes in 0.102.0 ... which
> Paul *must* have read, otherwise he would *not* have known of the
> existence of this parameter". Really?
>
> Does issuing "clamscan --help", and reading its output of 700
> words on 103 lines (according to wc), including one line about
> "--max-scantime", constitute guessing? Who knew?
>
> P.S. Up until 0.102.0, direct use of the clamscan command worked
> well for files like the Firefox download. Starting with 0.102.0,
> clamscan started giving Heuristic Limit errors. Since there was no
> indication as to *which* Limit was hit, I read the "--help" to see
> what to do.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 6, 2020, 9:50 PM

Post #11 of 16 (1568 views)

Permalink

Much of that time is almost certainly being consumed by loading the signature database into RAM. How long does it take using clamdscan?

Sent from my iPad

-Al-

On Apr 6, 2020, at 12:29, Paul Kosinski via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> It *does* take more than 120 secs for the clamscan command to fully
> scan the 62 MB Firefox installation file (.tar.bz2). Trying the scan
> with the default clamscan limits results in 62 MB "Data read" but
> *zero* "Data scanned"!

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 7, 2020, 10:38 AM

Post #12 of 16 (1565 views)

Permalink

I didn't want to screw around with my clamdscan (clamd.conf) settings,
so I ran my optioned-up clamscan command on a smaller and much less
complicated file. It took less than 11 seconds total time. (My previous
guess on clamscan's DB load time was apparently way off.)

This suggests that the ClamAV scanning process really does take a lot
of CPU to deal with a big, complicated file like a Firefox package:

time clamscan
--alert-exceeds-max=yes --max-scantime=999999 --max-scansize=4090M --max-filesize=4090M --max-files=30000
--max-recursion=30 --pcre-match-limit=999999999 --pcre-max-filesize=999999999
audiofile.wav

audiofile.wav: OK

----------- SCAN SUMMARY -----------
Known viruses: 6804144
Engine version: 0.102.1
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 1.74 MB
Data read: 1.73 MB (ratio 1.01:1)
Time: 10.836 sec (0 m 10 s)

real 0m10.851s
user 0m10.439s
sys 0m0.412s

P.S. This is an actual audio intermediate file, not just random bytes.

On Mon, 6 Apr 2020 21:50:15 -0700
Al Varnell via clamav-users <clamav-users@lists.clamav.net> wrote:

> Much of that time is almost certainly being consumed by loading the
> signature database into RAM. How long does it take using clamdscan?
>
> Sent from my iPad
>
> -Al-
>
> On Apr 6, 2020, at 12:29, Paul Kosinski via clamav-users
> <clamav-users@lists.clamav.net> wrote:
> >
> > It *does* take more than 120 secs for the clamscan command to fully
> > scan the 62 MB Firefox installation file (.tar.bz2). Trying the scan
> > with the default clamscan limits results in 62 MB "Data read" but
> > *zero* "Data scanned"!

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 10, 2020, 12:58 PM

Post #13 of 16 (1542 views)

Permalink

One issue ClamAV currently has with scanning Zip archives is that ClamAV's self-extracting zip detection logic has a flaw wherein it detects every file within a zip as a new self-extracting zip. As a result, I believe (and I could be wrong on this), that Clam ends up extracting and scanning every file in a zip *twice*. I'm still brainstorming the best way to fix this -- but I suspect this is a large part of why zip-based file formats take much longer than expected to scan.

-Micah

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

?On 4/7/20, 1:38 PM, "clamav-users on behalf of Paul Kosinski via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

I didn't want to screw around with my clamdscan (clamd.conf) settings,
so I ran my optioned-up clamscan command on a smaller and much less
complicated file. It took less than 11 seconds total time. (My previous
guess on clamscan's DB load time was apparently way off.)

This suggests that the ClamAV scanning process really does take a lot
of CPU to deal with a big, complicated file like a Firefox package:

time clamscan
--alert-exceeds-max=yes --max-scantime=999999 --max-scansize=4090M --max-filesize=4090M --max-files=30000
--max-recursion=30 --pcre-match-limit=999999999 --pcre-max-filesize=999999999
audiofile.wav

audiofile.wav: OK

----------- SCAN SUMMARY -----------
Known viruses: 6804144
Engine version: 0.102.1
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 1.74 MB
Data read: 1.73 MB (ratio 1.01:1)
Time: 10.836 sec (0 m 10 s)

real 0m10.851s
user 0m10.439s
sys 0m0.412s

P.S. This is an actual audio intermediate file, not just random bytes.

On Mon, 6 Apr 2020 21:50:15 -0700
Al Varnell via clamav-users <clamav-users@lists.clamav.net> wrote:

> Much of that time is almost certainly being consumed by loading the
> signature database into RAM. How long does it take using clamdscan?
>
> Sent from my iPad
>
> -Al-
>
> On Apr 6, 2020, at 12:29, Paul Kosinski via clamav-users
> <clamav-users@lists.clamav.net> wrote:
> >
> > It *does* take more than 120 secs for the clamscan command to fully
> > scan the 62 MB Firefox installation file (.tar.bz2). Trying the scan
> > with the default clamscan limits results in 62 MB "Data read" but
> > *zero* "Data scanned"!

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 10, 2020, 3:55 PM

Post #14 of 16 (1541 views)

Permalink

Is this a generic problem with compressed archives (like the Firefox
".tar.bz2") or is it zip specific?

If it is zip specific, there are 2 files in the Firefox distribution
file that are zip format compressed which might explain the slowness.
(They are both named omni.ja, but have different contents).

On Fri, 10 Apr 2020 19:58:35 +0000
"Micah Snyder (micasnyd)" <micasnyd@cisco.com> wrote:

> One issue ClamAV currently has with scanning Zip archives is that
> ClamAV's self-extracting zip detection logic has a flaw wherein it
> detects every file within a zip as a new self-extracting zip. As a
> result, I believe (and I could be wrong on this), that Clam ends up
> extracting and scanning every file in a zip *twice*. I'm still
> brainstorming the best way to fix this -- but I suspect this is a
> large part of why zip-based file formats take much longer than
> expected to scan.
>
> -Micah
>
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.
>
>
>
>
> ?On 4/7/20, 1:38 PM, "clamav-users on behalf of Paul Kosinski via
> clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of
> clamav-users@lists.clamav.net> wrote:
>
> I didn't want to screw around with my clamdscan (clamd.conf)
> settings, so I ran my optioned-up clamscan command on a smaller and
> much less complicated file. It took less than 11 seconds total time.
> (My previous guess on clamscan's DB load time was apparently way off.)
>
> This suggests that the ClamAV scanning process really does take a
> lot of CPU to deal with a big, complicated file like a Firefox
> package:
> time clamscan
> --alert-exceeds-max=yes --max-scantime=999999
> --max-scansize=4090M --max-filesize=4090M --max-files=30000
> --max-recursion=30 --pcre-match-limit=999999999
> --pcre-max-filesize=999999999 audiofile.wav
> audiofile.wav: OK
>
> ----------- SCAN SUMMARY -----------
> Known viruses: 6804144
> Engine version: 0.102.1
> Scanned directories: 0
> Scanned files: 1
> Infected files: 0
> Data scanned: 1.74 MB
> Data read: 1.73 MB (ratio 1.01:1)
> Time: 10.836 sec (0 m 10 s)
>
> real 0m10.851s
> user 0m10.439s
> sys 0m0.412s
>
> P.S. This is an actual audio intermediate file, not just random
> bytes.
>
>
> On Mon, 6 Apr 2020 21:50:15 -0700
> Al Varnell via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> > Much of that time is almost certainly being consumed by loading
> > the signature database into RAM. How long does it take using
> > clamdscan?
> >
> > Sent from my iPad
> >
> > -Al-
> >
> > On Apr 6, 2020, at 12:29, Paul Kosinski via clamav-users
> > <clamav-users@lists.clamav.net> wrote:
> > >
> > > It *does* take more than 120 secs for the clamscan command to
> > > fully scan the 62 MB Firefox installation file (.tar.bz2).
> > > Trying the scan with the default clamscan limits results in
> > > 62 MB "Data read" but *zero* "Data scanned"!

>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 12, 2020, 10:39 AM

Post #15 of 16 (1539 views)

Permalink

Paul,

I investigated further and realize now that it ISN'T double-extracting files from plain zips. It is double-extracting files from zips within other raw image file formats, like TAR or image file formats. For a plain zip, It detects the file entries twice, but doesn't extract them if the parent file is a zip.

I tested this by making a simple zip with two text files in it, then tar.gz'd it. Scanning the zip.tar.gz file resulted in double-extraction of both text files.

Funny story, the omni.ja file is not a real zip. The author of the format decided to place the central directory header at the beginning of the file instead of at the end, resulting in a new zip-like file format. We're able to parse out the files from omni.ja okay because we have self-extracting zip signatures that identify the individual file entries and because the omni.ja file itself is detected as "binary data" (so the ZIPSFX-in-a-ZIP exclusion rule does not apply).

Anyhow, I now suspect that the omni.ja file in a tar.gz file will also get double-extracted. The simplest option would be to disable file-type-recognition scans for embedded files file formats in TAR files (and also GPT and other non-compressed archive file formats). I had been wanting to do this anyways after investigating a closely related issue regarding ISO/GPT file formats. This definitely gives us more reason to do so.

-Micah

?On 4/10/20, 6:55 PM, "Paul Kosinski" <clamav-users@iment.com> wrote:

Is this a generic problem with compressed archives (like the Firefox
".tar.bz2") or is it zip specific?

If it is zip specific, there are 2 files in the Firefox distribution
file that are zip format compressed which might explain the slowness.
(They are both named omni.ja, but have different contents).

On Fri, 10 Apr 2020 19:58:35 +0000
"Micah Snyder (micasnyd)" <micasnyd@cisco.com> wrote:

> One issue ClamAV currently has with scanning Zip archives is that
> ClamAV's self-extracting zip detection logic has a flaw wherein it
> detects every file within a zip as a new self-extracting zip. As a
> result, I believe (and I could be wrong on this), that Clam ends up
> extracting and scanning every file in a zip *twice*. I'm still
> brainstorming the best way to fix this -- but I suspect this is a
> large part of why zip-based file formats take much longer than
> expected to scan.
>
> -Micah
>
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.
>
>
>
>
> On 4/7/20, 1:38 PM, "clamav-users on behalf of Paul Kosinski via
> clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of
> clamav-users@lists.clamav.net> wrote:
>
> I didn't want to screw around with my clamdscan (clamd.conf)
> settings, so I ran my optioned-up clamscan command on a smaller and
> much less complicated file. It took less than 11 seconds total time.
> (My previous guess on clamscan's DB load time was apparently way off.)
>
> This suggests that the ClamAV scanning process really does take a
> lot of CPU to deal with a big, complicated file like a Firefox
> package:
> time clamscan
> --alert-exceeds-max=yes --max-scantime=999999
> --max-scansize=4090M --max-filesize=4090M --max-files=30000
> --max-recursion=30 --pcre-match-limit=999999999
> --pcre-max-filesize=999999999 audiofile.wav
> audiofile.wav: OK
>
> ----------- SCAN SUMMARY -----------
> Known viruses: 6804144
> Engine version: 0.102.1
> Scanned directories: 0
> Scanned files: 1
> Infected files: 0
> Data scanned: 1.74 MB
> Data read: 1.73 MB (ratio 1.01:1)
> Time: 10.836 sec (0 m 10 s)
>
> real 0m10.851s
> user 0m10.439s
> sys 0m0.412s
>
> P.S. This is an actual audio intermediate file, not just random
> bytes.
>
>
> On Mon, 6 Apr 2020 21:50:15 -0700
> Al Varnell via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> > Much of that time is almost certainly being consumed by loading
> > the signature database into RAM. How long does it take using
> > clamdscan?
> >
> > Sent from my iPad
> >
> > -Al-
> >
> > On Apr 6, 2020, at 12:29, Paul Kosinski via clamav-users
> > <clamav-users@lists.clamav.net> wrote:
> > >
> > > It *does* take more than 120 secs for the clamscan command to
> > > fully scan the 62 MB Firefox installation file (.tar.bz2).
> > > Trying the scan with the default clamscan limits results in
> > > 62 MB "Data read" but *zero* "Data scanned"!

>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Heuristics.Limits.Exceeded FOUND [ In reply to ]

clamav-users at lists

Apr 12, 2020, 5:24 PM

Post #16 of 16 (1539 views)

Permalink

Micah,

Yeah, now that you mention it, I remember having read somewhere about
".ja" files as being not quite zip format. On Linux, the (latest)
"file" command identifies ".ja" as "Mozilla archive" format. But
(recent) unzip commands don't seem to have any trouble (unlike ARK,
which can't find a suitable plugin).

But my favorite file manager -- emacs (!) -- expands them nicely into a
virtual file list so you can look inside the individual files.

P.S. Speaking of ISO and GPT formats, proper handling of them will need
going to 64-bit file sizes and offsets.

On Sun, 12 Apr 2020 17:39:22 +0000
"Micah Snyder (micasnyd)" <micasnyd@cisco.com> wrote:

> Paul,
>
> I investigated further and realize now that it ISN'T
> double-extracting files from plain zips. It is double-extracting
> files from zips within other raw image file formats, like TAR or
> image file formats. For a plain zip, It detects the file entries
> twice, but doesn't extract them if the parent file is a zip.
>
> I tested this by making a simple zip with two text files in it, then
> tar.gz'd it. Scanning the zip.tar.gz file resulted in
> double-extraction of both text files.
>
> Funny story, the omni.ja file is not a real zip. The author of the
> format decided to place the central directory header at the beginning
> of the file instead of at the end, resulting in a new zip-like file
> format. We're able to parse out the files from omni.ja okay because
> we have self-extracting zip signatures that identify the individual
> file entries and because the omni.ja file itself is detected as
> "binary data" (so the ZIPSFX-in-a-ZIP exclusion rule does not apply).
>
> Anyhow, I now suspect that the omni.ja file in a tar.gz file will
> also get double-extracted. The simplest option would be to disable
> file-type-recognition scans for embedded files file formats in TAR
> files (and also GPT and other non-compressed archive file formats).
> I had been wanting to do this anyways after investigating a closely
> related issue regarding ISO/GPT file formats. This definitely gives
> us more reason to do so.
>
> -Micah
>
> ?On 4/10/20, 6:55 PM, "Paul Kosinski" <clamav-users@iment.com> wrote:
>
> Is this a generic problem with compressed archives (like the
> Firefox ".tar.bz2") or is it zip specific?
>
> If it is zip specific, there are 2 files in the Firefox
> distribution file that are zip format compressed which might explain
> the slowness. (They are both named omni.ja, but have different
> contents).

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Mailing List Archive

Mailing List Archive

Attached Files: