Mailing List Archive: [clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes

[clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes

Oct 1, 2021, 9:09 AM

Post #1 of 7 (617 views)

Hi,
I have a requirement (from the business) to AV scan all docker
containers we create.
I started experimenting with tomcat:latest, which is handy because you
can follow along at home easily!
Someone else has already recommended a scan command :

clamscan <file> \
--infected \
--recursive=yes \
--alert-exceeds-max=yes \
--max-recursion=2000000 \
--max-dir-recursion=2000000 \
--max-files=2000000 \
--max-filesize=2000M \
--max-scansize=2000M \
--max-embeddedpe=2000M \
--max-htmlnormalize=2000M \
--max-htmlnotags=2000M \
--max-scriptnormalize=2000M \
--max-ziptypercg=2000M \
--max-partitions=2000000 \
--max-iconspe=2000000 \
--max-rechwp3=2000000 \
--pcre-match-limit=2000000 \
--pcre-recmatch-limit=2000000 \
--pcre-max-filesize=2000M -a

So, if you run the tomcat:latest container, apt update, apt install
clamav, freshclam and run that scan command against
/usr/local/openjdk-11/lib/src.zip you will probably get a failure :

/usr/local/openjdk-11/lib/src.zip: Heuristics.Limits.Exceeded FOUND
/usr/local/openjdk-11/lib/src.zip!(0)ZIP:jdk.zipfs/jdk/nio/zipfs/ZipInfo.java:
Heuristics.Limits.Exceeded FOUND
---------- SCAN SUMMARY -----------
Known viruses: 8570214
Engine version: 0.103.3
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 290.07 MB
Data read: 55.52 MB (ratio 5.22:1)
Time: 260.438 sec (4 m 20 s)
Start Date: 2021:10:01 13:39:47
End Date: 2021:10:01 13:44:07

However, if I extract that zip file to /src and then run clamscan on
/src then it passes without a problem :

----------- SCAN SUMMARY -----------
Known viruses: 8570214
Engine version: 0.103.3
Scanned directories: 2076
Scanned files: 18415
Infected files: 0
Data scanned: 333.04 MB
Data read: 170.92 MB (ratio 1.95:1)
Time: 320.573 sec (5 m 20 s)
Start Date: 2021:10:01 13:23:39
End Date: 2021:10:01 13:29:00

(There are indeed 18415 files in that .zip according to unzip -l)

Or even scan the single file :

clamscan ZipInfo.java --infected --recursive=yes
--alert-exceeds-max=yes --max-recursion=2000000
--max-dir-recursion=2000000 --max-files=2000000
--max-filesize=2000M --max-scansize=2000M --max-embeddedpe=2000M
--max-htmlnormalize=2000M --max-htmlnotags=2000M
--max-scriptnormalize=2000M --max-ziptypercg=2000M
--max-partitions=2000000 --max-iconspe=2000000
--max-rechwp3=2000000 --pcre-match-limit=2000000
--pcre-recmatch-limit=2000000 --pcre-max-filesize=2000M -a

----------- SCAN SUMMARY -----------
Known viruses: 8570214
Engine version: 0.103.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.01 MB
Data read: 0.01 MB (ratio 1.50:1)
Time: 68.326 sec (1 m 8 s)
Start Date: 2021:10:01 16:03:14
End Date: 2021:10:01 16:04:22

Clearly the content of src.zip (ZipInfo.java) IS scannable, when
extracted, but for some reason not scannable when it is in a zip
file... Is this a bug? Or am I specifying some options that are
causing it??

(clamscan -V
ClamAV 0.103.3/26309/Fri Oct 1 09:03:53 2021 )

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes [ In reply to ]

clamav-users at lists

Oct 1, 2021, 10:04 AM

Post #2 of 7 (617 views)

Permalink

Dear Max Allan,

Heuristics.Limits.Exceeded doesn't mean the file is infected, but it's
only a warning telling you that something went above the limits you set.
It give the warning this way because of --alert-exceeds-max=yes

ClamAV managed to go search into those files in each case as you can see
from the scan summaries. It will count the zip as a single file.

I would recommend against copy-pasting all those parameters without
having given proper thought into what you are doing.

Best regards,
Laurent

On 01.10.21 18:09, Max Allan via clamav-users wrote:
> Hi,
> I have a requirement (from the business) to AV scan all docker
> containers we create.
> I started experimenting with tomcat:latest, which is handy because you
> can follow along at home easily!
> Someone else has already recommended a scan command :
>
> clamscan <file> \
> --infected \
> --recursive=yes \
> --alert-exceeds-max=yes \
> --max-recursion=2000000 \
> --max-dir-recur
sion=2000000 \
> --max-files=2000000 \
> --max-filesize=2000M \
> --max-scansize=2000M \
> --max-embeddedpe=2000M \
> --max-htmlnormalize=2000M \
> --max-htmlnotags=2000M \
> --max-scriptnormalize=2000M \
> --max-ziptypercg=2000M \
> --max-partitions=2000000 \
> --max-iconspe=2000000 \
> --max-rechwp3=2000000 \
> --pcre-match-limit=2000000 \
> --pcre-recmatch-limit=2000000 \
> --pcre-max-filesize=2000M -a
>
> So, if you run the tomcat:latest container, apt update, apt install
> clamav, freshclam and run that scan command against
> /usr/local/openjdk-11/lib/src.zip you will probably get a failure :
>
> /usr/local/openjdk-11/lib/src.zip: Heuristics.Limits.Exceeded FOUND
> /usr/local/openjdk-11/lib/src.zip!(0)ZIP:jdk.zipfs/jdk/nio/zipfs/ZipInfo.java:
> Heuristics.Limits.Exceeded FOUND
> ---------- SCAN SUMMARY -----------
> Known viruses: 8570214
> Engine version: 0.103.3
> Scanned directories: 0
> Scanned files: 1
>
Infected files: 1
> Data scanned: 290.07 MB
> Data read: 55.52 MB (ratio 5.22:1)
> Time: 260.438 sec (4 m 20 s)
> Start Date: 2021:10:01 13:39:47
> End Date: 2021:10:01 13:44:07
>
>
> However, if I extract that zip file to /src and then run clamscan on
> /src then it passes without a problem :
>
> ----------- SCAN SUMMARY -----------
> Known viruses: 8570214
> Engine version: 0.103.3
> Scanned directories: 2076
> Scanned files: 18415
> Infected files: 0
> Data scanned: 333.04 MB
> Data read: 170.92 MB (ratio 1.95:1)
> Time: 320.573 sec (5 m 20 s)
> Start Date: 2021:10:01 13:23:39
> End Date: 2021:10:01 13:29:00
>
> (There are indeed 18415 files in that .zip according to unzip -l)
>
> Or even scan the single file :
>
> clamscan ZipInfo.java --infected --recursive=yes
> --alert-exceeds-max=yes --max-recursion=2000000
> --max-dir-recursion=2000000 --max-files=2000000
> --max-filesize=2000M --max-scansize=2000M --max-embeddedpe=2000M
> -
-max-htmlnormalize=2000M --max-htmlnotags=2000M
> --max-scriptnormalize=2000M --max-ziptypercg=2000M
> --max-partitions=2000000 --max-iconspe=2000000
> --max-rechwp3=2000000 --pcre-match-limit=2000000
> --pcre-recmatch-limit=2000000 --pcre-max-filesize=2000M -a
>
> ----------- SCAN SUMMARY -----------
> Known viruses: 8570214
> Engine version: 0.103.3
> Scanned directories: 0
> Scanned files: 1
> Infected files: 0
> Data scanned: 0.01 MB
> Data read: 0.01 MB (ratio 1.50:1)
> Time: 68.326 sec (1 m 8 s)
> Start Date: 2021:10:01 16:03:14
> End Date: 2021:10:01 16:04:22
>
>
>
> Clearly the content of src.zip (ZipInfo.java) IS scannable, when
> extracted, but for some reason not scannable when it is in a zip
> file... Is this a bug? Or am I specifying some options that are
> causing it??
>
> (clamscan -V
> ClamAV 0.103.3/26309/Fri Oct 1 09:03:53 2021 )
>
> _______________________________________________
>
> clamav-users mailing list
> clam
av-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

Re: [clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes [ In reply to ]

clamav-users at lists

Oct 4, 2021, 1:37 AM

Post #3 of 7 (613 views)

Permalink

Thanks Laurent,

The results say : "Infected files: 1". Therefore our automated systems
cannot differentiate between this file being infected and not. And if
the AV scanner has stopped scanning before everything was scanned, it
MAY be infected and I cannot allow the file in.

If it went above a limit, which limit do I need to increase to make it
scan. There is only one extra level of zip nesting and none of the
files are large. So I can presumably increase one of those limits by
"1" to count for the extra level of nesting perhaps. But I increased
the limits related to recursion massively and it still fails. All of
those limits are far, far bigger than the content (mostly 2GB and
hundreds of thousands of files). And content that it CAN scan when
extracted, so clearly there is nothing there that is beyond a limit.

I do not want files beyond a limit to remain unscanned, I want them
all to be scanned. And I can achieve that manually by extracting the
.zip. But it fails to scan when the zip still compressed. It is not
feasible to have an automated AV system uncompress every zip file it
finds outside of the scan solution and feed in all found zip files as
exclusions from the scan.

So I ask again, why does it hit a limit when in a .zip file but not
when the zip is expanded, when all the limits are clearly much higher
than anything it will encounter?

Max

On Fri, 1 Oct 2021 at 18:06, Laurent S. via clamav-users
<clamav-users@lists.clamav.net> wrote:
>
> Dear Max Allan,
>
> Heuristics.Limits.Exceeded doesn't mean the file is infected, but it's
> only a warning telling you that something went above the limits you set.
> It give the warning this way because of --alert-exceeds-max=yes
>
> ClamAV managed to go search into those files in each case as you can see
> from the scan summaries. It will count the zip as a single file.
>
> I would recommend against copy-pasting all those parameters without
> having given proper thought into what you are doing.
>
> Best regards,
> Laurent
>
> On 01.10.21 18:09, Max Allan via clamav-users wrote:
> > Hi,
> > I have a requirement (from the business) to AV scan all docker
> > containers we create.
> > I started experimenting with tomcat:latest, which is handy because you
> > can follow along at home easily!
> > Someone else has already recommended a scan command :
> >
> > clamscan <file> \
> > --infected \
> > --recursive=yes \
> > --alert-exceeds-max=yes \
> > --max-recursion=2000000 \
> > --max-dir-recur
> sion=2000000 \
> > --max-files=2000000 \
> > --max-filesize=2000M \
> > --max-scansize=2000M \
> > --max-embeddedpe=2000M \
> > --max-htmlnormalize=2000M \
> > --max-htmlnotags=2000M \
> > --max-scriptnormalize=2000M \
> > --max-ziptypercg=2000M \
> > --max-partitions=2000000 \
> > --max-iconspe=2000000 \
> > --max-rechwp3=2000000 \
> > --pcre-match-limit=2000000 \
> > --pcre-recmatch-limit=2000000 \
> > --pcre-max-filesize=2000M -a
> >
> > So, if you run the tomcat:latest container, apt update, apt install
> > clamav, freshclam and run that scan command against
> > /usr/local/openjdk-11/lib/src.zip you will probably get a failure :
> >
> > /usr/local/openjdk-11/lib/src.zip: Heuristics.Limits.Exceeded FOUND
> > /usr/local/openjdk-11/lib/src.zip!(0)ZIP:jdk.zipfs/jdk/nio/zipfs/ZipInfo.java:
> > Heuristics.Limits.Exceeded FOUND
> > ---------- SCAN SUMMARY -----------
> > Known viruses: 8570214
> > Engine version: 0.103.3
> > Scanned directories: 0
> > Scanned files: 1
> >
> Infected files: 1
> > Data scanned: 290.07 MB
> > Data read: 55.52 MB (ratio 5.22:1)
> > Time: 260.438 sec (4 m 20 s)
> > Start Date: 2021:10:01 13:39:47
> > End Date: 2021:10:01 13:44:07
> >
> >
> > However, if I extract that zip file to /src and then run clamscan on
> > /src then it passes without a problem :
> >
> > ----------- SCAN SUMMARY -----------
> > Known viruses: 8570214
> > Engine version: 0.103.3
> > Scanned directories: 2076
> > Scanned files: 18415
> > Infected files: 0
> > Data scanned: 333.04 MB
> > Data read: 170.92 MB (ratio 1.95:1)
> > Time: 320.573 sec (5 m 20 s)
> > Start Date: 2021:10:01 13:23:39
> > End Date: 2021:10:01 13:29:00
> >
> > (There are indeed 18415 files in that .zip according to unzip -l)
> >
> > Or even scan the single file :
> >
> > clamscan ZipInfo.java --infected --recursive=yes
> > --alert-exceeds-max=yes --max-recursion=2000000
> > --max-dir-recursion=2000000 --max-files=2000000
> > --max-filesize=2000M --max-scansize=2000M --max-embeddedpe=2000M
> > -
> -max-htmlnormalize=2000M --max-htmlnotags=2000M
> > --max-scriptnormalize=2000M --max-ziptypercg=2000M
> > --max-partitions=2000000 --max-iconspe=2000000
> > --max-rechwp3=2000000 --pcre-match-limit=2000000
> > --pcre-recmatch-limit=2000000 --pcre-max-filesize=2000M -a
> >
> > ----------- SCAN SUMMARY -----------
> > Known viruses: 8570214
> > Engine version: 0.103.3
> > Scanned directories: 0
> > Scanned files: 1
> > Infected files: 0
> > Data scanned: 0.01 MB
> > Data read: 0.01 MB (ratio 1.50:1)
> > Time: 68.326 sec (1 m 8 s)
> > Start Date: 2021:10:01 16:03:14
> > End Date: 2021:10:01 16:04:22
> >
> >
> >
> > Clearly the content of src.zip (ZipInfo.java) IS scannable, when
> > extracted, but for some reason not scannable when it is in a zip
> > file... Is this a bug? Or am I specifying some options that are
> > causing it??
> >
> > (clamscan -V
> > ClamAV 0.103.3/26309/Fri Oct 1 09:03:53 2021 )
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clam
> av-users@lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
> >
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes [ In reply to ]

clamav-users at lists

Oct 4, 2021, 3:21 AM

Post #4 of 7 (613 views)

Permalink

Hi Max,

Please be aware that at present, ClamAV can't scan files above 2G
because of internal variable limitations.

If your ZIP files are around 2G then this may be the limit you are
encountering.

Regards
Mark.

On 04/10/2021 09:37, Max Allan via clamav-users wrote:
> Thanks Laurent,
>
> The results say : "Infected files: 1". Therefore our automated systems
> cannot differentiate between this file being infected and not. And if
> the AV scanner has stopped scanning before everything was scanned, it
> MAY be infected and I cannot allow the file in.
>
> If it went above a limit, which limit do I need to increase to make it
> scan. There is only one extra level of zip nesting and none of the
> files are large. So I can presumably increase one of those limits by
> "1" to count for the extra level of nesting perhaps. But I increased
> the limits related to recursion massively and it still fails. All of
> those limits are far, far bigger than the content (mostly 2GB and
> hundreds of thousands of files). And content that it CAN scan when
> extracted, so clearly there is nothing there that is beyond a limit.
>
> I do not want files beyond a limit to remain unscanned, I want them
> all to be scanned. And I can achieve that manually by extracting the
> .zip. But it fails to scan when the zip still compressed. It is not
> feasible to have an automated AV system uncompress every zip file it
> finds outside of the scan solution and feed in all found zip files as
> exclusions from the scan.
>
> So I ask again, why does it hit a limit when in a .zip file but not
> when the zip is expanded, when all the limits are clearly much higher
> than anything it will encounter?
>
> Max
>
> On Fri, 1 Oct 2021 at 18:06, Laurent S. via clamav-users
> <clamav-users@lists.clamav.net> wrote:
>>
>> Dear Max Allan,
>>
>> Heuristics.Limits.Exceeded doesn't mean the file is infected, but it's
>> only a warning telling you that something went above the limits you set.
>> It give the warning this way because of --alert-exceeds-max=yes
>>
>> ClamAV managed to go search into those files in each case as you can see
>> from the scan summaries. It will count the zip as a single file.
>>
>> I would recommend against copy-pasting all those parameters without
>> having given proper thought into what you are doing.
>>
>> Best regards,
>> Laurent
>>
>> On 01.10.21 18:09, Max Allan via clamav-users wrote:
>>> Hi,
>>> I have a requirement (from the business) to AV scan all docker
>>> containers we create.
>>> I started experimenting with tomcat:latest, which is handy because you
>>> can follow along at home easily!
>>> Someone else has already recommended a scan command :
>>>
>>> clamscan <file> \
>>> --infected \
>>> --recursive=yes \
>>> --alert-exceeds-max=yes \
>>> --max-recursion=2000000 \
>>> --max-dir-recur
>> sion=2000000 \
>>> --max-files=2000000 \
>>> --max-filesize=2000M \
>>> --max-scansize=2000M \
>>> --max-embeddedpe=2000M \
>>> --max-htmlnormalize=2000M \
>>> --max-htmlnotags=2000M \
>>> --max-scriptnormalize=2000M \
>>> --max-ziptypercg=2000M \
>>> --max-partitions=2000000 \
>>> --max-iconspe=2000000 \
>>> --max-rechwp3=2000000 \
>>> --pcre-match-limit=2000000 \
>>> --pcre-recmatch-limit=2000000 \
>>> --pcre-max-filesize=2000M -a
>>>
>>> So, if you run the tomcat:latest container, apt update, apt install
>>> clamav, freshclam and run that scan command against
>>> /usr/local/openjdk-11/lib/src.zip you will probably get a failure :
>>>
>>> /usr/local/openjdk-11/lib/src.zip: Heuristics.Limits.Exceeded FOUND
>>> /usr/local/openjdk-11/lib/src.zip!(0)ZIP:jdk.zipfs/jdk/nio/zipfs/ZipInfo.java:
>>> Heuristics.Limits.Exceeded FOUND
>>> ---------- SCAN SUMMARY -----------
>>> Known viruses: 8570214
>>> Engine version: 0.103.3
>>> Scanned directories: 0
>>> Scanned files: 1
>>>
>> Infected files: 1
>>> Data scanned: 290.07 MB
>>> Data read: 55.52 MB (ratio 5.22:1)
>>> Time: 260.438 sec (4 m 20 s)
>>> Start Date: 2021:10:01 13:39:47
>>> End Date: 2021:10:01 13:44:07
>>>
>>>
>>> However, if I extract that zip file to /src and then run clamscan on
>>> /src then it passes without a problem :
>>>
>>> ----------- SCAN SUMMARY -----------
>>> Known viruses: 8570214
>>> Engine version: 0.103.3
>>> Scanned directories: 2076
>>> Scanned files: 18415
>>> Infected files: 0
>>> Data scanned: 333.04 MB
>>> Data read: 170.92 MB (ratio 1.95:1)
>>> Time: 320.573 sec (5 m 20 s)
>>> Start Date: 2021:10:01 13:23:39
>>> End Date: 2021:10:01 13:29:00
>>>
>>> (There are indeed 18415 files in that .zip according to unzip -l)
>>>
>>> Or even scan the single file :
>>>
>>> clamscan ZipInfo.java --infected --recursive=yes
>>> --alert-exceeds-max=yes --max-recursion=2000000
>>> --max-dir-recursion=2000000 --max-files=2000000
>>> --max-filesize=2000M --max-scansize=2000M --max-embeddedpe=2000M
>>> -
>> -max-htmlnormalize=2000M --max-htmlnotags=2000M
>>> --max-scriptnormalize=2000M --max-ziptypercg=2000M
>>> --max-partitions=2000000 --max-iconspe=2000000
>>> --max-rechwp3=2000000 --pcre-match-limit=2000000
>>> --pcre-recmatch-limit=2000000 --pcre-max-filesize=2000M -a
>>>
>>> ----------- SCAN SUMMARY -----------
>>> Known viruses: 8570214
>>> Engine version: 0.103.3
>>> Scanned directories: 0
>>> Scanned files: 1
>>> Infected files: 0
>>> Data scanned: 0.01 MB
>>> Data read: 0.01 MB (ratio 1.50:1)
>>> Time: 68.326 sec (1 m 8 s)
>>> Start Date: 2021:10:01 16:03:14
>>> End Date: 2021:10:01 16:04:22
>>>
>>>
>>>
>>> Clearly the content of src.zip (ZipInfo.java) IS scannable, when
>>> extracted, but for some reason not scannable when it is in a zip
>>> file... Is this a bug? Or am I specifying some options that are
>>> causing it??
>>>
>>> (clamscan -V
>>> ClamAV 0.103.3/26309/Fri Oct 1 09:03:53 2021 )
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clam
>> av-users@lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes [ In reply to ]

clamav-users at lists

Oct 4, 2021, 7:09 AM

Post #5 of 7 (612 views)

Permalink

Hi there,

On Mon, 4 Oct 2021, Max Allan via clamav-users wrote:

> ... if the AV scanner has stopped scanning before everything was
> scanned, it MAY be infected and I cannot allow the file in.

As a result of collecting statistics on mail servers for some years,
my estimate of probability that ClamAV, out of the box, will detect
some random threat is on the order of a few percent. Even if you used
all the available scanning products on the planet the probability will
not be much better than 80%. See some of my other posts to this list
for tables of numbers.

More succinctly if you rely just on scanning for protection, then by
about the fifth threat that you scan the game will probably be over.

So I hope you're not saying that a clean scan result will mean that
you allow the file in without further ado.

> So I ask again, why does it hit a limit when in a .zip file but not
> when the zip is expanded, when all the limits are clearly much higher
> than anything it will encounter?

I don't know, but I haven't carefully examined your command line - I
feel that it's insanely long, and quite possibly asking for trouble.
Have you read the warnings in the documentation about increased limits?

It isn't always obvious what ClamAV does under the hood when it scans
things like archives, but to help with any investigation you can for
example increase the debug log message verbosity and tell ClamAV to
keep any temporary files instead of deleting them after the scan.
Sometimes I've rebuilt ClamAV simply to add a debug message to find
out what's going on. Having the source code helps a lot of course. :)

Obviously I haven't seen any of the files that you're scanning. If
you can let me have one which you think will exhibit this behaviour
I'll be happy to give it a once-over when I get a minute.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes [ In reply to ]

clamav-users at lists

Oct 4, 2021, 7:31 AM

Post #6 of 7 (612 views)

Permalink

What are the compressed and uncompressed sizes of the problem file ?

On Fri, 1 Oct 2021, Max Allan via clamav-users wrote:

> Hi,
> I have a requirement (from the business) to AV scan all docker
> containers we create.
> I started experimenting with tomcat:latest, which is handy because you
> can follow along at home easily!
> Someone else has already recommended a scan command :
>
> clamscan <file> \
> --infected \
> --recursive=yes \
> --alert-exceeds-max=yes \
> --max-recursion=2000000 \
> --max-dir-recursion=2000000 \
> --max-files=2000000 \
> --max-filesize=2000M \
> --max-scansize=2000M \
> --max-embeddedpe=2000M \
> --max-htmlnormalize=2000M \
> --max-htmlnotags=2000M \
> --max-scriptnormalize=2000M \
> --max-ziptypercg=2000M \
> --max-partitions=2000000 \
> --max-iconspe=2000000 \
> --max-rechwp3=2000000 \
> --pcre-match-limit=2000000 \
> --pcre-recmatch-limit=2000000 \
> --pcre-max-filesize=2000M -a
>
> So, if you run the tomcat:latest container, apt update, apt install
> clamav, freshclam and run that scan command against
> /usr/local/openjdk-11/lib/src.zip you will probably get a failure :
>
> /usr/local/openjdk-11/lib/src.zip: Heuristics.Limits.Exceeded FOUND
> /usr/local/openjdk-11/lib/src.zip!(0)ZIP:jdk.zipfs/jdk/nio/zipfs/ZipInfo.java:
> Heuristics.Limits.Exceeded FOUND
> ---------- SCAN SUMMARY -----------
> Known viruses: 8570214
> Engine version: 0.103.3
> Scanned directories: 0
> Scanned files: 1
> Infected files: 1
> Data scanned: 290.07 MB
> Data read: 55.52 MB (ratio 5.22:1)
> Time: 260.438 sec (4 m 20 s)
> Start Date: 2021:10:01 13:39:47
> End Date: 2021:10:01 13:44:07
>
>
> However, if I extract that zip file to /src and then run clamscan on
> /src then it passes without a problem :
>
> ----------- SCAN SUMMARY -----------
> Known viruses: 8570214
> Engine version: 0.103.3
> Scanned directories: 2076
> Scanned files: 18415
> Infected files: 0
> Data scanned: 333.04 MB
> Data read: 170.92 MB (ratio 1.95:1)
> Time: 320.573 sec (5 m 20 s)
> Start Date: 2021:10:01 13:23:39
> End Date: 2021:10:01 13:29:00
>
> (There are indeed 18415 files in that .zip according to unzip -l)
>
> Or even scan the single file :
>
> clamscan ZipInfo.java --infected --recursive=yes
> --alert-exceeds-max=yes --max-recursion=2000000
> --max-dir-recursion=2000000 --max-files=2000000
> --max-filesize=2000M --max-scansize=2000M --max-embeddedpe=2000M
> --max-htmlnormalize=2000M --max-htmlnotags=2000M
> --max-scriptnormalize=2000M --max-ziptypercg=2000M
> --max-partitions=2000000 --max-iconspe=2000000
> --max-rechwp3=2000000 --pcre-match-limit=2000000
> --pcre-recmatch-limit=2000000 --pcre-max-filesize=2000M -a
>
> ----------- SCAN SUMMARY -----------
> Known viruses: 8570214
> Engine version: 0.103.3
> Scanned directories: 0
> Scanned files: 1
> Infected files: 0
> Data scanned: 0.01 MB
> Data read: 0.01 MB (ratio 1.50:1)
> Time: 68.326 sec (1 m 8 s)
> Start Date: 2021:10:01 16:03:14
> End Date: 2021:10:01 16:04:22
>
>
>
> Clearly the content of src.zip (ZipInfo.java) IS scannable, when
> extracted, but for some reason not scannable when it is in a zip
> file... Is this a bug? Or am I specifying some options that are
> causing it??
>
> (clamscan -V
> ClamAV 0.103.3/26309/Fri Oct 1 09:03:53 2021 )
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scanning a zip file fails, extract it, scan with the same options and it passes [ In reply to ]

clamav-users at lists

Oct 5, 2021, 5:38 AM

Post #7 of 7 (611 views)

Permalink

Hi Andrew,

The .zip file (252M compressed / 58M zipped) and offending file
(2683/9069bytes)
All well less than the maximums on the CLI (2GB) and by no means the
largest file.

Rubbing salt in the wound, the return code when it exceeds heuristics
is "1" which the man page says :

> Return Codes
> 0 : No virus found.
> 1 : Virus(es) found.
> 2 : Some error(s) occured.

Surely it should be a "2" not a "1".

Anyway, after running with --debug and -v I found an error about
exceeding scan time. Added --max-scantime=120000000
And now it does "the right thing". (scan doesn't report a virus found)

Thanks,
Max

On Mon, 4 Oct 2021 at 15:31, Andrew C Aitchison <clamav@aitchison.me.uk> wrote:
>
> What are the compressed and uncompressed sizes of the problem file ?
>

> >
> > /usr/local/openjdk-11/lib/src.zip: Heuristics.Limits.Exceeded FOUND
> > /usr/local/openjdk-11/lib/src.zip!(0)ZIP:jdk.zipfs/jdk/nio/zipfs/ZipInfo.java:
> > Heuristics.Limits.Exceeded FOUND
> > ---------- SCAN SUMMARY -----------
> > Known viruses: 8570214
> > Engine version: 0.103.3
> > Scanned directories: 0
> > Scanned files: 1
> > Infected files: 1
> > Data scanned: 290.07 MB

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Mailing List Archive

Mailing List Archive

Attached Files: