Mailing List Archive

I am not an authority here, but do recall having seen previous responses to similar suggestions and such an approach was not recommended. This has to do with the way many of the signatures are designed to look for multiple ascii or hex strings that could well occur with such strings located in different chucks that might not be covered by your overlap.

-Al-

> On Aug 30, 2023, at 11:41 PM, Ray <klammav33@busy-byte.org> wrote:
>
> Hi there,
>
> in my previous post I learned that there is in fact a hard file size limit in ClamAV of 2GByte.
>
> My company is not using ClamAV for the ususal e-Mail scanning, but for documents uploaded to a collaboration platform. Some of the documents there are larger than 2 GByte. Therefore the company is considering moving on to a commercial product for virus scanning.
>
> My question to the anti-virus experts here would be: Would it be possible to scan large files in a chunked manner with clamd? I guess it would be easy to implement on the application side to send 1000 MByte chunks to clamd for scanning (perpaps offsetting the scan window of subsequent chunks by some value to create some overlap):
>
> Chunk 1: 0 MByte to 1000 MByte
> Chunk 2: 999 MByte to 1999 MByte
> Chunk 3: 1998 MByte to 2998 MByte
> Chunk 4: 2997 MByte to 3997 MByte
> ...
>
> The 1 MByte overlap for consecutive chunks should ensure not missing malware stretching across chunk boundaries.
>
> Question 1: Would this method still detect all malware in the original, unchunked file?
> Question 2: What could be a sensible size for the overlap? (1 MByte above was just some wild guess)
>
> Best,
> Ray
>
>
> _______________________________________________
>
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
>
> https://docs.clamav.net/#mailing-lists-and-chat

_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

Hi Ray,

Sorry, chunking won't work. Clam treats files differently based on file type, and signatures are written based on looking at the whole file.

Thanks,
Andy

________________________________
From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Ray <klammav33@busy-byte.org>
Sent: Thursday, August 31, 2023 2:41 AM
To: ClamAV users ML <clamav-users@lists.clamav.net>
Subject: [clamav-users] clamd: Is chunked scanning possible/sensible for files > 2Gbyte?

Hi there,

in my previous post I learned that there is in fact a hard file size
limit in ClamAV of 2GByte.

My company is not using ClamAV for the ususal e-Mail scanning, but for
documents uploaded to a collaboration platform. Some of the documents
there are larger than 2 GByte. Therefore the company is considering
moving on to a commercial product for virus scanning.

My question to the anti-virus experts here would be: Would it be
possible to scan large files in a chunked manner with clamd? I guess it
would be easy to implement on the application side to send 1000 MByte
chunks to clamd for scanning (perpaps offsetting the scan window of
subsequent chunks by some value to create some overlap):

Chunk 1: 0 MByte to 1000 MByte
Chunk 2: 999 MByte to 1999 MByte
Chunk 3: 1998 MByte to 2998 MByte
Chunk 4: 2997 MByte to 3997 MByte
...

The 1 MByte overlap for consecutive chunks should ensure not missing
malware stretching across chunk boundaries.

Question 1: Would this method still detect all malware in the original,
unchunked file?
Question 2: What could be a sensible size for the overlap? (1 MByte
above was just some wild guess)

Best,
Ray


_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat