Mailing List Archive

On Sun, 22 Jan 2023 05:40:18 +0900
Tsutomu Oyamada <oyamada@promark-inc.com> wrote:

> How do I set up clamd?
> Setting MaxFileSize to "0" is unlimited, but internally files larger than 2GB in size cannot be scanned.
> In this case, do you treat the file as clean without scanning it at all?


I've complained about the 2 GiB limit now and then for several years. It's a holdover from when 32-bit numbers were all that CPUs supported, and lots of code used C's signed "int" for file size and offset.

Way back in 1996, FAT32 had this limit, but even it was extended to 4 GiB (via unsigned 32-bit numbers) when Large File Support was added.

These days, media files are often bigger than 2 GiB, as are some archive files (esp. disk images). Furthermore, almost all recent versions of standard OSes are 64-bit, and very few 32-bit CPUs are even being sold. In other words, the 2 GiB limit is quite an anachronism.

_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

Paul is sort-of correct but the 2GB limit isn't artificial as he has implied.

ClamAV code contains a lot of signed and unsigned 32bit variables that must be upgraded to 64bit variables to support larger files. Before raising the limit, a tedious audit process must be completed to ensure that all variables are upgraded in all modules. We cannot simply remove the limit and cross our fingers.

Regards,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.
________________________________
From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Paul Kosinski via clamav-users <clamav-users@lists.clamav.net>
Sent: Saturday, January 21, 2023 8:02 PM
To: clamav-users@lists.clamav.net <clamav-users@lists.clamav.net>
Cc: Paul Kosinski <clamav-users@iment.com>
Subject: Re: [clamav-users] About scanning files larger than 2 GB in size

On Sun, 22 Jan 2023 05:40:18 +0900
Tsutomu Oyamada <oyamada@promark-inc.com> wrote:

> How do I set up clamd?
> Setting MaxFileSize to "0" is unlimited, but internally files larger than 2GB in size cannot be scanned.
> In this case, do you treat the file as clean without scanning it at all?


I've complained about the 2 GiB limit now and then for several years. It's a holdover from when 32-bit numbers were all that CPUs supported, and lots of code used C's signed "int" for file size and offset.

Way back in 1996, FAT32 had this limit, but even it was extended to 4 GiB (via unsigned 32-bit numbers) when Large File Support was added.

These days, media files are often bigger than 2 GiB, as are some archive files (esp. disk images). Furthermore, almost all recent versions of standard OSes are 64-bit, and very few 32-bit CPUs are even being sold. In other words, the 2 GiB limit is quite an anachronism.

_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

I don't think I implied that the 2 GiB limit was "artificial" in the sense of trivial, or made up. I think I very clearly stated that
"It's a holdover from when 32-bit numbers were all that CPUs supported" and now "the 2 GiB limit is quite an anachronism".

Note that this question has been around for at least 7 years:
https://security.stackexchange.com/questions/107132/linux-antivirus-and-files-bigger-than-4gb

Clearly, much code review would have to be done. But Linux file I/O interfaces were successfully updated from 32-bit to 64-bit sizes and offsets some years ago, so the infrastructure is there. Also, the analogous Y2038 problem, which requires going from 32 to 64 bit as well (for time-stamps), is being seriously worked on. (And note that the Y2K problem, which was a *much* bigger issue, was indeed fixed.)

Paul

P.S. Do many current commercial AV suites for Windows have this limit?



On Thu, 26 Jan 2023 00:14:27 +0000
"Micah Snyder (micasnyd)" <micasnyd@cisco.com> wrote:

> Paul is sort-of correct but the 2GB limit isn't artificial as he has implied.
>
> ClamAV code contains a lot of signed and unsigned 32bit variables that must be upgraded to 64bit variables to support larger files. Before raising the limit, a tedious audit process must be completed to ensure that all variables are upgraded in all modules. We cannot simply remove the limit and cross our fingers.
>
> Regards,
> Micah
>
_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat

P.S. Do many current commercial AV suites for Windows have this limit?

I have no idea. Does anyone else know?


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.
________________________________
From: Paul Kosinski <clamav-users@iment.com>
Sent: Thursday, January 26, 2023 11:32 AM
To: Micah Snyder (micasnyd) <micasnyd@cisco.com>; clamav-users@lists.clamav.net <clamav-users@lists.clamav.net>
Subject: Re: [clamav-users] About scanning files larger than 2 GB in size

I don't think I implied that the 2 GiB limit was "artificial" in the sense of trivial, or made up. I think I very clearly stated that
"It's a holdover from when 32-bit numbers were all that CPUs supported" and now "the 2 GiB limit is quite an anachronism".

Note that this question has been around for at least 7 years:
https://security.stackexchange.com/questions/107132/linux-antivirus-and-files-bigger-than-4gb

Clearly, much code review would have to be done. But Linux file I/O interfaces were successfully updated from 32-bit to 64-bit sizes and offsets some years ago, so the infrastructure is there. Also, the analogous Y2038 problem, which requires going from 32 to 64 bit as well (for time-stamps), is being seriously worked on. (And note that the Y2K problem, which was a *much* bigger issue, was indeed fixed.)

Paul

P.S. Do many current commercial AV suites for Windows have this limit?



On Thu, 26 Jan 2023 00:14:27 +0000
"Micah Snyder (micasnyd)" <micasnyd@cisco.com> wrote:

> Paul is sort-of correct but the 2GB limit isn't artificial as he has implied.
>
> ClamAV code contains a lot of signed and unsigned 32bit variables that must be upgraded to 64bit variables to support larger files. Before raising the limit, a tedious audit process must be completed to ensure that all variables are upgraded in all modules. We cannot simply remove the limit and cross our fingers.
>
> Regards,
> Micah
>

> Tsutomu Oyamada asked what actually happens when a large file is
> scanned, not why the limit is there.

The default behavior is to treat the file as clean if any of the scan limits are exceeded (scan time, scan size, file size, etc).

If you want an alert if the limits are exceeded, then you can use the following options:
For ClamD, set "AlertExceedsMax yes" in the "clamd.conf" file.
For ClamScan, use the "--alert-exceeds-max" option on the command line.

This will cause clamav to report one of the following signatures when the limits are exceeded:
- Heuristics.Limits.Exceeded.MaxFileSize
- Heuristics.Limits.Exceeded.MaxScanSize
- Heuristics.Limits.Exceeded.MaxFiles
- Heuristics.Limits.Exceeded.MaxRecursion
- Heuristics.Limits.Exceeded.MaxScanTime
- Heuristics.Limits.Exceeded.EmailLineFoldcnt
- Heuristics.Limits.Exceeded.EmailHeaderBytes
- Heuristics.Limits.Exceeded.EmailHeaders
- Heuristics.Limits.Exceeded.EmailMIMEPartsPerMessage
- Heuristics.Limits.Exceeded.EmailMIMEArguments
and possibly more with the "Heuristics.Limits.Exceeded." prefix.

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

________________________________
From: Andrew C Aitchison <andrew@aitchison.me.uk>
Sent: Wednesday, January 25, 2023 10:59 PM
To: Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net>
Cc: Micah Snyder (micasnyd) <micasnyd@cisco.com>
Subject: Re: [clamav-users] About scanning files larger than 2 GB in size

On Thu, 26 Jan 2023, Micah Snyder (micasnyd) via clamav-users wrote:

> Paul is sort-of correct but the 2GB limit isn't artificial as he has implied.

Paul did not answer the original poster's question.
Tsutomu Oyamada asked what actually happens when a large file is
scanned, not why the limit is there.

> On Sun, 22 Jan 2023 05:40:18 +0900
> Tsutomu Oyamada <oyamada@promark-inc.com> wrote:
>
>> How do I set up clamd?
>> Setting MaxFileSize to "0" is unlimited, but internally files
>> larger than 2GB in size cannot be scanned. In this case, do you
>> treat the file as clean without scanning it at all?

> ClamAV code contains a lot of signed and unsigned 32bit variables
> that must be upgraded to 64bit variables to support larger files.
> Before raising the limit, a tedious audit process must be completed
> to ensure that all variables are upgraded in all modules. We cannot
> simply remove the limit and cross our fingers.

A static analyzer such as cppcheck, PVS-Studio or the ones built into
gcc and clang may be useful tools in the tedious audit.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

Thank you for the information.
I understand that files larger than 2GB will be treated as clean files without the "AlertExceedsMax yes" setting.
I want to wait for the day when I can properly scan files larger than 2GB.

T.O

On Thu, 26 Jan 2023 22:27:12 +0000
"Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net> wrote:

> > Tsutomu Oyamada asked what actually happens when a large file is
> > scanned, not why the limit is there.
>
> The default behavior is to treat the file as clean if any of the scan limits are exceeded (scan time, scan size, file size, etc).
>
> If you want an alert if the limits are exceeded, then you can use the following options:
> For ClamD, set "AlertExceedsMax yes" in the "clamd.conf" file.
> For ClamScan, use the "--alert-exceeds-max" option on the command line.
>
> This will cause clamav to report one of the following signatures when the limits are exceeded:
> - Heuristics.Limits.Exceeded.MaxFileSize
> - Heuristics.Limits.Exceeded.MaxScanSize
> - Heuristics.Limits.Exceeded.MaxFiles
> - Heuristics.Limits.Exceeded.MaxRecursion
> - Heuristics.Limits.Exceeded.MaxScanTime
> - Heuristics.Limits.Exceeded.EmailLineFoldcnt
> - Heuristics.Limits.Exceeded.EmailHeaderBytes
> - Heuristics.Limits.Exceeded.EmailHeaders
> - Heuristics.Limits.Exceeded.EmailMIMEPartsPerMessage
> - Heuristics.Limits.Exceeded.EmailMIMEArguments
> and possibly more with the "Heuristics.Limits.Exceeded." prefix.
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.
>
> ________________________________
> From: Andrew C Aitchison <andrew@aitchison.me.uk>
> Sent: Wednesday, January 25, 2023 10:59 PM
> To: Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net>
> Cc: Micah Snyder (micasnyd) <micasnyd@cisco.com>
> Subject: Re: [clamav-users] About scanning files larger than 2 GB in size
>
> On Thu, 26 Jan 2023, Micah Snyder (micasnyd) via clamav-users wrote:
>
> > Paul is sort-of correct but the 2GB limit isn't artificial as he has implied.
>
> Paul did not answer the original poster's question.
> Tsutomu Oyamada asked what actually happens when a large file is
> scanned, not why the limit is there.
>
> > On Sun, 22 Jan 2023 05:40:18 +0900
> > Tsutomu Oyamada <oyamada@promark-inc.com> wrote:
> >
> >> How do I set up clamd?
> >> Setting MaxFileSize to "0" is unlimited, but internally files
> >> larger than 2GB in size cannot be scanned. In this case, do you
> >> treat the file as clean without scanning it at all?
>
> > ClamAV code contains a lot of signed and unsigned 32bit variables
> > that must be upgraded to 64bit variables to support larger files.
> > Before raising the limit, a tedious audit process must be completed
> > to ensure that all variables are upgraded in all modules. We cannot
> > simply remove the limit and cross our fingers.
>
> A static analyzer such as cppcheck, PVS-Studio or the ones built into
> gcc and clang may be useful tools in the tedious audit.
>
> --
> Andrew C. Aitchison Kendal, UK
> andrew@aitchison.me.uk


_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat