Mailing List Archive

[clamav-users] ClamAV® blog: ClamAV 0.103.0 release candidate
>
> https://blog.clamav.net/2020/08/clamav-01030-release-candidate.html <https://blog.clamav.net/2020/08/clamav-01030-release-candidate.html>
>
> ClamAV 0.103.0 release candidate
>
> Today we are pleased to announce the ClamAV 0.103.0 release candidate <https://www.clamav.net/downloads>!
>
> Please help us validate this release. We need your feedback so let us know what you find and join us on the clamav mailing list <https://lists.clamav.net/mailman/listinfo/clamav-users>, in #clamav on irc.freenode.net, or on our Discord <https://discord.gg/sGaxA5Q>, which is bridged with our IRC.
>
> Please submit bugs to on our Bugzilla <https://bugzilla.clamav.net/enter_bug.cgi?product=ClamAV>.
>
> ClamAV 0.103.0 includes the following improvements and changes.
>
> Major changes
>
> clamd can now reload the signature database without blocking scanning. This multi-threaded database reload improvement was made possible thanks to a community effort.
> Non-blocking database reloads are now the default behavior. Some systems that are more constrained on RAM may need to disable non-blocking reloads as it will temporarily consume 2x as much memory. For this purpose we have added a new clamd config option ConcurrentDatabaseReload which may be set to no.
>
> Special thanks to the following for making this feature a reality:
> Alberto Wu
> Alexander Sulfrian
> Arjen de Korte
> David Heidelberg
> Ged Haywood
> Julius Plenz
> Michael Orlitzky
>
> Thank you all for your patience waiting for this feature.
>
> Notable changes
>
>
> The DLP module has been enhanced with additional credit card ranges and a new engine option which allows ClamAV to alert only on credit cards (and not, for instance, gift cards) when scannning with the DLP module. This feature enhancement was made by John Schember, with input from Alexander Sulfrian.
>
> Support for Adobe Reader X PDF encryption, an overhaul of PNG scanning to detect PNG specific exploits, and a major change to GIF parsing which makes it more tolerant to problematic files and adds the ability to scan overlays, all thanks to work and patches submitted by Aldo Mazzeo.
>
> clamdtop.exe now available for Windows users. Functionality is somewhat limited when compared with clamdtop on Linux. PDCurses is required to build clamdtop.exe for ClamAV on Windows.
>
> The phishing detection module will now print "Suspicious link found!" along with the "Real URL" and "Display URL" each time phishing is detected. In a future version, we would like to print out alert-related metadata like this at the end of a scan, but for now this detail will help users understand why a given file is being flagged as phishing.
>
> Added new *experimental* CMake build tooling. CMake is not yet recommended for production builds. Our team would appreciate any assistance improving the CMake build tooling so we can one day deprecate Autotools and remove the Visual Studio solutions.
> Please see the new CMake installation instructions found in INSTALL.cmake.md for detailed instructions on how to build ClamAV with CMake.
>
> Added --ping and --wait options to the clamdscan and clamonacc client applications.
> The --ping (-p) command will attempt to ping clamd up to a specified maximum number of attempts at an optional interval. If the interval isn't specified, a default 1-second interval is used. It will exit with status code `0` when it receives a PONG from clamd or status code `21` if the timeout expires before it receives a response.
> Example:
> clamdscan -p 120 will attempt to ping clamd 120 at a 1 second interval.
> The --wait (-w) command will wait up to 30 seconds for clamd to start. This option may be used in tandem with the --ping option to customize the max # of attempts and the attempt interval. As with --ping, the scanning client may exit with status code 21 if the timeout expires before a connection is made to clamd.
> Example:
> clamdscan -p 30:2 -w <file> will attempt a scan, waiting up to 60 seconds for clamd to start and receive the scan request.
> The ping-and-wait feature is particularly useful for those wishing to start clamd and start clamonacc at startup, ensuring that clamd is ready before clamonacc starts. It is also useful for those wishing to start clamd immediately before initiating scans with clamdscan rather than having the clamd service run continuously.
> Added Excel 4.0 (XLM) macro detection and extraction support. Significantly improved VBA detection and extraction as well. Work courtesy of Jonas Zaddach.
> This support not yet added to sigtool, as the VBA extraction feature in sigtool is separate from the one used for scanning and will still need to be updated or replaced in the future.
> Improvements to the layout and legibility of temp files created during a scan. Improvements to legibility and content of the metadata JSON generated during a scan.
> To review the scan temp files and metadata JSON, run:
>
> clamscan --tempdir=<path> --leave-temps --gen-json <target>
>
>
> Viewing the scan temp files and metadata.json file provides some insight into how ClamAV analyzes a given file and can also be useful to analysts for initial triage of potentially malicious files.
>
> Other improvements
>
>
> Added ability for freshclam and clamsubmit to override default use of OpenSSL CA bundle with a custom CA bundle. On Linux/Unix platforms (excluding macOS), users may specify a custom CA bundle by setting the CURL_CA_BUNDLE environment variable. On macOS and Windows, users are expected to add CA certificates to their respective system's keychain/certificate store. Patch courtesy of Sebastian A. Siewior
> clamscan and clamdscan now print the scan start and end dates in the scan summary.
> The clamonacc on-access scanning daemon for Linux now installs to sbin instead of bin.
> Improvements to the freshclam progress bar so the width of the text does not shift around as information changes and will not spill exceed 80-characters even on very slow connections. Time is now displayed in Xm XXs (or Xh XXm) for values of 60 seconds or more. Bytes display now changes units at the proper 1024 B/KiB instead of 2048 B/KiB. Patch courtesy of Zachary Murden.
> Improve column alignment and line wrap rendering for clamdtop. Also fixed an issue on Windows where clamdtop would occasionally disconnect from clamd and fail to reconnect. Patch courtesy of Zachary Murden.
> Improvements to the AutoIT parser.
> Loosened the curl version requirements in order to build and use clamonacc. You may now build ClamAV with any version of libcurl. However clamonacc's file descriptor-passing (FD-passing) capability will only be available with libcurl 7.40 or newer. FD-passing is ordinarily the default way to perform scans with clamonacc as it is significantly faster than streaming.
> Added LZMA and BZip2 decompression routines to the bytecode signature API.
> Disabled embedded type recognition for specific archive and disk image file types. This change reduces file type misclassification and improves scan time performance by reducing duplicated file scanning.
>
> Bug fixes
>
>
> Fixed issue scanning directories on Windows with clamdscan.exe that was introduced when mitigating against symlink quarantine attacks.
> Fixed behavior of freshclam --quiet option. Patch courtesy of Reio Remma.
> Fixed behavior of freshclam's OnUpdateExecute, OnErrorExecute, and OnOutdatedExecute config options on Windows when in daemon-mode so it can handle multiple arguments. Patch courtesy of Zachary Murden.
> Fixed an error in the heuristic alert mechanism that would cause a single detection within an archive to alert once for every subsequent file scanned, potentially resulting in thousands of alerts for a single scan.
> Fixed clamd, clamav-milter, and freshclam to create PID files before dropping privileges, to avoid the possibility of an unprivileged user from changing the PID file so that a service manager will kill a different process. This change does make the services unable to clean up the PID file on exit.
> Fixed the false positive (.fp) signature feature. In prior versions, the hash in a false positive signature would be checked only against the current layer of a file being scanned. In 0.103, every file layer is hashed, and the hashes for each in the scan recursion list are checked. This ensures that .fp signatures containing a hash for any layer in the scan leading up to the alert will negate the alert.
> As an example, a hash for a zip containing the file which alerts would not prevent the detection in prior versions. Only the hash of the embedded file would work. For some file types where the outermost is always an archive, eg. docx files, this made .fp signatures next to useless. For certain file types where the scanned content was a normalized version of the original content, eg. HTML, the normalized version was never hashed and this meant that .fp signatures never worked.
> Fixed Trusted & Revoked Windows executable (PE) file signature rules (.crb) maximum functionality level (FLEVEL) which had been being treated as the minimum FLEVEL. These signatures enable ClamAV to trust executables that are digitally signed by trusted publishers, or to alert on executables signed with compromised signing-certificates. The minimum and maximum FLEVELS enable or disable signatures at load time depending on the current ClamAV version.
> Fixed a bug wherein you could not build ClamAV with --enable-libclamav-only if curl was not installed on the system.
> Various other bug fixes, improvements, and documentation improvements.
>
> New Requirements
>
>
> Autotools (automake, autoconf, m4, pkg-config, libtool) are now required in order to build from a Git clone because the files generated by these tools have been removed from the Git repository. To generate theses files before you compile ClamAV, run autogen.sh. Users building with Autotools from the release tarball should be unaffected.
> Flex and Bison are now required in order to build from a Git clone. Flex and Bison are also required to build with CMake. Users building with Autotools from the release tarball should be unaffected.
>
> Acknowledgements
>
> The ClamAV team thanks the following individuals for their code submissions:
>
> Aldo Mazzeo
> Ángel
> Antonino Cangialosi
> Clement Lecigne
> Jamie Biggar
> Jan Smutny
> John Schember
> Jonathan Sabbe
> lutianxiong
> Reio Remma
> Sebastian A. Siewior
> Zachary Murden
Re: [clamav-users] ClamAV® blog: ClamAV 0.103.0 release candidate [ In reply to ]
Congrats guys, non-blocking was a long awaited improvement on my end…

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

> On Aug 18, 2020, at 5:57 PM, Joel Esler (jesler) via clamav-users <clamav-users@lists.clamav.net> wrote:
>
>
>>
>> https://blog.clamav.net/2020/08/clamav-01030-release-candidate.html <https://blog.clamav.net/2020/08/clamav-01030-release-candidate.html>
>>
>> ClamAV 0.103.0 release candidate
>>
>> Today we are pleased to announce the ClamAV 0.103.0 release candidate <https://www.clamav.net/downloads>!
>>
>> Please help us validate this release. We need your feedback so let us know what you find and join us on the clamav mailing list <https://lists.clamav.net/mailman/listinfo/clamav-users>, in #clamav on irc.freenode.net <http://irc.freenode.net/>, or on our Discord <https://discord.gg/sGaxA5Q>, which is bridged with our IRC.
>>
>> Please submit bugs to on our Bugzilla <https://bugzilla.clamav.net/enter_bug.cgi?product=ClamAV>.
>>
>> ClamAV 0.103.0 includes the following improvements and changes.
>>
>> Major changes
>>
>> clamd can now reload the signature database without blocking scanning. This multi-threaded database reload improvement was made possible thanks to a community effort.
>> Non-blocking database reloads are now the default behavior. Some systems that are more constrained on RAM may need to disable non-blocking reloads as it will temporarily consume 2x as much memory. For this purpose we have added a new clamd config option ConcurrentDatabaseReload which may be set to no.
>>
>> Special thanks to the following for making this feature a reality:
>> Alberto Wu
>> Alexander Sulfrian
>> Arjen de Korte
>> David Heidelberg
>> Ged Haywood
>> Julius Plenz
>> Michael Orlitzky
>>
>> Thank you all for your patience waiting for this feature.
>>
>> Notable changes
>>
>>
>> The DLP module has been enhanced with additional credit card ranges and a new engine option which allows ClamAV to alert only on credit cards (and not, for instance, gift cards) when scannning with the DLP module. This feature enhancement was made by John Schember, with input from Alexander Sulfrian.
>>
>> Support for Adobe Reader X PDF encryption, an overhaul of PNG scanning to detect PNG specific exploits, and a major change to GIF parsing which makes it more tolerant to problematic files and adds the ability to scan overlays, all thanks to work and patches submitted by Aldo Mazzeo.
>>
>> clamdtop.exe now available for Windows users. Functionality is somewhat limited when compared with clamdtop on Linux. PDCurses is required to build clamdtop.exe for ClamAV on Windows.
>>
>> The phishing detection module will now print "Suspicious link found!" along with the "Real URL" and "Display URL" each time phishing is detected. In a future version, we would like to print out alert-related metadata like this at the end of a scan, but for now this detail will help users understand why a given file is being flagged as phishing.
>>
>> Added new *experimental* CMake build tooling. CMake is not yet recommended for production builds. Our team would appreciate any assistance improving the CMake build tooling so we can one day deprecate Autotools and remove the Visual Studio solutions.
>> Please see the new CMake installation instructions found in INSTALL.cmake.md for detailed instructions on how to build ClamAV with CMake.
>>
>> Added --ping and --wait options to the clamdscan and clamonacc client applications.
>> The --ping (-p) command will attempt to ping clamd up to a specified maximum number of attempts at an optional interval. If the interval isn't specified, a default 1-second interval is used. It will exit with status code `0` when it receives a PONG from clamd or status code `21` if the timeout expires before it receives a response.
>> Example:
>> clamdscan -p 120 will attempt to ping clamd 120 at a 1 second interval.
>> The --wait (-w) command will wait up to 30 seconds for clamd to start. This option may be used in tandem with the --ping option to customize the max # of attempts and the attempt interval. As with --ping, the scanning client may exit with status code 21 if the timeout expires before a connection is made to clamd.
>> Example:
>> clamdscan -p 30:2 -w <file> will attempt a scan, waiting up to 60 seconds for clamd to start and receive the scan request.
>> The ping-and-wait feature is particularly useful for those wishing to start clamd and start clamonacc at startup, ensuring that clamd is ready before clamonacc starts. It is also useful for those wishing to start clamd immediately before initiating scans with clamdscan rather than having the clamd service run continuously.
>> Added Excel 4.0 (XLM) macro detection and extraction support. Significantly improved VBA detection and extraction as well. Work courtesy of Jonas Zaddach.
>> This support not yet added to sigtool, as the VBA extraction feature in sigtool is separate from the one used for scanning and will still need to be updated or replaced in the future.
>> Improvements to the layout and legibility of temp files created during a scan. Improvements to legibility and content of the metadata JSON generated during a scan.
>> To review the scan temp files and metadata JSON, run:
>>
>> clamscan --tempdir=<path> --leave-temps --gen-json <target>
>>
>>
>> Viewing the scan temp files and metadata.json file provides some insight into how ClamAV analyzes a given file and can also be useful to analysts for initial triage of potentially malicious files.
>>
>> Other improvements
>>
>>
>> Added ability for freshclam and clamsubmit to override default use of OpenSSL CA bundle with a custom CA bundle. On Linux/Unix platforms (excluding macOS), users may specify a custom CA bundle by setting the CURL_CA_BUNDLE environment variable. On macOS and Windows, users are expected to add CA certificates to their respective system's keychain/certificate store. Patch courtesy of Sebastian A. Siewior
>> clamscan and clamdscan now print the scan start and end dates in the scan summary.
>> The clamonacc on-access scanning daemon for Linux now installs to sbin instead of bin.
>> Improvements to the freshclam progress bar so the width of the text does not shift around as information changes and will not spill exceed 80-characters even on very slow connections. Time is now displayed in Xm XXs (or Xh XXm) for values of 60 seconds or more. Bytes display now changes units at the proper 1024 B/KiB instead of 2048 B/KiB. Patch courtesy of Zachary Murden.
>> Improve column alignment and line wrap rendering for clamdtop. Also fixed an issue on Windows where clamdtop would occasionally disconnect from clamd and fail to reconnect. Patch courtesy of Zachary Murden.
>> Improvements to the AutoIT parser.
>> Loosened the curl version requirements in order to build and use clamonacc. You may now build ClamAV with any version of libcurl. However clamonacc's file descriptor-passing (FD-passing) capability will only be available with libcurl 7.40 or newer. FD-passing is ordinarily the default way to perform scans with clamonacc as it is significantly faster than streaming.
>> Added LZMA and BZip2 decompression routines to the bytecode signature API.
>> Disabled embedded type recognition for specific archive and disk image file types. This change reduces file type misclassification and improves scan time performance by reducing duplicated file scanning.
>>
>> Bug fixes
>>
>>
>> Fixed issue scanning directories on Windows with clamdscan.exe that was introduced when mitigating against symlink quarantine attacks.
>> Fixed behavior of freshclam --quiet option. Patch courtesy of Reio Remma.
>> Fixed behavior of freshclam's OnUpdateExecute, OnErrorExecute, and OnOutdatedExecute config options on Windows when in daemon-mode so it can handle multiple arguments. Patch courtesy of Zachary Murden.
>> Fixed an error in the heuristic alert mechanism that would cause a single detection within an archive to alert once for every subsequent file scanned, potentially resulting in thousands of alerts for a single scan.
>> Fixed clamd, clamav-milter, and freshclam to create PID files before dropping privileges, to avoid the possibility of an unprivileged user from changing the PID file so that a service manager will kill a different process. This change does make the services unable to clean up the PID file on exit.
>> Fixed the false positive (.fp) signature feature. In prior versions, the hash in a false positive signature would be checked only against the current layer of a file being scanned. In 0.103, every file layer is hashed, and the hashes for each in the scan recursion list are checked. This ensures that .fp signatures containing a hash for any layer in the scan leading up to the alert will negate the alert.
>> As an example, a hash for a zip containing the file which alerts would not prevent the detection in prior versions. Only the hash of the embedded file would work. For some file types where the outermost is always an archive, eg. docx files, this made .fp signatures next to useless. For certain file types where the scanned content was a normalized version of the original content, eg. HTML, the normalized version was never hashed and this meant that .fp signatures never worked.
>> Fixed Trusted & Revoked Windows executable (PE) file signature rules (.crb) maximum functionality level (FLEVEL) which had been being treated as the minimum FLEVEL. These signatures enable ClamAV to trust executables that are digitally signed by trusted publishers, or to alert on executables signed with compromised signing-certificates. The minimum and maximum FLEVELS enable or disable signatures at load time depending on the current ClamAV version.
>> Fixed a bug wherein you could not build ClamAV with --enable-libclamav-only if curl was not installed on the system.
>> Various other bug fixes, improvements, and documentation improvements.
>>
>> New Requirements
>>
>>
>> Autotools (automake, autoconf, m4, pkg-config, libtool) are now required in order to build from a Git clone because the files generated by these tools have been removed from the Git repository. To generate theses files before you compile ClamAV, run autogen.sh. Users building with Autotools from the release tarball should be unaffected.
>> Flex and Bison are now required in order to build from a Git clone. Flex and Bison are also required to build with CMake. Users building with Autotools from the release tarball should be unaffected.
>>
>> Acknowledgements
>>
>> The ClamAV team thanks the following individuals for their code submissions:
>>
>> Aldo Mazzeo
>> Ángel
>> Antonino Cangialosi
>> Clement Lecigne
>> Jamie Biggar
>> Jan Smutny
>> John Schember
>> Jonathan Sabbe
>> lutianxiong
>> Reio Remma
>> Sebastian A. Siewior
>> Zachary Murden
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
Re: [clamav-users] [External] ClamAV® blog: ClamAV 0.103.0 release candidate [ In reply to ]
Morning All,

I tested the RC on a machine with 0.102.4 on it.

I did the same configure line, make, make install and ldconfig -v and
then tried to start the clamd daemon.  After 29 minutes i gave up and
reverted to 0.102.4. 

What can I get from the system that might be helpful for debugging purposes?

Anything I should do differently for testing?

Regards,
KAM

On 8/18/2020 5:57 PM, Joel Esler (jesler) via clamav-users wrote:
>
>>
>> https://blog.clamav.net/2020/08/clamav-01030-release-candidate.html
>>
>>
>> ClamAV 0.103.0 release candidate
>>
>> Today we are pleased to announce the ClamAV 0.103.0 release candidate
>> <https://www.clamav.net/downloads>!
>>
>> Please help us validate this release. We need your feedback so let us
>> know what you find and join us on the clamav mailing list
>> <https://lists.clamav.net/mailman/listinfo/clamav-users>, in #clamav
>> on irc.freenode.net <http://irc.freenode.net>, or on our Discord
>> <https://discord.gg/sGaxA5Q>, which is bridged with our IRC.
>>
>> Please submit bugs to on our Bugzilla
>> <https://bugzilla.clamav.net/enter_bug.cgi?product=ClamAV>. 
>>
>> ClamAV 0.103.0 includes the following improvements and changes.
>>
>>
>>  Major changes
>>
>> * clamd can now reload the signature database without blocking
>> scanning. This multi-threaded database reload improvement was
>> made possible thanks to a community effort.
>> o Non-blocking database reloads are now the default behavior.
>> Some systems that are more constrained on RAM may need to
>> disable non-blocking reloads as it will temporarily consume
>> 2x as much memory. For this purpose we have added a new clamd
>> config option ConcurrentDatabaseReload which may be set to no.
>>
>>
>> Special thanks to the following for making this feature a reality:
>>
>> * Alberto Wu
>> * Alexander Sulfrian
>> * Arjen de Korte
>> * David Heidelberg
>> * Ged Haywood
>> * Julius Plenz
>> * Michael Orlitzky
>>
>>
>> Thank you all for your patience waiting for this feature. 
>>
>>
>> Notable changes
>>
>>
>> * The DLP module has been enhanced with additional credit card
>> ranges and a new engine option which allows ClamAV to alert only
>> on credit cards (and not, for instance, gift cards) when
>> scannning with the DLP module. This feature enhancement was made
>> by John Schember, with input from Alexander Sulfrian.
>>
>>
>> * Support for Adobe Reader X PDF encryption, an overhaul of PNG
>> scanning to detect PNG specific exploits, and a major change to
>> GIF parsing which makes it more tolerant to problematic files and
>> adds the ability to scan overlays, all thanks to work and patches
>> submitted by Aldo Mazzeo.
>>
>>
>> * clamdtop.exe now available for Windows users. Functionality is
>> somewhat limited when compared with clamdtop on Linux. PDCurses
>> is required to build clamdtop.exe for ClamAV on Windows.
>>
>>
>> * The phishing detection module will now print "Suspicious link
>> found!" along with the "Real URL" and "Display URL" each time
>> phishing is detected. In a future version, we would like to print
>> out alert-related metadata like this at the end of a scan, but
>> for now this detail will help users understand why a given file
>> is being flagged as phishing.
>>
>>
>> * Added new */experimental/* CMake build tooling. CMake is not yet
>> recommended for production builds. /Our team would appreciate any
>> assistance improving the CMake build tooling so we can one day
>> deprecate Autotools and remove the Visual Studio solutions/.
>> o Please see the new CMake installation instructions found in
>> INSTALL.cmake.md for detailed instructions on how to build
>> ClamAV with CMake.
>>
>>
>> * Added --ping and --wait options to the clamdscan and clamonacc
>> client applications.
>> o The --ping (-p) command will attempt to ping clamd up to a
>> specified maximum number of attempts at an optional interval.
>> If the interval isn't specified, a default 1-second interval
>> is used. It will exit with status code `0` when it receives a
>> PONG from clamd or status code `21` if the timeout expires
>> before it receives a response.
>>
>> Example:
>>
>> * clamdscan -p 120 will attempt to ping clamd 120 at a 1 second
>> interval.
>> o The --wait (-w) command will wait up to 30 seconds for clamd
>> to start. This option may be used in tandem with the --ping
>> option to customize the max # of attempts and the attempt
>> interval. As with --ping, the scanning client may exit with
>> status code 21 if the timeout expires before a connection is
>> made to clamd.
>>
>> Example:
>>
>> * clamdscan -p 30:2 -w </file/> will attempt a scan, waiting up to
>> 60 seconds for clamd to start and receive the scan request.
>> o The ping-and-wait feature is particularly useful for those
>> wishing to start clamd and start clamonacc at startup,
>> ensuring that clamd is ready before clamonacc starts. It is
>> also useful for those wishing to start clamd immediately
>> before initiating scans with clamdscan rather than having the
>> clamd service run continuously.
>>
>> * Added Excel 4.0 (XLM) macro detection and extraction support.
>> Significantly improved VBA detection and extraction as well. Work
>> courtesy of Jonas Zaddach.
>> o This support not yet added to sigtool, as the VBA extraction
>> feature in sigtool is separate from the one used for scanning
>> and will still need to be updated or replaced in the future.
>>
>> * Improvements to the layout and legibility of temp files created
>> during a scan. Improvements to legibility and content of the
>> metadata JSON generated during a scan.
>>
>> To review the scan temp files and metadata JSON, run:
>>
>>   clamscan --tempdir=</path/> --leave-temps --gen-json </target/>
>>
>>
>>   Viewing the scan temp files and metadata.json file provides some
>> insight into how ClamAV analyzes a given file and can also be useful
>> to analysts for initial triage of potentially malicious files.
>>  
>>
>>
>> Other improvements
>>
>>
>> * Added ability for freshclam and clamsubmit to override default
>> use of OpenSSL CA bundle with a custom CA bundle. On Linux/Unix
>> platforms (excluding macOS), users may specify a custom CA bundle
>> by setting the CURL_CA_BUNDLE environment variable. On macOS and
>> Windows, users are expected to add CA certificates to their
>> respective system's keychain/certificate store. Patch courtesy of
>> Sebastian A. Siewior
>>
>> * clamscan and clamdscan now print the scan start and end dates in
>> the scan summary.
>>
>> * The clamonacc on-access scanning daemon for Linux now installs to
>> sbin instead of bin.
>>
>> * Improvements to the freshclam progress bar so the width of the
>> text does not  shift around as information changes and will not
>> spill exceed 80-characters even on very slow connections. Time is
>> now displayed in Xm XXs (or Xh XXm) for values of 60 seconds or
>> more. Bytes display now changes units at the proper 1024 B/KiB
>> instead of 2048 B/KiB. Patch courtesy of Zachary Murden.
>>
>> * Improve column alignment and line wrap rendering for clamdtop.
>> Also fixed an issue on Windows where clamdtop would occasionally
>> disconnect from clamd and fail to reconnect. Patch courtesy of
>> Zachary Murden.
>>
>> * Improvements to the AutoIT parser.
>>
>> * Loosened the curl version requirements in order to build and use
>> clamonacc. You may now build ClamAV with any version of libcurl.
>> However clamonacc's file descriptor-passing (FD-passing)
>> capability will only be available with libcurl 7.40 or newer.
>> FD-passing is ordinarily the default way to perform scans with
>> clamonacc as it is significantly faster than streaming.
>>
>> * Added LZMA and BZip2 decompression routines to the bytecode
>> signature API.
>>
>> * Disabled embedded type recognition for specific archive and disk
>> image file types. This change reduces file type misclassification
>> and improves scan time performance by reducing duplicated file
>> scanning.
>>
>>  
>>
>>
>> Bug fixes
>>
>>
>> * Fixed issue scanning directories on Windows with clamdscan.exe
>> that was introduced when mitigating against symlink quarantine
>> attacks.
>>
>> * Fixed behavior of freshclam --quiet option. Patch courtesy of
>> Reio Remma.
>>
>> * Fixed behavior of freshclam's OnUpdateExecute, OnErrorExecute,
>> and OnOutdatedExecute config options on Windows when in
>> daemon-mode so it can handle multiple arguments. Patch courtesy
>> of Zachary Murden.
>>
>> * Fixed an error in the heuristic alert mechanism that would cause
>> a single detection within an archive to alert once for every
>> subsequent file scanned, potentially resulting in thousands of
>> alerts for a single scan.
>>
>> * Fixed clamd, clamav-milter, and freshclam to create PID files
>> before dropping privileges, to avoid the possibility of an
>> unprivileged user from changing the PID file so that a service
>> manager will kill a different process. This change does make the
>> services unable to clean up the PID file on exit.
>>
>> * Fixed the false positive (.fp) signature feature. In prior
>> versions, the hash in a false positive signature would be checked
>> only against the current layer of a file being scanned. In 0.103,
>> every file layer is hashed, and the hashes for each in the scan
>> recursion list are checked. This ensures that .fp signatures
>> containing a hash for any layer in the scan leading up to the
>> alert will negate the alert.
>> o As an example, a hash for a zip containing the file which
>> alerts would not prevent the detection in prior versions.
>> Only the hash of the embedded file would work. For some file
>> types where the outermost is always an archive, eg. docx
>> files, this made .fp signatures next to useless. For certain
>> file types where the scanned content was a normalized version
>> of the original content, eg. HTML, the normalized version was
>> never hashed and this meant that .fp signatures never worked.
>>
>> * Fixed Trusted & Revoked Windows executable (PE) file signature
>> rules (.crb) maximum functionality level (FLEVEL) which had been
>> being treated as the minimum FLEVEL. These signatures enable
>> ClamAV to trust executables that are digitally signed by trusted
>> publishers, or to alert on executables signed with compromised
>> signing-certificates. The minimum and maximum FLEVELS enable or
>> disable signatures at load time depending on the current ClamAV
>> version.
>>
>> * Fixed a bug wherein you could not build ClamAV with
>> --enable-libclamav-only if curl was not installed on the system.
>>
>> * Various other bug fixes, improvements, and documentation
>> improvements.
>>
>>  
>>
>>
>> New Requirements
>>
>>
>> * Autotools (automake, autoconf, m4, pkg-config, libtool) are now
>> required in order to build from a Git clone because the files
>> generated by these tools have been removed from the Git
>> repository. To generate theses files before you compile ClamAV,
>> run autogen.sh. Users building with Autotools from the release
>> tarball should be unaffected.
>>
>> * Flex and Bison are now required in order to build from a Git
>> clone. Flex and Bison are also required to build with CMake. 
>> Users building with Autotools from the release tarball should be
>> unaffected.
>>
>>  
>>
>>
>> Acknowledgements
>>
>> The ClamAV team thanks the following individuals for their code
>> submissions:
>>
>> * Aldo Mazzeo
>> * Ángel
>> * Antonino Cangialosi
>> * Clement Lecigne
>> * Jamie Biggar
>> * Jan Smutny
>> * John Schember
>> * Jonathan Sabbe
>> * lutianxiong
>> * Reio Remma
>> * Sebastian A. Siewior
>> * Zachary Murden 
>>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
--
*Kevin A. McGrail*
CEO Emeritus

Peregrine Computer Consultants Corporation
10311 Cascade Lane
Fairfax, VA 22032

http://www.pccc.com/

703-359-9700 / 800-823-8402 (Toll-Free)
703-798-0171 (wireless)
KMcGrail@PCCC.com <mailto:kmcgrail@pccc.com>

https://www.linkedin.com/in/kmcgrail
Re: [clamav-users] [External] ClamAV? blog: ClamAV 0.103.0 release candidate [ In reply to ]
Hi there,

On Wed, 19 Aug 2020, Kevin A. McGrail via clamav-users wrote:
> On 8/18/2020 5:57 PM, Joel Esler (jesler) via clamav-users wrote:
>
>>
>> https://blog.clamav.net/2020/08/clamav-01030-release-candidate.html
>>
>>
>> ClamAV 0.103.0 release candidate
>>
>> Today we are pleased to announce the ClamAV 0.103.0 release candidate
>> <https://www.clamav.net/downloads>!
>
> I tested the RC on a machine with 0.102.4 on it.

Could you tell us what operating system you're using?

> I did the same configure line, make, make install and ldconfig -v and

Please give the full, precise details.

> then tried to start the clamd daemon.? After 29 minutes i gave up and
> reverted to 0.102.4.?

Again, details please. Where did you look for feedback about the
state of the clamd process? Did you look for running processes
(e.g. 'top') or in any logs (e.g. in /var/log); did you look at things
like memory and CPU usage?

> What can I get from the system that might be helpful for debugging purposes?

There should be logs of the results of the configuration and build in
config.log in the source top level directory (where the configuration
script and makefiles live). That should be useful but it's probably a
lot to post so if you can put it somewhere where it can be downloaded
and post a link to it that will minimize the mailing list traffic.

Do you know where the binaries should have been placed? Can you
confirm that they exist there and that the timestamps look right?
Can you see anything at all in the logs that might shed any light?

> Anything I should do differently for testing?

If you mean anything different from building 0.102.4 to build 0.103,
then you might need to install the right libcurl and libcurl-dev
libraries.

You could run 'script' before you start the build process, then after
you finish the build type 'exit'. You will then have a file called
'typescript' which contains everything that happened. In addition to
being useful to convey what happened to others, the 'typescript' file
can be helpful if warnings and errors shoot off the top of the screen
while you're watching the build and you don't know how important they
might have been. :) If your typing is as bad as mine and you do a lot
of backspacing and things like that during the build session then the
'typescript' file will contain some interesting characters but apart
from that it will pretty much show you exactly what you typed and the
results of typing it. I haven't yet had a chance to build 0.103 on a
production server, but this is from my laptop just now:

laptop3:~/Downloads/src/net/mail/clamav-0.103.0-rc$ >>> ls -l typescript
-rw-r--r-- 1 ged ged 48484 Aug 19 18:48 typescript
laptop3:~/Downloads/src/net/mail/clamav-0.103.0-rc$ >>> grep -i warning typescript
/usr/bin/ld: warning: libssl.so.1.0.2, needed by /usr/lib/x86_64-linux-gnu/libcurl.so, may conflict with libssl.so.1.1
/usr/bin/ld: warning: libcrypto.so.1.0.2, needed by /usr/lib/x86_64-linux-gnu/libcurl.so, may conflict with libcrypto.so.1.1
/usr/bin/ld: warning: libssl.so.1.0.2, needed by /usr/lib/x86_64-linux-gnu/libcurl.so, may conflict with libssl.so.1.1
/usr/bin/ld: warning: libcrypto.so.1.0.2, needed by /usr/lib/x86_64-linux-gnu/libcurl.so, may conflict with libcrypto.so.1.1

I'm afraid I don't have time right now to look into this. I haven't
tried to run anything from this build yet, there's no point my trying
to do that on the laptop.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] [External] ClamAV® blog: ClamAV 0.103.0 release candidate [ In reply to ]
Hi Kevin,

Do I understand you correctly that clamd is hanging indefinitely?

A few things that may help…


1. Lets track this in a Bugzilla ticket: https://bugzilla.clamav.net/enter_bug.cgi?product=ClamAV
2. Can you provide the clamconf output?
3. Try running clamd Debug yes set in clamd.conf. The output probably won’t show much, but perhaps there will be something.
4. Kevin’s note about using `script` to record and review the log is also interesting. The libssl version mismatch between what libclamav pulled in and what libcurl was compiled with (observed by Ged) is certainly concerning, thought it shouldn’t affect clamd startup (no curl dependency there). Did you observe anything similar?

-Micah

From: clamav-users <clamav-users-bounces@lists.clamav.net> On Behalf Of Kevin A. McGrail via clamav-users
Sent: Wednesday, August 19, 2020 3:26 AM
To: clamav-users@lists.clamav.net
Cc: Kevin A. McGrail <kmcgrail@pccc.com>
Subject: Re: [clamav-users] [External] ClamAV® blog: ClamAV 0.103.0 release candidate


Morning All,

I tested the RC on a machine with 0.102.4 on it.

I did the same configure line, make, make install and ldconfig -v and then tried to start the clamd daemon. After 29 minutes i gave up and reverted to 0.102.4.

What can I get from the system that might be helpful for debugging purposes?

Anything I should do differently for testing?

Regards,
KAM
On 8/18/2020 5:57 PM, Joel Esler (jesler) via clamav-users wrote:


https://blog.clamav.net/2020/08/clamav-01030-release-candidate.html
ClamAV 0.103.0 release candidate
Today we are pleased to announce the ClamAV 0.103.0 release candidate<https://www.clamav.net/downloads>!
Please help us validate this release. We need your feedback so let us know what you find and join us on the clamav mailing list<https://lists.clamav.net/mailman/listinfo/clamav-users>, in #clamav on irc.freenode.net<http://irc.freenode.net>, or on our Discord<https://discord.gg/sGaxA5Q>, which is bridged with our IRC.
Please submit bugs to on our Bugzilla<https://bugzilla.clamav.net/enter_bug.cgi?product=ClamAV>.
ClamAV 0.103.0 includes the following improvements and changes.
Major changes

* clamd can now reload the signature database without blocking scanning. This multi-threaded database reload improvement was made possible thanks to a community effort.

* Non-blocking database reloads are now the default behavior. Some systems that are more constrained on RAM may need to disable non-blocking reloads as it will temporarily consume 2x as much memory. For this purpose we have added a new clamd config option ConcurrentDatabaseReload which may be set to no.

Special thanks to the following for making this feature a reality:


* Alberto Wu
* Alexander Sulfrian
* Arjen de Korte
* David Heidelberg
* Ged Haywood
* Julius Plenz
* Michael Orlitzky


Thank you all for your patience waiting for this feature.
Notable changes


* The DLP module has been enhanced with additional credit card ranges and a new engine option which allows ClamAV to alert only on credit cards (and not, for instance, gift cards) when scannning with the DLP module. This feature enhancement was made by John Schember, with input from Alexander Sulfrian.



* Support for Adobe Reader X PDF encryption, an overhaul of PNG scanning to detect PNG specific exploits, and a major change to GIF parsing which makes it more tolerant to problematic files and adds the ability to scan overlays, all thanks to work and patches submitted by Aldo Mazzeo.



* clamdtop.exe now available for Windows users. Functionality is somewhat limited when compared with clamdtop on Linux. PDCurses is required to build clamdtop.exe for ClamAV on Windows.



* The phishing detection module will now print "Suspicious link found!" along with the "Real URL" and "Display URL" each time phishing is detected. In a future version, we would like to print out alert-related metadata like this at the end of a scan, but for now this detail will help users understand why a given file is being flagged as phishing.



* Added new *experimental* CMake build tooling. CMake is not yet recommended for production builds. Our team would appreciate any assistance improving the CMake build tooling so we can one day deprecate Autotools and remove the Visual Studio solutions.

* Please see the new CMake installation instructions found in INSTALL.cmake.md for detailed instructions on how to build ClamAV with CMake.



* Added --ping and --wait options to the clamdscan and clamonacc client applications.

* The --ping (-p) command will attempt to ping clamd up to a specified maximum number of attempts at an optional interval. If the interval isn't specified, a default 1-second interval is used. It will exit with status code `0` when it receives a PONG from clamd or status code `21` if the timeout expires before it receives a response.
Example:


* clamdscan -p 120 will attempt to ping clamd 120 at a 1 second interval.

* The --wait (-w) command will wait up to 30 seconds for clamd to start. This option may be used in tandem with the --ping option to customize the max # of attempts and the attempt interval. As with --ping, the scanning client may exit with status code 21 if the timeout expires before a connection is made to clamd.
Example:


* clamdscan -p 30:2 -w <file> will attempt a scan, waiting up to 60 seconds for clamd to start and receive the scan request.

* The ping-and-wait feature is particularly useful for those wishing to start clamd and start clamonacc at startup, ensuring that clamd is ready before clamonacc starts. It is also useful for those wishing to start clamd immediately before initiating scans with clamdscan rather than having the clamd service run continuously.

* Added Excel 4.0 (XLM) macro detection and extraction support. Significantly improved VBA detection and extraction as well. Work courtesy of Jonas Zaddach.

* This support not yet added to sigtool, as the VBA extraction feature in sigtool is separate from the one used for scanning and will still need to be updated or replaced in the future.

* Improvements to the layout and legibility of temp files created during a scan. Improvements to legibility and content of the metadata JSON generated during a scan.
To review the scan temp files and metadata JSON, run:

clamscan --tempdir=<path> --leave-temps --gen-json <target>


Viewing the scan temp files and metadata.json file provides some insight into how ClamAV analyzes a given file and can also be useful to analysts for initial triage of potentially malicious files.

Other improvements


* Added ability for freshclam and clamsubmit to override default use of OpenSSL CA bundle with a custom CA bundle. On Linux/Unix platforms (excluding macOS), users may specify a custom CA bundle by setting the CURL_CA_BUNDLE environment variable. On macOS and Windows, users are expected to add CA certificates to their respective system's keychain/certificate store. Patch courtesy of Sebastian A. Siewior

* clamscan and clamdscan now print the scan start and end dates in the scan summary.

* The clamonacc on-access scanning daemon for Linux now installs to sbin instead of bin.

* Improvements to the freshclam progress bar so the width of the text does not shift around as information changes and will not spill exceed 80-characters even on very slow connections. Time is now displayed in Xm XXs (or Xh XXm) for values of 60 seconds or more. Bytes display now changes units at the proper 1024 B/KiB instead of 2048 B/KiB. Patch courtesy of Zachary Murden.

* Improve column alignment and line wrap rendering for clamdtop. Also fixed an issue on Windows where clamdtop would occasionally disconnect from clamd and fail to reconnect. Patch courtesy of Zachary Murden.

* Improvements to the AutoIT parser.

* Loosened the curl version requirements in order to build and use clamonacc. You may now build ClamAV with any version of libcurl. However clamonacc's file descriptor-passing (FD-passing) capability will only be available with libcurl 7.40 or newer. FD-passing is ordinarily the default way to perform scans with clamonacc as it is significantly faster than streaming.

* Added LZMA and BZip2 decompression routines to the bytecode signature API.

* Disabled embedded type recognition for specific archive and disk image file types. This change reduces file type misclassification and improves scan time performance by reducing duplicated file scanning.

Bug fixes


* Fixed issue scanning directories on Windows with clamdscan.exe that was introduced when mitigating against symlink quarantine attacks.

* Fixed behavior of freshclam --quiet option. Patch courtesy of Reio Remma.

* Fixed behavior of freshclam's OnUpdateExecute, OnErrorExecute, and OnOutdatedExecute config options on Windows when in daemon-mode so it can handle multiple arguments. Patch courtesy of Zachary Murden.

* Fixed an error in the heuristic alert mechanism that would cause a single detection within an archive to alert once for every subsequent file scanned, potentially resulting in thousands of alerts for a single scan.

* Fixed clamd, clamav-milter, and freshclam to create PID files before dropping privileges, to avoid the possibility of an unprivileged user from changing the PID file so that a service manager will kill a different process. This change does make the services unable to clean up the PID file on exit.

* Fixed the false positive (.fp) signature feature. In prior versions, the hash in a false positive signature would be checked only against the current layer of a file being scanned. In 0.103, every file layer is hashed, and the hashes for each in the scan recursion list are checked. This ensures that .fp signatures containing a hash for any layer in the scan leading up to the alert will negate the alert.

* As an example, a hash for a zip containing the file which alerts would not prevent the detection in prior versions. Only the hash of the embedded file would work. For some file types where the outermost is always an archive, eg. docx files, this made .fp signatures next to useless. For certain file types where the scanned content was a normalized version of the original content, eg. HTML, the normalized version was never hashed and this meant that .fp signatures never worked.

* Fixed Trusted & Revoked Windows executable (PE) file signature rules (.crb) maximum functionality level (FLEVEL) which had been being treated as the minimum FLEVEL. These signatures enable ClamAV to trust executables that are digitally signed by trusted publishers, or to alert on executables signed with compromised signing-certificates. The minimum and maximum FLEVELS enable or disable signatures at load time depending on the current ClamAV version.

* Fixed a bug wherein you could not build ClamAV with --enable-libclamav-only if curl was not installed on the system.

* Various other bug fixes, improvements, and documentation improvements.

New Requirements


* Autotools (automake, autoconf, m4, pkg-config, libtool) are now required in order to build from a Git clone because the files generated by these tools have been removed from the Git repository. To generate theses files before you compile ClamAV, run autogen.sh. Users building with Autotools from the release tarball should be unaffected.

* Flex and Bison are now required in order to build from a Git clone. Flex and Bison are also required to build with CMake. Users building with Autotools from the release tarball should be unaffected.

Acknowledgements
The ClamAV team thanks the following individuals for their code submissions:

* Aldo Mazzeo
* Ángel
* Antonino Cangialosi
* Clement Lecigne
* Jamie Biggar
* Jan Smutny
* John Schember
* Jonathan Sabbe
* lutianxiong
* Reio Remma
* Sebastian A. Siewior
* Zachary Murden





_______________________________________________



clamav-users mailing list

clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>

https://lists.clamav.net/mailman/listinfo/clamav-users





Help us build a comprehensive ClamAV guide:

https://github.com/vrtadmin/clamav-faq



http://www.clamav.net/contact.html#ml
--
Kevin A. McGrail
CEO Emeritus

Peregrine Computer Consultants Corporation
10311 Cascade Lane
Fairfax, VA 22032

http://www.pccc.com/

703-359-9700 / 800-823-8402 (Toll-Free)
703-798-0171 (wireless)
KMcGrail@PCCC.com<mailto:kmcgrail@pccc.com>

https://www.linkedin.com/in/kmcgrail
[cid:image001.gif@01D6760E.2C634010]
Re: [clamav-users] [External] ClamAV® blog: ClamAV 0.103.0 release candidate [ In reply to ]
Citeren "Micah Snyder (micasnyd) via clamav-users"
<clamav-users@lists.clamav.net>:

> Hi Kevin,
>
> Do I understand you correctly that clamd is hanging indefinitely?
>
> A few things that may help…
>
>
> 1. Lets track this in a Bugzilla ticket:
> https://bugzilla.clamav.net/enter_bug.cgi?product=ClamAV
> 2. Can you provide the clamconf output?
> 3. Try running clamd Debug yes set in clamd.conf. The output
> probably won’t show much, but perhaps there will be something.
> 4. Kevin’s note about using `script` to record and review the log
> is also interesting. The libssl version mismatch between what
> libclamav pulled in and what libcurl was compiled with (observed by
> Ged) is certainly concerning, thought it shouldn’t affect clamd
> startup (no curl dependency there). Did you observe anything similar?

Same behavior on openSUSE Tumbleweed with 0.103.0-rc. No build errors,
but clamd service start timeouts after 5 minutes (as configured in the
clamd.service file). One notable change is when starting 0.103.0-rc
spawns two /usr/sbin/clamd processes rather than just one for 0.102.4
(and before) while memory usage is about the same. If memory serves,
the second process is used when reloading the database, but not on
initial startup.

Unfortunately, I don't have time to further dig into this until later
this week. Buildlog and binaries are available at
https://build.opensuse.org/package/show/home:adkorte/clamav


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml