Mailing List Archive

On 23 May 2023 21:59:22 Paul Netpresto <paul@netpresto.co.uk> wrote:

> Hello
>
> What should the behaviour of a running clamd be when it comes across a
> malformed database during a signature-reload.
>
> Clamd.conf has setting "ConcurrentDatabaseReload no"
>
> Regards Paul


Hi Paul,

Is there is a malformed database freshclam will ignore it and shouldn't update.

If it's a manually updated database, clamd will report the error in logs.

That options means....

concurrentDatabaseReload BOOL
Enable non-blocking (multi-threaded/concurrent) database reloads. This
feature will temporarily load a second scanning engine while scanning
continues using the first engine. Once loaded, the new engine takes over.
The old engine is removed as soon as all scans using the old engine have
completed. This feature requires more RAM, so this option is provided in
case users are willing to block scans during reload in exchange for lower
RAM requirements.
Default: yes


Cheers,


Steve
Sanesecurity.com
3rdparty ClamAV signatures


>
> _______________________________________________
>
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
>
> https://docs.clamav.net/#mailing-lists-and-chat


Cheers,

Steve
Twitter: @sanesecurity

Hi

I have found that 1.0.1 and 0.103.8 both behave badly if they find a
malformed db. Agreed freshclam checks out the clamav/cisco db's.

I have yet to determine what unofficial db caused the failure. They
should all have been verified before being placed in /var/lib/clamav/

Clamd ends up only partially running accepting connections creating a
/tmp/clamav.... file then giving up on the scan part of the job.

Eventually clamd has 1024 open /tmp/clamav... files and further
connections generate massive logs very quickly (like 3.5G in an hour )

It would be better if it exited when it cannot continue.

Regards Paul

On 24/05/2023 07:17, Steve Basford via clamav-users wrote:
> On 23 May 2023 21:59:22 Paul Netpresto <paul@netpresto.co.uk> wrote:
>
>> Hello
>>
>> What should the behaviour of a running clamd be when it comes across a
>> malformed database during a signature-reload.
>>
>> Clamd.conf has setting "ConcurrentDatabaseReload no"
>>
>> Regards Paul
>
>
> Hi Paul,
>
> Is there is a malformed database freshclam will ignore it and
> shouldn't update.
>
> If it's a manually updated database, clamd will report the error in logs.
>
> That options means....
>
> *concurrentDatabaseReload BOOL*
> <https://manpages.debian.org/unstable/clamav-daemon/clamd.conf.5.en.html#ConcurrentDatabaseReload>
> Enable non-blocking (multi-threaded/concurrent) database reloads.
> This feature will temporarily load a second scanning engine while
> scanning continues using the first engine. Once loaded, the new
> engine takes over. The old engine is removed as soon as all scans
> using the old engine have completed. This feature requires more
> RAM, so this option is provided in case users are willing to block
> scans during reload in exchange for lower RAM requirements.
> Default: yes
>
> Cheers,
>
> Steve
> Sanesecurity.com <http://Sanesecurity.com>
> 3rdparty ClamAV signatures
>
>>
>> _______________________________________________
>>
>> Manage your clamav-users mailing list subscription / unsubscribe:
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/Cisco-Talos/clamav-documentation
>>
>> https://docs.clamav.net/#mailing-lists-and-chat
>
>
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> _______________________________________________
>
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
>
> https://docs.clamav.net/#mailing-lists-and-chat

Hello! I believe this is how to contact the customer care.
I was wondering whether or not ClamAV has real time protection for your
system? And if so, how do I turn it on? I can't find it in my GUI settings
and it does not seem to be running either way.
Thanks!
Alex

On Wed, May 24, 2023, 12:00 AM Paul Netpresto <paul@netpresto.co.uk> wrote:

> Hello
>
> What should the behaviour of a running clamd be when it comes across a
> malformed database during a signature-reload.
>
> Clamd.conf has setting "ConcurrentDatabaseReload no"
>
> Regards Paul
>
> _______________________________________________
>
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
>
> https://docs.clamav.net/#mailing-lists-and-chat
>

On 24 May 2023 18:52:04 Paul Netpresto <paul@netpresto.co.uk> wrote:
> Hi
>
>
> I have found that 1.0.1 and 0.103.8 both behave badly if they find a
> malformed db. Agreed freshclam checks out the clamav/cisco db's.
>
> I have yet to determine what unofficial db caused the failure. They should
> all have been verified before being placed in /var/lib/clamav/
How are you downloading the 3rd party sigs...

This script checks integrity... before copying to live folder...


https://github.com/extremeshok/clamav-unofficial-sigs

I check db integrity before uploading to mirrors.

Please email me off list with some logs....

Cheers,

Steve
Twitter: @sanesecurity

On 24 May 2023 18:52:04 Paul Netpresto <paul@netpresto.co.uk> wrote:
> Hi
>
>
> I have found that 1.0.1 and 0.103.8 both behave badly if they find a
> malformed db. Agreed freshclam checks out the clamav/cisco db's.
>
> I have yet to determine what unofficial db caused the failure. They should
> all have been verified before being placed in /var/lib/clamav/

Also this fab download script....

https://github.com/rseichter/fangfrisch


Cheers,

Steve
Twitter: @sanesecurity

Hi Steve

Note it would be nice if clamd said which db it did not like ..

I reckon the start of the problem is "Database reload failed, keeping
the previous instance" when there is no previous instance.

Mon May 22 13:04:40 2023 -> Reading databases from /var/lib/clamav/
Mon May 22 13:05:01 2023 -> ERROR: reload_th: Database load failed:
Malformed da
tabase
Mon May 22 13:05:02 2023 -> Database reload completed.
Mon May 22 13:05:02 2023 -> WARNING: Database reload failed, keeping the
previou
s instance
Mon May 22 13:06:30 2023 -> ERROR: cl_engine_addref() failed
Mon May 22 13:06:30 2023 -> ERROR: Command dispatch failed
Mon May 22 13:06:30 2023 -> ERROR: INSTREAM: Can't write to temporary file.
Mon May 22 13:06:30 2023 -> ERROR: cl_engine_addref() failed
Mon May 22 13:06:30 2023 -> ERROR: Command dispatch failed
Mon May 22 13:06:30 2023 -> ERROR: INSTREAM: Can't write to temporary file.
Mon May 22 13:06:46 2023 -> ERROR: cl_engine_addref() failed
Mon May 22 13:06:46 2023 -> ERROR: Command dispatch failed
Mon May 22 13:08:31 2023 -> ERROR: cl_engine_addref() failed
Mon May 22 13:08:31 2023 -> ERROR: Command dispatch failed

    Lots more of the above snipped

Note a /tmp/clamav-*** is created for each connection containing
whatever was submitted till max files open limit is reached.


Then this starts

Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files

3.5 G later /var/ is full !!

On 24/05/2023 19:39, Steve Basford via clamav-users wrote:
>
> On 24 May 2023 18:52:04 Paul Netpresto <paul@netpresto.co.uk> wrote:
>
>> Hi
>>
>> I have found that 1.0.1 and 0.103.8 both behave badly if they find a
>> malformed db. Agreed freshclam checks out the clamav/cisco db's.
>>
>> I have yet to determine what unofficial db caused the failure. They
>> should all have been verified before being placed in /var/lib/clamav/
>>
>>
> How are you downloading the 3rd party sigs...
>
> This script checks integrity... before copying to live folder...
>
>
> https://github.com/extremeshok/clamav-unofficial-sigs
>
> I check db integrity before uploading to mirrors.
>
> Please email me off list with some logs....
>
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> _______________________________________________
>
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
>
> https://docs.clamav.net/#mailing-lists-and-chat

Could you do a ls of the clamav database folder... So I can see what
databases you are using

Does the database name appear in the logs when clamd.con

# Enable verbose logging.
# Default: no
LogVerbose yes
If you run clamscan -- database=clamav database folder test.file does it
report database errors
How much memory/disk space....

What download script... Any errors logs there to look at?

Sorry for the number of questions...
On 24 May 2023 19:54:57 Paul Netpresto <paul@netpresto.co.uk> wrote:
> Hi Steve
> Note it would be nice if clamd said which db it did not like ..
> I reckon the start of the problem is "Database reload failed, keeping the
> previous instance" when there is no previous instance.
> Mon May 22 13:04:40 2023 -> Reading databases from /var/lib/clamav/
> Mon May 22 13:05:01 2023 -> ERROR: reload_th: Database load failed:
> Malformed da
> tabase
> Mon May 22 13:05:02 2023 -> Database reload completed.
> Mon May 22 13:05:02 2023 -> WARNING: Database reload failed, keeping the
> previou
> s instance
> Mon May 22 13:06:30 2023 -> ERROR: cl_engine_addref() failed
> Mon May 22 13:06:30 2023 -> ERROR: Command dispatch failed
> Mon May 22 13:06:30 2023 -> ERROR: INSTREAM: Can't write to temporary file.
> Mon May 22 13:06:30 2023 -> ERROR: cl_engine_addref() failed
> Mon May 22 13:06:30 2023 -> ERROR: Command dispatch failed
> Mon May 22 13:06:30 2023 -> ERROR: INSTREAM: Can't write to temporary file.
> Mon May 22 13:06:46 2023 -> ERROR: cl_engine_addref() failed
> Mon May 22 13:06:46 2023 -> ERROR: Command dispatch failed
> Mon May 22 13:08:31 2023 -> ERROR: cl_engine_addref() failed
> Mon May 22 13:08:31 2023 -> ERROR: Command dispatch failedLots more of the
> above snipped
> Note a /tmp/clamav-*** is created for each connection containing whatever
> was submitted till max files open limit is reached.
>
> Then this starts
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
> Mon May 22 13:45:02 2023 -> ERROR: accept() failed: Too many open files
>
> 3.5 G later /var/ is full !!
> On 24/05/2023 19:39, Steve Basford via clamav-users wrote:
>> On 24 May 2023 18:52:04 Paul Netpresto <paul@netpresto.co.uk> wrote:
>>> Hi
>>> I have found that 1.0.1 and 0.103.8 both behave badly if they find a
>>> malformed db. Agreed freshclam checks out the clamav/cisco db's.
>>> I have yet to determine what unofficial db caused the failure. They should
>>> all have been verified before being placed in /var/lib/clamav/
>> How are you downloading the 3rd party sigs...
>>
>> This script checks integrity... before copying to live folder...
>>
>>
>> https://github.com/extremeshok/clamav-unofficial-sigs
>>
>> I check db integrity before uploading to mirrors.
>>
>> Please email me off list with some logs....
>>
>> Cheers,
>>
>> Steve
>> Twitter: @sanesecurity
>>
>> _______________________________________________ Manage your clamav-users
>> mailing list subscription / unsubscribe:
>> https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a
>> comprehensive ClamAV guide:
>> https://github.com/Cisco-Talos/clamav-documentation
>> https://docs.clamav.net/#mailing-lists-and-chat
> _______________________________________________
>
> Manage your clamav-users mailing list subscription / unsubscribe:
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/Cisco-Talos/clamav-documentation
>
> https://docs.clamav.net/#mailing-lists-and-chat

Cheers,
Steve
Twitter: @sanesecurity

On 24 May 2023 21:57:33 Steve Basford via clamav-users
<clamav-users@lists.clamav.net> wrote:
> Could you do a ls of the clamav database folder... So I can see what
> databases you are using
Sorry all should have been of list... Duh ;)

Cheers,

Steve
Twitter: @sanesecurity

Hi Steve

I am sure I can get to the bottom of how/what db was malformed.

I am more concerned on how clamd behaves when reloading db's hits an
issue and there is no previous  DB instance  available.

I am 99% sure clamd simply terminated prior to multi instance DB images
being introduced . Now it runs amok in my opinion certainly 3G of errors
in logs within an hour is not good

Thanks for the links I will check if the current scripts used for
unofficial db's can be improved.

Regards Paul


On 24/05/2023 21:57, Steve Basford via clamav-users wrote:
> when there is no previous instance.
_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat