Mailing List Archive

Incremental updates and server memory
I guess that this would be a long term project ...

The malware databases are updated with cdiffs, which means that the
whole database does not have to be re-downloaded with each update.

However, the running daemon has to re-read the whole database from
disk (temporarily doubling the memory requirement). Would it make
sense to be able to load the cdiff and avoid reloading from sratch ?

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk
_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat
Re: Incremental updates and server memory [ In reply to ]
On 08.09.22 18:20, Andrew C Aitchison via clamav-users wrote:
>The malware databases are updated with cdiffs, which means that the
>whole database does not have to be re-downloaded with each update.
>
>However, the running daemon has to re-read the whole database from
>disk (temporarily doubling the memory requirement). Would it make
>sense to be able to load the cdiff and avoid reloading from sratch ?

I gues the main point is that it must be implemented.

otherwise, yes, it makes perfect sense. It would lowed cpu and memory
usage and speed up reload.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Silvester Stallone: Father of the RISC concept.
_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat
Re: Incremental updates and server memory [ In reply to ]
Hi there,

On Thu, 8 Sep 2022, Andrew C Aitchison via clamav-users wrote:

> I guess that this would be a long term project ...
>
> The malware databases are updated with cdiffs, which means that the
> whole database does not have to be re-downloaded with each update.
>
> However, the running daemon has to re-read the whole database from
> disk (temporarily doubling the memory requirement).

At the expense of not being able to scan during database reloads you
can prevent the doubling of memory use by of the configuration option

ConcurrentDatabaseReload no

> Would it make sense to be able to load the cdiff and avoid reloading
> from sratch ?

This kind of thing has been discussed in the past, I think the upshot
was that it isn't feasible because of the way that the engine builds a
(highly optimized) internal representation of millions of sigs.

I've now got around the issue for Yara rules (which are modified here
much more often that the ClamAV signatures) by using a separate Yara
engine to scan for those rules. This has the added advantage that it
uses an up to date version of Yara rather than the version in ClamAV,
which is a decade or more out of date.

--

73,
Ged.
_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat