Hi there,
On Mon, 12 Sep 2022, Eric Tykwinski via clamav-users wrote:
> I’ve been more and more moving things over to K8s from Docker ...
Could you explain that a bit more for me? My understanding was that
Kubernetes and Docker were more than a little bit complementary. [1]
Disclaimer: I've never actually used any of this new-fangled stuff [2]
and I'm wondering if we might be able to help each other here.
> just wondering if anyone is running a stateful set, IE I only want 1
> server to run freshclam, but use the same defs for all other clamd
Maybe if I give you my understanding of how things hang together for
clamd it will help.
As far as clamd is concerned, the signature database is read-only. It
resides in a single directory. For the 'official' signatures that's
at least three files - main, daily and bytecode - but if you have e.g.
third-party signatures and/or your own Yara rules in the database, it
can be many; there are 82 at the moment in our own database directory.
On startup the clamd daemon reads the whole thing and builds in-memory
a somewhat optimized representation of what it's found. The in-memory
representation is, as far as the engine is concerned, itself also then
read-only. In will consume of the order of a gigabyte, so it can take
a while to build it during which time the engine can't scan anything. [3]
The freshclam utility is normally what changes files in the database
directory, but third-party tools exist which also do that and you can
even do it manually if you wish. I used to do that all the time for
my Yara rules, but after years of pain I've given up on ClamAV's Yara
implementation and now use a separate Yara engine with separate rules.
It's much more efficient, easier to work with, and I haven't yet found
anything in Yara 4.2.2 which behaves other than exactly as documented.
> I’m assuming I can just put Example in freshclam.conf, and send a
> clamdscan —reload to the service to hit them all?
After the database has been read, clamd does not read any of the files
again until it's time to reload the whole thing. This can be because
clamd itself detects some change in one of the files in the directory
(there's an internal timeout specified in clamd's configuration file)
or because an instruction is sent to clamd via the socket on which it
is listening. You can command a 'RELOAD' using clamdscan or simply by
sending the command to the socket, e.g. using 'telnet' or 'socat' from
the command line after modifying signature files. After an update by
freshclam it sends the command if so configured. All the methods of
causing a reload have exactly the same effect. They aren't mutually
exclusive, but I don't know what might happen if you tried to use two
of them at once. :)
I haven't thought about what advantage if any might result from using
multiple clamd daemons running in containers as compared with running
more threads on a single clamd server. My gut feel is that it would
probably be more efficient just to run more threads. By now I guess
that's a pretty well tested approach, and there have been issues with
containers but I'm not well informed about them. The issues on github
are probably the best place to look for that kind of thing.
Now I'm going into the realms of conjecture. Because you might have
multiple clamd daemons running on a single host sharing resources with
some containerization method, and because the in-memory representation
of the signature database is AFAIK read-only, it stands to reason that
you might carry the extraction beyond containerization, sharing memory
between all the daemons. I'd bet that would take serious coding if it
were to be done explicitly, but you might be able to get it to work by
accident (almost) in a container environment. The problem I see is
that each time an instance of clamd reloads its database it will write
all over its in-memory representation and mess up any optimizations of
the copy-on-write variety that the OS has probably already done. But
fundamentally, as far as the scanner is concerned, the ruleset is just
a pointer to a structure.
[1]
https://containerjournal.com/topics/container-ecosystems/kubernetes-vs-docker-a-primer/ [2] I use VMs. I feel I don't know nearly enough about containers to use them safely. [4]
[3] For this reason a recent improvement has been that the engine can have
a second, entirely separate in-memory representation, which it can build
while using the first for scanning. This means that while it is building
the second it can use up to twice as much memory, but after the second is
built, the first chunk of memory will be freed and returned to the OS. For
that reason, I think you might *not* want all your clamds to reload at once.
[4] e.g.
https://containerjournal.com/features/the-state-of-k8s-software-supply-chain-attacks/ HTH
A final question: You're putting quite a bit of work into this. Do
you have a feel for the probability that clamd will find what you're
asking it to look for?
--
73,
Ged.
_______________________________________________
Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation https://docs.clamav.net/#mailing-lists-and-chat