Mailing List Archive

[clamav-users] freshclam with lambda and S3
* I am hosting the ClamAV DB files on S3.
* I have a lambda routinely running as a cron job that downloads the latest DB files from S3 to a local dir and runs freshclam against said dir as its database directory.
* freshclam is correctly identifying the daily.cvd as out of date
* log: daily database available for update (local version: 27225, remote version: 27234)
* however, it is failing to unpack daily.cvd so it can be patched
* log: WARNING: Wed Apr 3 21:15:46 2024 -> [LibClamAV] cli_untgz: Wrote 0 instead of 512 (/tmp/clamav/db/tmp.21bd42c58a/clamav-1e208e14d7df16a662a09232b9ee56b8.tmp/daily.hsb)
* log: WARNING: Wed Apr 3 21:15:46 2024 -> [LibClamAV] CVD unpacking failed for: daily.cvd
* log: ERROR: Wed Apr 3 21:15:46 2024 -> mkdir_and_chdir_for_cdiff_tmp: Can't unpack daily.cvd into /tmp/clamav/db/tmp.21bd42c58a/clamav-1e208e14d7df16a662a09232b9ee56b8.tmp
* After this freshclam gives up trying to patch daily.cvd and downloads the full file again

Running the same image locally within Docker desktop everything works fine. The main difference I can see is that when running locally it's running as root.
I have tried to set total open permissions (chmod -R 777) on /tmp/clamav/db after obtaining the latest DB files from S3; didn't fix it.
I also notice that the clamav user is not present (when list users via getent passwd) when running as a lambda.

Lambda is running from a docker container; base image is alpine with clamav installed as "apk add --no-cache clamav-daemon clamav-libunrar".

Any ideas what I can look at next?

Thanks,
Matt


PRIVACY: This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify the sender immediately, and do not use, copy, or disclose to anyone any of the contents hereof.
Re: [clamav-users] freshclam with lambda and S3 [ In reply to ]
On Wed, 3 Apr 2024, Matthew Hibberd via clamav-users wrote:

> * I am hosting the ClamAV DB files on S3.
> * I have a lambda routinely running as a cron job that downloads the latest DB files from S3 to a local dir and runs freshclam against said dir as its database directory.
> * freshclam is correctly identifying the daily.cvd as out of date
> * log: daily database available for update (local version: 27225, remote version: 27234)
> * however, it is failing to unpack daily.cvd so it can be patched
> * log: WARNING: Wed Apr 3 21:15:46 2024 -> [LibClamAV] cli_untgz: Wrote 0 instead of 512 (/tmp/clamav/db/tmp.21bd42c58a/clamav-1e208e14d7df16a662a09232b9ee56b8.tmp/daily.hsb)
> * log: WARNING: Wed Apr 3 21:15:46 2024 -> [LibClamAV] CVD unpacking failed for: daily.cvd
> * log: ERROR: Wed Apr 3 21:15:46 2024 -> mkdir_and_chdir_for_cdiff_tmp: Can't unpack daily.cvd into /tmp/clamav/db/tmp.21bd42c58a/clamav-1e208e14d7df16a662a09232b9ee56b8.tmp
> * After this freshclam gives up trying to patch daily.cvd and downloads the full file again
>
> Running the same image locally within Docker desktop everything works fine. The main difference I can see is that when running locally it's running as root.
> I have tried to set total open permissions (chmod -R 777) on /tmp/clamav/db after obtaining the latest DB files from S3; didn't fix it.
> I also notice that the clamav user is not present (when list users via getent passwd) when running as a lambda.
>
> Lambda is running from a docker container; base image is alpine with clamav installed as "apk add --no-cache clamav-daemon clamav-libunrar".
>
> Any ideas what I can look at next?

How much RAM and disk do you have ?

(I have
CompressLocalDatabase no
in freshclam.conf and)
for me freshclam builds a daily.cld and updates that with the cdiffs.

This daily.cld is currently > 200million bytes.

Are you running a clam daemon on this VM ?
That will need > 1GB RAM, and twice that while updating, unless you
have the option to pause the service while updating.

Could it be timing out ? There are several timeouts in my freshclam.conf.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk
_______________________________________________

Manage your clamav-users mailing list subscription / unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/Cisco-Talos/clamav-documentation

https://docs.clamav.net/#mailing-lists-and-chat