Mailing List Archive

[clamav-users] Disk space utilization ever increasing until I kill clamonacc
Hi,

I’m wondering if anyone could assist me with an odd issue.

Recently when deploying new virtual machines which are built to use ClamAV’s on access features disk space is “consumed” over a period of many days until it eventually consumes all disk space and the server becomes non responsive (for obvious reasons).

I say consumed in quotes, because `df` reports disk space is consumed, but `du` does not. Here’s example output from a test box I spun up a few weeks back.

---------------------------------------------------------
~# du -d 1 -h /
...
...
2.6G    /
---------------------------------------------------------
~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           394M  624K  393M   1% /run
/dev/sda4        18G   14G  3.1G  82% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda1       470M   81M  365M  19% /boot
/dev/sda2       488M  3.5M  484M   1% /boot/efi
tmpfs           394M     0  394M   0% /run/user/1000
---------------------------------------------------------

On boxes that are busy doing something they'll typically "fill up" and hanging within 2-3 days. The example above is a box I've been testing with and has been up around 15 days. It's only up to 82% full but hasn't been up to much.

Until a few days ago I didn't know which piece of software was causing the issue. It felt like something was keeping a filelock open type issue.

Today I've discovered if I killed clamonacc df starts reporting the correct available disk space within a few minutes (see attached txt file). Interestingly it took about 60 seconds to make all the space available. Restarting the clamav-daemon or clamav-freshclam didn't have any effect, only clamonacc.

Before I killed clamonacc I checked if it had any open file descriptors and if so, how many.
---------------------------------------------------------
# ps -C clamonacc -o pid=
715
---------------------------------------------------------
# ls -l /proc/715/fd/ | wc -l
...
...
lrwx------ 1 root root 64 Feb 21 17:33 997 -> 'socket:[4212413]'
lrwx------ 1 root root 64 Feb 21 17:33 998 -> 'socket:[4212414]'
lrwx------ 1 root root 64 Feb 21 17:33 999 -> 'socket:[4220923]'
---------------------------------------------------------

All in all there's were 1025 references to socket "files" in /proc/715/fd/, all timestamped about 25 minutes ago.

I'm not sure if it's relevant but clamonacc is started on system start up by cron as root using the following command

---------------------------------------------------------
/usr/bin/sleep 60 && /usr/sbin/clamonacc --fdpass
---------------------------------------------------------

OS details
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Any help will be gratefully received!

Cheers

Steve
Re: [clamav-users] Disk space utilization ever increasing until I kill clamonacc [ In reply to ]
Hi there,

On Mon, 21 Feb 2022, Stephen Scotter via clamav-users wrote:

> ... when deploying new virtual machines which are built to use
> ClamAV’s on access features disk space is “consumed” over a period
> of many days until it eventually consumes all disk space and the
> server becomes non responsive (for obvious reasons).
> ...
> Until a few days ago I didn't know which piece of software was
> causing the issue. It felt like something was keeping a filelock
> open type issue.
> ...
> All in all there's were 1025 references to socket "files" in
> /proc/715/fd/, all timestamped about 25 minutes ago.

You don't say what version ov ClamAV you're using. There was a fix
recently for an issue which sounds like what you're seeing:

https://blog.clamav.net/2021/

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Disk space utilization ever increasing until I kill clamonacc [ In reply to ]
Hi Ged

> On Monday, 21 February 2022, 23:45:25 GMT, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:

> Hi there,

> On Mon, 21 Feb 2022, Stephen Scotter via clamav-users wrote:

> > ... when deploying new virtual machines which are built to use
> > ClamAV’s on access features disk space is “consumed” over a period
> > of many days until it eventually consumes all disk space and the
> > server becomes non responsive (for obvious reasons).
> > ...
> > Until a few days ago I didn't know which piece of software was
> > causing the issue. It felt like something was keeping a filelock
> > open type issue.
> > ...
> > All in all there's were 1025 references to socket "files" in
> > /proc/715/fd/, all timestamped about 25 minutes ago.

> You don't say what version ov ClamAV you're using.  There was a fix
> recently for an issue which sounds like what you're seeing:

>  https://blog.clamav.net/2021/

School boy error on my part not providing the version information. My bad!

# clamd -V
ClamAV 0.103.3/26461/Tue Feb 22 09:28:23 2022

I think you're on to something with thinking this is a bug which may have already been resolved.

https://blog.clamav.net/2021/11/clamav-01034-and-01041-patch-releases.html
---------------------------
Other fixes backported from 0.104.0:
ClamOnAcc: Fixed a socket file descriptor leak that could result in a crash when all available file descriptors are exhausted.
---------------------------

< Debian specific >
I check the test host in question and it didn't have the bullseye-updates repo enabled.

Added the following to /etc/apt/sources.list, then did a apt-get update && apt-get upgrade and now I'm on 0.103.5

---------------------------
deb http://ftp.uk.debian.org/debian bullseye-updates main
deb-src http://ftp.uk.debian.org/debian bullseye-updates main
---------------------------

---------------------------
# clamd -V
ClamAV 0.103.5/26461/Tue Feb 22 09:28:23 2022
---------------------------

< / Debian specific > 

Fingers crossed it resolve the issue. I'll do my best to feed back in a few days.

Thanks for you assistance.

Steve

> --

> 73,
> Ged.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Disk space utilization ever increasing until I kill clamonacc [ In reply to ]
I just wanted to close the loop on this for anyone else who follows in my footsteps.

Since upgrading to 0.103.5 I've not had any more problems with disk utilization seemingly increasing.

Thanks for your assistance.

Regards

Steve


On Tuesday, 22 February 2022, 11:57:46 GMT, Stephen Scotter <sscotter@yahoo.com> wrote:





Hi Ged

> On Monday, 21 February 2022, 23:45:25 GMT, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:

> Hi there,

> On Mon, 21 Feb 2022, Stephen Scotter via clamav-users wrote:

> > ... when deploying new virtual machines which are built to use
> > ClamAV’s on access features disk space is “consumed” over a period
> > of many days until it eventually consumes all disk space and the
> > server becomes non responsive (for obvious reasons).
> > ...
> > Until a few days ago I didn't know which piece of software was
> > causing the issue. It felt like something was keeping a filelock
> > open type issue.
> > ...
> > All in all there's were 1025 references to socket "files" in
> > /proc/715/fd/, all timestamped about 25 minutes ago.

> You don't say what version ov ClamAV you're using.  There was a fix
> recently for an issue which sounds like what you're seeing:

>  https://blog.clamav.net/2021/

School boy error on my part not providing the version information. My bad!

# clamd -V
ClamAV 0.103.3/26461/Tue Feb 22 09:28:23 2022

I think you're on to something with thinking this is a bug which may have already been resolved.

https://blog.clamav.net/2021/11/clamav-01034-and-01041-patch-releases.html
---------------------------
Other fixes backported from 0.104.0:
ClamOnAcc: Fixed a socket file descriptor leak that could result in a crash when all available file descriptors are exhausted.
---------------------------

< Debian specific >
I check the test host in question and it didn't have the bullseye-updates repo enabled.

Added the following to /etc/apt/sources.list, then did a apt-get update && apt-get upgrade and now I'm on 0.103.5

---------------------------
deb http://ftp.uk.debian.org/debian bullseye-updates main
deb-src http://ftp.uk.debian.org/debian bullseye-updates main
---------------------------

---------------------------
# clamd -V
ClamAV 0.103.5/26461/Tue Feb 22 09:28:23 2022
---------------------------

< / Debian specific > 

Fingers crossed it resolve the issue. I'll do my best to feed back in a few days.

Thanks for you assistance.

Steve


> --

> 73,
> Ged.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml