Mailing List Archive: [clamav-users] strace - select(13, [12], NULL, NULL, NULL) = -1 EBADF (Bad file descriptor)

[clamav-users] strace - select(13, [12], NULL, NULL, NULL) = -1 EBADF (Bad file descriptor) <0.000017>

Nov 7, 2019, 2:57 AM

Post #1 of 5 (304 views)

Good morning

We have an issues affecting multiple VM's running clamd (with On Access
scanning enabled). We see High sy CPU (98%) and very high load (20-50
1min), both 2 and 4 core VM's.

When we strace the process we that clamd cannot read its config files:

root@xxxxx ]# strace -T -tt -f -p 62279
strace: Process 62279 attached with 3 threads
[pid 62281] 10:08:50.970009 restart_syscall(<... resuming interrupted
poll ...> <unfinished ...>
[pid 62279] 10:08:51.491142 restart_syscall(<... resuming interrupted
poll ...> <unfinished ...>
[pid 62282] 10:08:52.047155 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000017>
[pid 62282] 10:08:52.047290 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047342 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047388 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000016>
[pid 62282] 10:08:52.047437 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000016>
[pid 62282] 10:08:52.047484 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047527 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047571 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047620 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047666 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000014>
[pid 62282] 10:08:52.047708 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000014>
[pid 62282] 10:08:52.047752 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047794 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000014>
[pid 62282] 10:08:52.047839 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047892 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047938 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.047980 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.048025 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000014>
[pid 62282] 10:08:52.048067 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.048111 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000014>
[pid 62282] 10:08:52.048153 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.048197 read(12, 0x7f2c66e0ad90, 4096) = -1 EBADF
(Bad file descriptor) <0.000015>
[pid 62282] 10:08:52.048240 select(13, [12], NULL, NULL, NULL) = -1
EBADF (Bad file descriptor) <0.000015>

I tried to launch clamdtop to see if there was a queue to view, which
failed:
[root@xxxxx ]# clamdtop
__ ____
_____/ /___ _____ ___ ____/ / /_____ ____
/ ___/ / __ `/ __ `__ \/ __ / __/ __ \/ __ \
/ /__/ / /_/ / / / / / / /_/ / /_/ /_/ / /_/ /
\___/_/\__,_/_/ /_/ /_/\__,_/\__/\____/ .___/
/_/
Connecting to: /var/run/clamd.scan/clamd.sock
Abnormal program termination: Failed to reconnect to clamd after
connection was lost in reconnect at line 762

this seem to result in a kill -11 on the pid:

Nov 7 10:10:12 prd-atv-int-opt03.int.tac.local systemd:
clamd@scan.service: main process exited, code=killed, status=11/SEGV
Nov 7 10:10:12 prd-atv-int-opt03.int.tac.local systemd: Unit
clamd@scan.service entered failed state.
Nov 7 10:10:12 prd-atv-int-opt03.int.tac.local systemd:
clamd@scan.service failed.
Nov 7 10:10:12 prd-atv-int-opt03.int.tac.local systemd:
clamd@scan.service holdoff time over, scheduling restart.
Nov 7 10:10:12 prd-atv-int-opt03.int.tac.local systemd: Stopped
Generic clamav scanner daemon.
Nov 7 10:10:12 prd-atv-int-opt03.int.tac.local systemd: Starting
Generic clamav scanner daemon...

Which brought clamd back to life and the system load returned to
normal. no idea is this is a OS bug, a ClamAV bug or some kind of user
error, any help here will be appreciated.

--
Thank you,
Tim

[Winner of the 2018 Consumer Credit Awards]

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] strace - select(13, [12], NULL, NULL, NULL) = -1 EBADF (Bad file descriptor) <0.000017> [ In reply to ]

clamav-users at lists

Nov 7, 2019, 4:52 AM

Post #2 of 5 (304 views)

Permalink

> Which brought clamd back to life and the system load returned to
> normal. no idea is this is a OS bug, a ClamAV bug or some kind of user
> error, any help here will be appreciated.

What version of ClamAV? What OS? What customization / edits to config
files have you made?

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] strace - select(13, [12], NULL, NULL, NULL) = -1 EBADF (Bad file descriptor) <0.000017> [ In reply to ]

clamav-users at lists

Nov 7, 2019, 7:55 AM

Post #3 of 5 (304 views)

Permalink

Hi there,

On Thu, 7 Nov 2019, J.R. via clamav-users wrote:

>> Which brought clamd back to life and the system load returned to
>> normal. no idea is this is a OS bug, a ClamAV bug or some kind of user
>> error, any help here will be appreciated.
>
> What version of ClamAV? What OS? What customization / edits to config
> files have you made?

And what are you scanning???

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] strace - select(13, [12], NULL, NULL, NULL) = -1 EBADF (Bad file descriptor) <0.000017> [ In reply to ]

tim.stubbs at telrock

Nov 8, 2019, 4:19 AM

Post #4 of 5 (300 views)

Permalink

thanks for the response;
we are experiancing this issues on a fresh install VM, a Java application VM & a Jump server with gnome. a mix of 2 and 4 coure VM's with 2,4 & 6GB RAM

[root@xxxxxxx<file:///root@x>]# uname -a
Linux xxxxxxxxxxxxxx 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@xxxxx<file:///root@x> ]# cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

# Config
LogFile /var/log/clamav/clamav.log
LogFileUnlock yes
LogFileMaxSize 10M
LogTime yes
LogSyslog no
LogRotate no
ExtendedDetectionInfo yes
PidFile /var/run/clamd.scan/clamd.pid
DatabaseDirectory /var/lib/clamav
LocalSocket /var/run/clamd.scan/clamd.sock
LocalSocketGroup virusgroup
LocalSocketMode 666
FixStaleSocket yes
MaxThreads 10
ReadTimeout 180
SendBufTimeout 200
MaxQueue 100
ExcludePath ^/proc/
ExcludePath ^/sys/
ExcludePath ^/root/
ExcludePath ^/var\/lib\/openvas\/plugins/
ExcludePath ^/opt\/metasploit/
ExcludePath ^/var\/mqm/
ExcludePath ^/var\/lib\/mysql/
ExcludePath ^/glusterfs/
ExcludePath ^/mnt/
ExcludePath ^/nfs/
ExcludePath ^/tmp\/clamav-.*/
MaxDirectoryRecursion 20
FollowDirectorySymlinks no
FollowFileSymlinks no
SelfCheck 600
ExitOnOOM yes
User root
ScanMail yes
ScanHTML yes
ScanOLE2 yes
ScanArchive yes
ForceToDisk no
ScanOnAccess yes
OnAccessIncludePath /bin
OnAccessIncludePath /boot
OnAccessIncludePath /etc
OnAccessIncludePath /home
OnAccessIncludePath /media
OnAccessIncludePath /mnt
OnAccessIncludePath /opt
OnAccessIncludePath /root
OnAccessIncludePath /sbin
OnAccessIncludePath /sftp
OnAccessIncludePath /usr
OnAccessExcludePath /opt/tomcat/.m2/repository
OnAccessExcludeRootUID yes
OnAccessMaxFileSize 5M
OnAccessDisableDDD no
OnAccessExtraScanning yes
DisableCertCheck no

I've got a few more bits of information;
- the FD it is missing is for 'anon_inode:inotify'

healthy system:
[root@xxxxxxxx<file:///root@x> ]# ls -l /proc/226347/fd
total 0
lr-x------. 1 root root 64 Nov 8 06:41 0 -> /dev/null
l-wx------. 1 root root 64 Nov 8 06:41 1 -> /dev/null
l-wx------. 1 root root 64 Nov 8 06:41 10 -> pipe:[2543521]
lrwx------. 1 root root 64 Nov 8 06:41 11 -> anon_inode:[fanotify]
lr-x------. 1 root root 64 Nov 8 06:41 12 -> anon_inode:inotify
l-wx------. 1 root root 64 Nov 8 06:41 2 -> /dev/null
lr-x------. 1 root root 64 Nov 8 06:41 3 -> /var/lib/sss/mc/initgroups
lrwx------. 1 root root 64 Nov 8 06:41 4 -> socket:[2543359]
l-wx------. 1 root root 64 Nov 8 03:26 5 -> /var/log/clamav/clamav.log
lrwx------. 1 root root 64 Nov 8 06:41 6 -> socket:[2544261]
lr-x------. 1 root root 64 Nov 8 06:41 7 -> pipe:[2543520]
l-wx------. 1 root root 64 Nov 8 06:41 8 -> pipe:[2543520]
lr-x------. 1 root root 64 Nov 8 06:41 9 -> pipe:[2543521]

Broken system:
[root@xxxxxxxxxx<file:///root@x> ]# ls -l /proc/33492/fd
total 0
lr-x------. 1 root root 64 Nov 7 10:58 0 -> /dev/null
l-wx------. 1 root root 64 Nov 7 10:58 1 -> /dev/null
l-wx------. 1 root root 64 Nov 7 10:58 10 -> pipe:[788328]
lrwx------. 1 root root 64 Nov 7 10:58 11 -> anon_inode:[fanotify]
lr-x------. 1 root root 64 Nov 5 09:52 13 -> /etc/clamd.d/scan.conf
lrwx------. 1 root root 64 Nov 5 09:52 14 -> /tmp/clamav-46ff34ef6c75cb2abc0435d1056ee697.tmp
l-wx------. 1 root root 64 Nov 7 10:58 2 -> /dev/null
lr-x------. 1 root root 64 Nov 7 10:58 3 -> /var/lib/sss/mc/initgroups
lrwx------. 1 root root 64 Nov 7 10:58 4 -> socket:[790831]
l-wx------. 1 root root 64 Nov 7 10:58 5 -> /var/log/clamav/clamav.log
lrwx------. 1 root root 64 Nov 7 10:58 6 -> socket:[790832]
lr-x------. 1 root root 64 Nov 7 10:58 7 -> pipe:[788327]
l-wx------. 1 root root 64 Nov 7 10:58 8 -> pipe:[788327]
lr-x------. 1 root root 64 Nov 7 10:58 9 -> pipe:[788328]

thanks
Tim

-----Original Message-----
From: G.W. Haywood via clamav-users <clamav-users@lists.clamav.net<mailto:%22G.W.%20Haywood%20via%20clamav-users%22%20%3cclamav-users@lists.clamav.net%3e>>
Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:ClamAV%20users%20ML%20%3cclamav-users@lists.clamav.net%3e>>
To: J.R. via clamav-users <clamav-users@lists.clamav.net<mailto:%22J.R.%20via%20clamav-users%22%20%3cclamav-users@lists.clamav.net%3e>>
Cc: G.W. Haywood <clamav@jubileegroup.co.uk<mailto:%22G.W.%20Haywood%22%20%3cclamav@jubileegroup.co.uk%3e>>
Subject: Re: [clamav-users] strace - select(13, [12], NULL, NULL, NULL) = -1 EBADF (Bad file descriptor) <0.000017>
Date: Thu, 07 Nov 2019 15:55:29 +0000

Hi there,

On Thu, 7 Nov 2019, J.R. via clamav-users wrote:

Which brought clamd back to life and the system load returned to

normal. no idea is this is a OS bug, a ClamAV bug or some kind of user

error, any help here will be appreciated.

What version of ClamAV? What OS? What customization / edits to config

files have you made?

And what are you scanning???

[Winner of the 2018 Consumer Credit Awards]

Re: [clamav-users] strace - select(13, [12], NULL, NULL, NULL) = -1 EBADF (Bad file descriptor) <0.000017> [ In reply to ]

tim.stubbs at telrock

Nov 10, 2019, 9:46 AM

Post #5 of 5 (298 views)

Permalink

update: I have now managed to recreate this issues on different
hardware, I can also simulate the sys load issues once the clamd
process is in its EBADF state.

I am still yet unable to trigger this issue, it seems to happen at
random, however we have now noticed the problems on more VM's running
all sorts of management applications.

Any ideas how I can debug this further to see what may be triggering
the problem? I haven't yet found any other references to this issue on
the internet?

thanks
Tim

-----Original Message-----
From: Tim Stubbs <tim.stubbs@telrock.com>
To: ClamAV users ML <clamav-users@lists.clamav.net>
Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
Subject: Re: [clamav-users] strace - select(13, [12], NULL, NULL, NULL)
= -1 EBADF (Bad file descriptor) <0.000017>
Date: Fri, 08 Nov 2019 12:19:27 +0000

thanks for the response;
we are experiancing this issues on a fresh install VM, a Java
application VM & a Jump server with gnome. a mix of 2 and 4 coure VM's
with 2,4 & 6GB RAM

[root@xxxxxxx]# uname -a
Linux xxxxxxxxxxxxxx 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13
22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@xxxxx ]# cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

# Config
LogFile /var/log/clamav/clamav.log
LogFileUnlock yes
LogFileMaxSize 10M
LogTime yes
LogSyslog no
LogRotate no
ExtendedDetectionInfo yes
PidFile /var/run/clamd.scan/clamd.pid
DatabaseDirectory /var/lib/clamav
LocalSocket /var/run/clamd.scan/clamd.sock
LocalSocketGroup virusgroup
LocalSocketMode 666
FixStaleSocket yes
MaxThreads 10
ReadTimeout 180
SendBufTimeout 200
MaxQueue 100
ExcludePath ^/proc/
ExcludePath ^/sys/
ExcludePath ^/root/
ExcludePath ^/var\/lib\/openvas\/plugins/
ExcludePath ^/opt\/metasploit/
ExcludePath ^/var\/mqm/
ExcludePath ^/var\/lib\/mysql/
ExcludePath ^/glusterfs/
ExcludePath ^/mnt/
ExcludePath ^/nfs/
ExcludePath ^/tmp\/clamav-.*/
MaxDirectoryRecursion 20
FollowDirectorySymlinks no
FollowFileSymlinks no
SelfCheck 600
ExitOnOOM yes
User root
ScanMail yes
ScanHTML yes
ScanOLE2 yes
ScanArchive yes
ForceToDisk no
ScanOnAccess yes
OnAccessIncludePath /bin
OnAccessIncludePath /boot
OnAccessIncludePath /etc
OnAccessIncludePath /home
OnAccessIncludePath /media
OnAccessIncludePath /mnt
OnAccessIncludePath /opt
OnAccessIncludePath /root
OnAccessIncludePath /sbin
OnAccessIncludePath /sftp
OnAccessIncludePath /usr
OnAccessExcludePath /opt/tomcat/.m2/repository
OnAccessExcludeRootUID yes
OnAccessMaxFileSize 5M
OnAccessDisableDDD no
OnAccessExtraScanning yes
DisableCertCheck no

I've got a few more bits of information;
- the FD it is missing is for 'anon_inode:inotify'

healthy system:
[root@xxxxxxxx ]# ls -l /proc/226347/fd
total 0
lr-x------. 1 root root 64 Nov 8 06:41 0 -> /dev/null
l-wx------. 1 root root 64 Nov 8 06:41 1 -> /dev/null
l-wx------. 1 root root 64 Nov 8 06:41 10 -> pipe:[2543521]
lrwx------. 1 root root 64 Nov 8 06:41 11 -> anon_inode:[fanotify]
lr-x------. 1 root root 64 Nov 8 06:41 12 -> anon_inode:inotify
l-wx------. 1 root root 64 Nov 8 06:41 2 -> /dev/null
lr-x------. 1 root root 64 Nov 8 06:41 3 -> /var/lib/sss/mc/initgroups
lrwx------. 1 root root 64 Nov 8 06:41 4 -> socket:[2543359]
l-wx------. 1 root root 64 Nov 8 03:26 5 -> /var/log/clamav/clamav.log
lrwx------. 1 root root 64 Nov 8 06:41 6 -> socket:[2544261]
lr-x------. 1 root root 64 Nov 8 06:41 7 -> pipe:[2543520]
l-wx------. 1 root root 64 Nov 8 06:41 8 -> pipe:[2543520]
lr-x------. 1 root root 64 Nov 8 06:41 9 -> pipe:[2543521]

Broken system:
[root@xxxxxxxxxx ]# ls -l /proc/33492/fd
total 0
lr-x------. 1 root root 64 Nov 7 10:58 0 -> /dev/null
l-wx------. 1 root root 64 Nov 7 10:58 1 -> /dev/null
l-wx------. 1 root root 64 Nov 7 10:58 10 -> pipe:[788328]
lrwx------. 1 root root 64 Nov 7 10:58 11 -> anon_inode:[fanotify]
lr-x------. 1 root root 64 Nov 5 09:52 13 -> /etc/clamd.d/scan.conf
lrwx------. 1 root root 64 Nov 5 09:52 14 -> /tmp/clamav-
46ff34ef6c75cb2abc0435d1056ee697.tmp
l-wx------. 1 root root 64 Nov 7 10:58 2 -> /dev/null
lr-x------. 1 root root 64 Nov 7 10:58 3 -> /var/lib/sss/mc/initgroups
lrwx------. 1 root root 64 Nov 7 10:58 4 -> socket:[790831]
l-wx------. 1 root root 64 Nov 7 10:58 5 -> /var/log/clamav/clamav.log
lrwx------. 1 root root 64 Nov 7 10:58 6 -> socket:[790832]
lr-x------. 1 root root 64 Nov 7 10:58 7 -> pipe:[788327]
l-wx------. 1 root root 64 Nov 7 10:58 8 -> pipe:[788327]
lr-x------. 1 root root 64 Nov 7 10:58 9 -> pipe:[788328]

thanks
Tim

-----Original Message-----
From: G.W. Haywood via clamav-users <clamav-users@lists.clamav.net>
Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
To: J.R. via clamav-users <clamav-users@lists.clamav.net>
Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
Subject: Re: [clamav-users] strace - select(13, [12], NULL, NULL, NULL)
= -1 EBADF (Bad file descriptor) <0.000017>
Date: Thu, 07 Nov 2019 15:55:29 +0000

Hi there,

On Thu, 7 Nov 2019, J.R. via clamav-users wrote:

> > Which brought clamd back to life and the system load returned to
> > normal. no idea is this is a OS bug, a ClamAV bug or some kind of
> > user
> > error, any help here will be appreciated.
>
> What version of ClamAV? What OS? What customization / edits to config
> files have you made?

And what are you scanning???

[Winner of the 2018 Consumer Credit Awards]

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml