Hello,
> > So I have Clam setup in network mode.
>
> I'm not sure that I know what that means. Please elaborate in as much
> detail as it would take for me to reproduce your system.
>
> The whole setup is in AWS, I have one instance setup as a "ClamAV server"
and 7 instances setup as "ClamAV clients" this link
<
https://xn--blgg-hra.no/2016/03/clamav-clientserver-setup/> was the main
reference for this setup. So basically I have in the clamd.conf file on
each instance the TCP port and IP Address lines uncommented and configured
with the IP address of the "server" instance. We have the TCP port 3310
allowed between all the instances involved. I installed Clam on each
instance from source, on the "server" clamd and freshclam are both running
and on the "clients" I have disabled clamd and freshclam, I have clamonacc
setup as a service to watch a specific uploads directory on each instance
also a script setup to run a full system clamdscan starting at 2 am on each
instance staggered by 15 minutes. The config files on the "clients" as I
said previously are setup to point to the "server", I also have the
file/directory exclusions setup in them. The "server's" config file is
where the VirusEvent is configured.
> On the server I have the VirusEvent line in the clamd.conf file
>
> So I guess you're running clamd. Be aware that there have been some
> problems with the VirusEvent feature which have only fairly recently
> been fixed (as late as October 2019 - see for example this link:
> https://blog.clamav.net/2019/10/clamav-01020-has-been-released.html),
> and you might expect that, depending on your use case, there could be
> relatively new code in there which hasn't yet been as well exercised
> as some of the other code has been.
>
> You are correct, on the server I am running clamd. I was not aware the
the code was that new, I'll review the link you provided.
Please tell us
> What is the server; what resources it has (particularly CPU & memory);
> what operating system it uses; what version of ClamAV it uses and how
> that was installed; the full configuration files; the exact VirusEvent
> script; what you are scanning, how, and how it is presented to ClamAV;
> an example line of the log file that you're looking for; how you know
> that the last line is the one you're looking for; what other processes
> are running on the sever and what resources are used by them; relevant
> log extracts etc.; and as much about the client(s) too - how many of
> them; what they are; what load they present to the server; etc..
The "server" instance is a t3a.small, 2 CPUs and 2 GB of memory, running
Ubuntu 18.04, ClamAV version is 0.102.1, and as I stated previously it was
installed from source using the guide on the ClamAV site. I've attached
the server's full configuration file (cleansed) as well as the VirusEvent
script (also cleansed). I have both clamonacc and clamdscan scanning.
Clamonacc is scanning a particular directory on each client 24/7, I have
created a clamonacc.service file and loaded that into systemd. Clamdscan
is setup to do a full system scan on each client instance starting at 2 am
EST staggered by 15 minutes each a cron job kicks the scan off. Here is
the log output in question:
Wed Feb 5 20:00:16 2020 -> instream(10.5.1.217@44956):
Eicar-Test-Signature(aa991d6e29bf8eb4c1b56c599dffce0a:70) FOUND
Wed Feb 5 20:00:16 2020 -> ERROR: VirusEvent: fork failed.
I know that the script is giving me the last line of the log file because
it includes the timestamp. So if I run the script myself and then look at
the log file I can see that the last line of the log is the same as the
line that the script included in the email. The server instance was spun
up specifically for this so other then what is included in the default
Ubuntu AWS image the only things installed on it are ClamAV and msmtp (for
sending the emails). The 7 clients are all running Ubuntu as well, as for
load, they are not all speaking to the server at once, here is a screenshot
of the resources box presented in clamdtop when one of the servers is
running a full scan:
[image: image.png]
> Starting two days ago the email stopped being sent when a virus was
>> > found when I was running tests. Saw the "fork failed" error and after
>> > some troubleshooting which did not reveal anything
>> Please tell us
>> the EXACT error message; where you found it; what the troubleshooting
>> was; the test results; what you were doing at the time; and what you
>> were looking for which was not revealed in the test results.
>>
>
Exact error message is above, but to keep things clear here it is again:
Wed Feb 5 20:00:16 2020 -> ERROR: VirusEvent: fork failed.
Admittedly the troubleshooting I did was fairly basic. First, I made sure
that msmtp could still send an email from the server (it could). Second,
made sure the script was able to be ran (it is). Third, made sure that the
client instance was connecting to the server instance when running a scan
(it was, verifed that by watching clamdtop on the server instance while the
client was running the scan).
> I tried rebooting the server. After the server came back up
> > VirusEvent started working
> It seems like the server might have been running out of resources, but
> that's just my conjecture. Please tell us what you have done to
> verify that the server has enough resources to do the tasks which it
> has to do - for example, have you studied the 'man' page for 'top'?
If I watch the resources being used on the server during a scan on one of
the clients the CPU usage averages around 85%.
>
> > so I chalked it up to the server just needing a reboot.
> Very woolly thinking, a bit like working with Windows boxes. I run
> servers for sometimes more than a year without a reboot, including
> servers which run several clamd daemons. I never expect any server to
> be "just needing a reboot", and if a production server does need a
> reboot to make it work, in the absence of extenuating circumstances I
> will consider it broken, and fix it.
Fair enough.
> Yesterday same thing started to happen, during testing I realized
> > that the emails were not being sent.
> Please describe the testing - carefully - and the mail system.
Testing the clamdscan cron job. I have msmtp installed on the server, it
is setup to send emails through AWS SES.
> Checked the logs on the server and saw the "fork failed" error
> > again, tried another reboot but this time that has not worked.
> Please tell us what IS working; what resources are being used; etc.
Scans are working/running, clamd and freshclam on working/running, virus
detection is happening, just the VirusEvent is not working.
> I have found two other threads in this mailing list with the same
> > error, but neither has any solutions to the problem. I know this
> > setup can work I'm just stuck on why this error keeps popping up.
> Please point us to those threads as I'm sure some of the list threads
> about failed forks are not relevant to this issue. The only one I see
> which might be relevant is over three years old (January 2017, which
> is very old in terms of ClamAV development) and, as you say, it was in
> any case uninformative all round.
I found two, one is the same one that you reference, the other I cannot
find now but contained even less information than the January '17 one.
On Fri, Feb 7, 2020 at 9:53 AM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:
> Hi there,
>
> On Thu, 6 Feb 2020, Tom Ossman via clamav-users wrote:
>
> > So I have Clam setup in network mode.
>
> I'm not sure that I know what that means. Please elaborate in as much
> detail as it would take for me to reproduce your system.
>
> > On the server I have the VirusEvent line in the clamd.conf file
>
> So I guess you're running clamd. Be aware that there have been some
> problems with the VirusEvent feature which have only fairly recently
> been fixed (as late as October 2019 - see for example this link:
> https://blog.clamav.net/2019/10/clamav-01020-has-been-released.html),
> and you might expect that, depending on your use case, there could be
> relatively new code in there which hasn't yet been as well exercised
> as some of the other code has been.
>
> > uncommented and in place of the example I have it set to run a
> > script which is supposed to grab the last line of the clamd.log file
> > add that to a text file which is then emailed to us.
>
> Please tell us
>
> What is the server; what resources it has (particularly CPU & memory);
> what operating system it uses; what version of ClamAV it uses and how
> that was installed; the full configuration files; the exact VirusEvent
> script; what you are scanning, how, and how it is presented to ClamAV;
> an example line of the log file that you're looking for; how you know
> that the last line is the one you're looking for; what other processes
> are running on the sever and what resources are used by them; relevant
> log extracts etc.; and as much about the client(s) too - how many of
> them; what they are; what load they present to the server; etc..
>
> > Starting two days ago the email stopped being sent when a virus was
> > found when I was running tests. Saw the "fork failed" error and after
> > some troubleshooting which did not reveal anything
>
> Please tell us
>
> the EXACT error message; where you found it; what the troubleshooting
> was; the test results; what you were doing at the time; and what you
> were looking for which was not revealed in the test results.
>
> > I tried rebooting the server. After the server came back up
> > VirusEvent started working
>
> It seems like the server might have been running out of resources, but
> that's just my conjecture. Please tell us what you have done to
> verify that the server has enough resources to do the tasks which it
> has to do - for example, have you studied the 'man' page for 'top'?
>
> > so I chalked it up to the server just needing a reboot.
>
> Very woolly thinking, a bit like working with Windows boxes. I run
> servers for sometimes more than a year without a reboot, including
> servers which run several clamd daemons. I never expect any server to
> be "just needing a reboot", and if a production server does need a
> reboot to make it work, in the absence of extenuating circumstances I
> will consider it broken, and fix it.
>
> > Yesterday same thing started to happen, during testing I realized
> > that the emails were not being sent.
>
> Please describe the testing - carefully - and the mail system.
>
> > Checked the logs on the server and saw the "fork failed" error
> > again, tried another reboot but this time that has not worked.
>
> Please tell us what IS working; what resources are being used; etc.
>
> > I have found two other threads in this mailing list with the same
> > error, but neither has any solutions to the problem. I know this
> > setup can work I'm just stuck on why this error keeps popping up.
>
> Please point us to those threads as I'm sure some of the list threads
> about failed forks are not relevant to this issue. The only one I see
> which might be relevant is over three years old (January 2017, which
> is very old in terms of ClamAV development) and, as you say, it was in
> any case uninformative all round.
>
> > Is there anything I can do to get more information from Clam about
> > what is happening to hopefully point me to a solution?
>
> You might enable debug logging, but at the moment the issues are more
> about us getting information from you than you getting it from ClamAV.
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
