Mailing List Archive

Ged,

That's a fair assessment. This is why I asked.

Thanks,
Micah

?On 9/13/19, 11:26 AM, "clamav-users on behalf of G.W. Haywood via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

Hi there,

On Fri, 13 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote:

> One thing we could do is have clamd "start" before loading the
> database. That is to say that it would immediately begin listening
> on the unix/tcp socket for requests and fork into the background so
> as not to block the boot process. All scan requests would then be
> blocked while the database loads. I imagine this would solve most
> of the frustration around boot-up load time.

I don't think you should be trying to second-guess stuff like this,
and I don't quite see how in these days of parallel boot processes
that anything will get blocked that doesn't need to be blocked. Will
you be looking at the network interfaces? The routes? You'll end up
writing another systemd. The system administrator/integrator needs to
earn his living somehow; not asking a utility to do things when it's
not yet ready to do them is one of his jobs. It's why there are all
those symlinks in /etc/rc3.d/.

> Does this have any appeal?

Seems like a waste of effort to me.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

I'm sorry, Ged. I didn't mean to demean the work of Julius Plenz, asulfrian, or yourself. I stepped into my current position on the ClamAV team just over two years ago and in my time here there have been many tasks that have been on the backburner or ignored entirely for multiple years. We have been working to try to rectify this but it is a slow process. I do appreciate your work to try to update and test the original patch.

Alberto saw the discussions here demonstrating a need for this feature. He offered to share a similar patch he had working in production on his systems, one that was updated to apply cleanly with the 0.102 code base. I wanted to share it immediately with you all. You're right that I should've given more credit to the authors of the prior work. It was only my intention to credit Alberto so as not to give the false impression that his work was my own.

Regards,
Micah

?On 9/13/19, 11:15 AM, "clamav-users on behalf of G.W. Haywood via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

Hi there,

On Thu, 12 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote:

> https://bugzilla.clamav.net/show_bug.cgi?id=10979#c19
> This patch applies to the current head of dev/0.102 ...

If the development version is a step too far, the two files which I
posted on September 10th implement a patch which has been sitting on
the ClamAV Bugzilla (at #c2) for nearly three years:

https://bugzilla.clamav.net/show_bug.cgi?id=10979#c13
https://bugzilla.clamav.net/show_bug.cgi?id=10979#c14

These replace two files in the current (v0.101.4) release, to produce
results very similar to those from the patch at #c19 for v0.102.x.

Unfortunately there are so many cosmetic changes in the development
version that a direct comparison of the patches might be tedious, but
the essentials are the same. Load new data in a separate thread, and
in the meantime scan using the old database; switch database pointers
(virtually instantaneous) on reload completion; ignore database reload
requests if reloading is already in progress; and when the old data is
no longer needed, drop it. Test results and/or observations welcome.

This will not of course help start-up times at all, but it's easy to
arrange to load a smaller database at startup if that's what you feel
you must do - there has been a discussion about using what I'll call
non-standard databases recently. Personally I don't see the need for
anything like that; the runtimes of my clamd daemons are rarely less
than months, even if I'm testing things, so it's of no consequence if
loading the data at the beginning of a run takes a couple of minutes.
Since I'm only scanning mail, rather than scan it with less than the
full deck I'll just delay it a couple of minutes. Until I worked on
this patch, that's what I'd been doing on every database reload and,
as I've always maintained, it's really no big deal.

> ...do not confuse the fact that we are paid with the thought that
> you are paying us.

I'm not sure that ham-fisted attempt at a justification was entirely
called for, Micah.

You had a patch for several years. Then, two and a half days after I
posted the two files shown above, you're galvanized into action; but
you studiously avoid mention of the prior work by several people, and
then imply that people are confused when everything is crystal clear.

> We of course always appreciate help from the community ...

Perhaps you could try to make it a little more obvious.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

* Micah Snyder via clamav-users:

> [ClamAV] would immediately begin listening on the unix/tcp socket for
> requests and fork into the background so as not to block the boot
> process.

To me, slowing down the boot process is just the (admittedly annoying)
symptom of an underlying ClamAV issue. Based on the delays that we have
seen over the past months, I'd say that ClamAV's database handling does
not scale well enough, and I think that's what needs fixing.

-Ralph

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi Micah,

On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote:

> I'm sorry, Ged...

Apology accepted. :)

I'm now running the development (0.102) version of clamd, patched with
Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched
one, and one with the patch that I posted on Bugzilla).

The milter scans all mail with all three daemons. On the arrival of a
message, if the database is not already being reloaded I start a fresh
reload before the scan so that, for all scans, a reload always executes
concurrently. Nothing seems to have broken, and so far there's nothing
terribly interesting to report other than the strange failure to detect
which I sent to Joel early this week (and which I'm sure has nothing to
do with these patches).

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

> One thing we could do is have clamd "start" before loading the database.
> That is to say that it would immediately begin listening on the unix/tcp socket
> for requests and fork into the background so as not to block the boot process.
> All scan requests would then be blocked while the database loads.
> I imagine this would solve most of the frustration around boot-up load time

I guess I kind of jumped the gun on this one, chalk it up to the
late-night message posting...

While it is an older linux distro with the init startup, I simply
moved ClamAV to near the end of the boot process... problem solved. I
noticed some people made that recommendation too.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote:
> Hi Micah,
>
> On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote:
>
>> I'm sorry, Ged...
>
> Apology accepted. :)
>
> I'm now running the development (0.102) version of clamd, patched with
> Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched
> one, and one with the patch that I posted on Bugzilla).
>
> The milter scans all mail with all three daemons.  On the arrival of a
> message, if the database is not already being reloaded I start a fresh
> reload before the scan so that, for all scans, a reload always executes
> concurrently.  Nothing seems to have broken, and so far there's nothing
> terribly interesting to report other than the strange failure to detect
> which I sent to Joel early this week (and which I'm sure has nothing to
> do with these patches).

I've been running a patched 101.4 for a few weeks now and unfortunately
I'm observing a memory leak from the multithreaded database reloads.

I'm observing clamd memory usage going up when the new database loads
and then eventually dropping down to 1.3G again. For some reason
"eventually" means the memory usage drops down only after clamd
processes the next e-mail.

The problem however shows itself if clamd happens to reload its database
2 times if a row with no mail processed in between. Seemingly it will
have 3 databases in memory then and the next mail being processed
releases one of them, but the extra database will remain "somewhere".

All sorts of weird problems always keep popping up on due to low traffic
on the server. :)

Reio

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi Reio,

On Mon, 28 Oct 2019, Reio Remma via clamav-users wrote:

> ...
> I've been running a patched 101.4 for a few weeks now and unfortunately
> I'm observing a memory leak from the multithreaded database reloads.
>
> I'm observing clamd memory usage going up when the new database loads
> ...
> The problem however shows itself if clamd happens to reload its database
> 2 times if a row with no mail processed in between. Seemingly it will
> have 3 databases in memory then and the next mail being processed
> releases one of them, but the extra database will remain "somewhere".
> ..

As I said I'm using 0.102-rc with the older patch, and I haven't seen
this behaviour (but I have been looking for it, and anything like it,
using Nagios etc.). On our servers there's no risk of clamd reloading
databases without processing a message inbetween the reloads, but I'm
sure I could arrange it if neccessary. :) Unfortunately at the moment
I have no time to investigate but I guess it will be simple to fix if
it isn't something peculiar to your setup - for example it might be a
problem with threads in a library. From my reading of the code, going
back admittedly a little while now, it seemed very clear that the old
database should be freed unconditionally after the new one was loaded.

I'd suggest that you raise an issue in the ClamAV Bugzilla.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 28/10/2019 12:55, Reio Remma via clamav-users wrote:
> On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote:
>> Hi Micah,
>>
>> On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote:
>>
>>> I'm sorry, Ged...
>>
>> Apology accepted. :)
>>
>> I'm now running the development (0.102) version of clamd, patched with
>> Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched
>> one, and one with the patch that I posted on Bugzilla).
>>
>> The milter scans all mail with all three daemons.  On the arrival of a
>> message, if the database is not already being reloaded I start a fresh
>> reload before the scan so that, for all scans, a reload always executes
>> concurrently.  Nothing seems to have broken, and so far there's nothing
>> terribly interesting to report other than the strange failure to detect
>> which I sent to Joel early this week (and which I'm sure has nothing to
>> do with these patches).
>
> I've been running a patched 101.4 for a few weeks now and
> unfortunately I'm observing a memory leak from the multithreaded
> database reloads.
>
> I'm observing clamd memory usage going up when the new database loads
> and then eventually dropping down to 1.3G again. For some reason
> "eventually" means the memory usage drops down only after clamd
> processes the next e-mail.
>
> The problem however shows itself if clamd happens to reload its
> database 2 times if a row with no mail processed in between. Seemingly
> it will have 3 databases in memory then and the next mail being
> processed releases one of them, but the extra database will remain
> "somewhere".
>
> All sorts of weird problems always keep popping up on due to low
> traffic on the server. :)

Fortunately 0.102.0 with the patch from ClamAV team doesn't have that
issue and seems to release the extra memory right away.

Happily running 0.102.0 now.

Good luck,
Reio

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 31/10/2019 12:04, Reio Remma wrote:
> On 28/10/2019 12:55, Reio Remma via clamav-users wrote:
>> On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote:
>>> Hi Micah,
>>>
>>> On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote:
>>>
>>>> I'm sorry, Ged...
>>>
>>> Apology accepted. :)
>>>
>>> I'm now running the development (0.102) version of clamd, patched with
>>> Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched
>>> one, and one with the patch that I posted on Bugzilla).
>>>
>>> The milter scans all mail with all three daemons.  On the arrival of a
>>> message, if the database is not already being reloaded I start a fresh
>>> reload before the scan so that, for all scans, a reload always executes
>>> concurrently.  Nothing seems to have broken, and so far there's nothing
>>> terribly interesting to report other than the strange failure to detect
>>> which I sent to Joel early this week (and which I'm sure has nothing to
>>> do with these patches).
>>
>> I've been running a patched 101.4 for a few weeks now and
>> unfortunately I'm observing a memory leak from the multithreaded
>> database reloads.
>>
>> I'm observing clamd memory usage going up when the new database loads
>> and then eventually dropping down to 1.3G again. For some reason
>> "eventually" means the memory usage drops down only after clamd
>> processes the next e-mail.
>>
>> The problem however shows itself if clamd happens to reload its
>> database 2 times if a row with no mail processed in between.
>> Seemingly it will have 3 databases in memory then and the next mail
>> being processed releases one of them, but the extra database will
>> remain "somewhere".
>>
>> All sorts of weird problems always keep popping up on due to low
>> traffic on the server. :)
>
> Fortunately 0.102.0 with the patch from ClamAV team doesn't have that
> issue and seems to release the extra memory right away.
>
> Happily running 0.102.0 now.

Has anyone got the threaded reload patch working with 0.102.2?

When rebuilding my RPM with 0.102.2, I get the following error when the
patch is being applied:

+ echo 'Patch #0 (clamd-threaded-reloading.patch):'
Patch #0 (clamd-threaded-reloading.patch):
+ /usr/bin/cat ~/rpmbuild/SOURCES/clamd-threaded-reloading.patch
+ /usr/bin/patch -p1 -b --suffix .threaded_reloading --fuzz=0
patching file clamd/clamd.c
Reversed (or previously applied) patch detected! Assume -R? [n]

Thanks,
Reio