Mailing List Archive

On 17.10.2019 19:04, Micah Snyder (micasnyd) via clamav-users wrote:
> Vladislav, Ged:
>
> Reloading select databases is not feasible at this time, because signatures are loaded into the same structures in memory and that entire thing is recreated on reload.
>
> Regarding the threaded reload feature ( ticket: https://bugzilla.clamav.net/show_bug.cgi?id=10979 )...
>
> The main reason the "threaded reload" patch is held back at present is primarily because the recent work and interest in the patch came at the same time that 0.102 development was in code freeze while we tested and applied bug fixes for release. Reloading in a separate thread means that the memory usage will double (going from roughly ~750MB to ~1500MB) during the reload before it frees the original signatures and drops back to ~750MB.
>
> We already have many complaints about freshclam and clamd memory usage, and this change in behavior could cause trouble for some users, so we want to provide an option to reload the traditional way. That's the second reason why the patch isn't been merged for 0.103 yet. We have to dedicate some time to code the ability to reload either way. It is absolutely on our to-do list.

Great to hear work is ongoing.

I've switched to patched 0.104 just this hour.

I can easily deal with higher memory usage, but loss of service for 1-3
minutes is much harder to deal with.

Thanks!
Reio

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 17/10/2019 17:44, G.W. Haywood via clamav-users wrote:
> Hello again,
>
> On Thu, 17 Oct 2019, Vladislav Kurz via clamav-users wrote:
>
>> Is there anything blocking this patch from being accepted ?
>
> As far as I know, only the (significant) pressures and (AFAICT equally
> significant) limitations on developer time at Cisco/Talos/SourceFire.
>
>> I'm noticing clamav-reload related timeouts on more and more (mostly
>> older or low-end) servers, which were running just fine a year or two
>> ago.
>
> There other ways of dealing with this, as I'm sure you're aware, but
> using the patched daemon you only have to worry about the increased
> memory consumption during databse reloads.

Hello

Well, the only option other than using your patch I know of is to
increase the AV scan timeout in SMTP server. But I'm afraid that the
sender might give up waiting for the final acknowlegement of DATA. And I
do not want to accept the message before it is scanned (to avoid
backscatter or silent discard of messages).

> It seems to me that the amount of junk mail grows ever more quickly.
> When not testing clamd, I routinely block for example all connections
> from more than a hundred countries, a similar number of ASNs, and all
> hosts which score a total of three or more in our weighted DNSBL list.
> That's quite apart from the more targeted block lists.  Obviously this
> isn't an option for everyone, but here it makes the difference between
> email being useful, and email being nothing but a nuisance.

The speed of scan itself is not a problem, just the reload takes a few
minutes on low-end/old servers.



--
S pozdravem
Vladislav Kurz

Centrála: Celní 17/5, 63900 Brno, CZ
Web: http://www.webstep.net
E-Mail: podpora@webstep.net
Tel: 840 840 700, +420 548 214 711
Obchodní podmínky: https://zkrat.to/op

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Fri, 18 Oct 2019, Vladislav Kurz via clamav-users wrote:
> On 17/10/2019 17:44, G.W. Haywood via clamav-users wrote:
>
>> There other ways of dealing with this, as I'm sure you're aware, but
>> using the patched daemon you only have to worry about the increased
>> memory consumption during databse reloads.
>
> Well, the only option other than using your patch I know of is to
> increase the AV scan timeout in SMTP server. But I'm afraid that the
> sender might give up waiting for the final acknowlegement of DATA. ...

In my experience the only senders who give up early are the spammers,
and rather than at the SMTP 'DATA' stage it's usually at a fairly long
greetpause (I don't want to say how long that is in public). You will
probably not be surprised that I don't care about the spammers.

Here are some of the default Sendmail timeouts as distributed with the
latest Sendmail sources. I'm sure *many* installations are using them:

$ grep -C2 confTO_DATA sendmail-8.16.0.41/cf/README
confTO_RCPT Timeout.rcpt [1h] The timeout waiting for a response
to the RCPT command.
confTO_DATAINIT Timeout.datainit
[5m] The timeout waiting for a 354
response from the DATA command.
confTO_DATABLOCK Timeout.datablock
[1h] The timeout waiting for a block
during DATA phase.
confTO_DATAFINAL Timeout.datafinal
[1h] The timeout waiting for a response
to the final "." that terminates a

As you can see, at one hour, the data 'block' and 'final' timeouts
(where any scan will take place) are well in excess of likely database
reload times, so I do not think you have to worry about extending your
timeouts a few minutes longer. Don't forget that electronic mail came
out in the 1970s (I was there at the time:() and it largely replaced
putting a piece of paper in an envelope and licking a stamp. Way back
then, timeouts of around an hour for mail delivery seemed fairly short
to us, and to me they still seem quite reasonable now. Remember that
email is not (and never was supposed to be) Instant Messaging. Don't
forget that very often on a Linux box, the default connection timeout
for the TCP ESTABLISHED state is five days.

You could also run more than one clamd, and reload them on different
schedules; you could just refuse mail connections while reloading; or
(similarly) you could schedule out-of-service periods for SMTP, only
reloading during those out-of-service periods.

As I've maintained for some considerable time, this reload time issue
isn't really a big deal, and in any case it's manageable. Far more
pressing in my opinion are the issues of coverage and accuracy.

> ... I do not want to accept the message before it is scanned (to
> avoid backscatter or silent discard of messages).

Sure, backscatter is a plague.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Vladislav,

If you are going to put everything on hold while your AV database
reloads, be sure you have appropriate timeout settings for your milter
or whatever else is handling things so the email program doesn't
timeout waiting for a response from it.

While the *default* timeouts for email chatter are rather high, many
administrators do lower those values (in Sendmail, Postfix, Exim, etc)
to prevent lingering connections for one reason or another (since most
email servers don't close the connection automatically until the
client side sends a quit (which many spammers don't), it then falls
back to the timeout setting to close)....

This page shows both the default and *minimum* (recommended?) values
for sendmail, note how some timeouts drop from 1h to 5m...

https://sendmail.org/~ca/email/doc8.12/op-sh-4.html

But also you don't know what the client-side connection settings are
that are sending you mail...

If you *must* receive the email then and there, then using the patch
as talked about before would eliminate any wait times during reloads.
Otherwise you might as well just make sure your timeout values are
high enough and have everything sit & wait while the DB reloads. Worst
case it times out on the client-side and if it's a legit email server
it would treat it as a tempfail and retry later.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi there,

On Fri, 18 Oct 2019, J.R. via clamav-users wrote:

> ... most email servers don't close the connection automatically
> until the client side sends a quit (which many spammers don't), it
> then falls back to the timeout setting to close ...

Time was when you could rely on that, but even some of the big players
get it round their necks. I frequently see delays between 20 and 300
seconds before the QUIT command after EOM. Amazon for example is one
big offender. And with almost 98% of the mail I see now being spam it
would be asking for denial of service to rely on clients closing their
connections. Unless I'm testing clamd, the moment they trip over one
of the wires (and there are *lots* of wires), I'll drop the connection
and throw another IP into the tarpit.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Wednesday 04 September 2019, Sergey wrote:

> Since some time there has been a noticeable increase a launch time.

Good acceleration in the new version!

0.101.4 (124 sec):
Tue Nov 26 21:08:26 2019 -> Bytecode: Security mode set to "TrustSigned".
Tue Nov 26 21:10:30 2019 -> Loaded 6565044 signatures.


0.101.5 (22 sec):
Tue Nov 26 21:12:02 2019 -> Bytecode: Security mode set to "TrustSigned".
Tue Nov 26 21:12:24 2019 -> Loaded 6565044 signatures.

--
Regards,
Sergey

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Tuesday 26 November 2019, Sergey wrote:

> 0.101.5 (22 sec):
> Tue Nov 26 21:12:02 2019 -> Bytecode: Security mode set to "TrustSigned".
> Tue Nov 26 21:12:24 2019 -> Loaded 6565044 signatures.

Hm... It's for big cld files. More compact cvd files loaded about 10 seconds
longer:

Tue Nov 26 21:37:35 2019 -> Bytecode: Security mode set to "TrustSigned".
Tue Nov 26 21:38:03 2019 -> Loaded 6565044 signatures.

Previously the difference between cvd and cld was not so visible.

--
Regards,
Sergey

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

In addition to the improvements in 0.101.5, 0.102.1, we shipped an update to main & daily yesterday and this morning that reduced load time by removing ignored signatures (signatures in main that we wished to drop, and thus ignored in daily.ign2/daily.ign).

On my laptop, I observed

0.102.0, databases about 1 week old:
Time: 72.238 sec

0.102.1, databases about 1 day old:
Time: 35.780 sec

0.102.1, databases up to date:
Time: 18.174 sec

Happy scanning!
-Micah


?On 11/26/19, 12:19 PM, "clamav-users on behalf of Sergey" <clamav-users-bounces@lists.clamav.net on behalf of a_s_y@sama.ru> wrote:

On Wednesday 04 September 2019, Sergey wrote:

> Since some time there has been a noticeable increase a launch time.

Good acceleration in the new version!

0.101.4 (124 sec):
Tue Nov 26 21:08:26 2019 -> Bytecode: Security mode set to "TrustSigned".
Tue Nov 26 21:10:30 2019 -> Loaded 6565044 signatures.


0.101.5 (22 sec):
Tue Nov 26 21:12:02 2019 -> Bytecode: Security mode set to "TrustSigned".
Tue Nov 26 21:12:24 2019 -> Loaded 6565044 signatures.

--
Regards,
Sergey

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 26.11.2019 20:12, Micah Snyder (micasnyd) via clamav-users wrote:
> In addition to the improvements in 0.101.5, 0.102.1, we shipped an update to main & daily yesterday and this morning that reduced load time by removing ignored signatures (signatures in main that we wished to drop, and thus ignored in daily.ign2/daily.ign).
>
> On my laptop, I observed
>
> 0.102.0, databases about 1 week old:
> Time: 72.238 sec
>
> 0.102.1, databases about 1 day old:
> Time: 35.780 sec
>
> 0.102.1, databases up to date:
> Time: 18.174 sec
>
> Happy scanning!
> -Micah

Very nice indeed! We're down to 17-18 seconds as well now with fully
updated  1.3M signatures.

Thanks and good luck!
Reio

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On Tuesday 26 November 2019, Micah Snyder (micasnyd) via clamav-users wrote:

> In addition to the improvements in 0.101.5, 0.102.1, we shipped an update
> to main & daily yesterday

I known and my test was with new main & daily for 0.101.4 and 0.101.5 both.
So it shows improvement of clamd's code in clear. Nice works, thanks!

Special thanks for the new main also.

--
Regards,
Sergey

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml