Mailing List Archive

Regexes can be slow or even extremely slow to apply, depending on the
implementation. Backtracking is the worst, perhaps taking exponential
time, but often is cut off by artificial limits.

Does ClamAV perchance precompute Deterministic Finite Automata for the
regexes? These run fast, but take time exponential in regex length to
set up:

https://en.wikipedia.org/wiki/Regular_expression#Implementations_and_running_times



On Thu, 11 Apr 2019 00:56:04 +0000
"Micah Snyder \(micasnyd\) via clamav-users"
<clamav-users@lists.clamav.net> wrote:

> JME,
>
> As you've pointed out, it appears that some signatures containing a
> PCRE regex components are responsible for slow scan times on larger
> email files.
>
> I did a bunch of profiling similar to what Maarten did earlier in
> order to narrow it down. I found that Email.Phishing.VOF2 signatures
> are performing slower with the eml sample you sent me.
> Email.Phishing.VOF2 signatures contain a PCRE regex component to
> alert on email attachments with specific names. Now that we've
> determined which signatures are performing slowly in these cases, I
> am hopeful that we will be able to optimize the Email.Phishing.VOF2
> signatures to improve performance.
>
> I will note that your idea to lower the PCRERecMatchLimit setting to
> 1 will effectively neuter all signatures that rely on regexes and so
> I can't recommend this.
>
> Regards,
> Micah


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Hi Micah,

Sorry to pester you, but have you any update on when the remaining
Phishtank signatures will be getting removed? It would be really great to
get scan times properly back to normal.

Best regards
Mark

On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com>
wrote:

> Mark,
>
>
> Yes, the plan is still to remove the rest of the Phishtank signatures. We
> wanted to get things back to relative normal and resolve the immediate
> crisis. We’ll remove the rest of them soon.
>
>
>
> Best,
>
> Micah
>
>
>
> *From: *Mark Allan <markjallan@gmail.com>
> *Date: *Tuesday, April 9, 2019 at 6:26 AM
> *To: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net>
> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>
>
>
> The scan times are definitely better than they were - in fact, they're
> back to how they were before last week's inclusion of the Phishtank
> signatures. They're still almost double what they used to be though, and as
> far as I can see, there are still almost 4000 Phishtank signatures in the
> DB:
>
> $ sigtool --find Phishtank | wc -l
>
> 3968
>
>
>
> Can I request that those ones also be removed please?
>
>
>
> Best regards
>
> Mark
>
>
>
> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com>
> wrote:
>
> Tim,
>
>
>
> There are a couple of ways for users to drop specific categories of
> signatures at this time. Sadly, they wouldn’t have helped this last week.
> These include bytecode signatures, PUA (potentially unwanted applications)
> signatures, Email.Phishing and HTML.Phishing signatures, and the
> Safebrowsing database.
>
>
>
> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or
> Email.Phishing.Phishtank then they could have been disabled with the
> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>
>
>
> Maybe a better option would be for us to create a new optional database
> for phishing signatures. However, the names for the databases are hardcoded
> into freshclam, so it is non-trivial to add a new database and would
> require a few changes to ClamAV’s code. We have talked about making the
> databases easier to add/remove in the future so users can have more
> categories to enable/disable. In this light, it ties in well with existing
> plans.
>
>
>
> Of note the Phishtank sigs from Friday’s daily were removed yesterday and
> scan times should be back to normal.
>
>
>
> Regards,
>
> Micah
>
>
>
> *From: *Tim Hawkins <tim.hawkins@redflaggroup.com>
> *Date: *Friday, April 5, 2019 at 6:06 PM
> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan <
> markjallan@gmail.com>
> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>
>
>
> Hi Micah
>
>
> Does clamav partition the database so that signatures that are mainly
> associated with email scanning can be dropped out for folks only needing
> filesystems scans, none of our systems use email, and we dont make use of
> the mailer extension.
>
> Having to load all the email focused signatures could as you have observed
> impact performance.
>
> Sent from Nine <http://www.9folders.com/>
> ------------------------------
>
> *From:* "Micah Snyder (micasnyd) via clamav-users" <
> clamav-users@lists.clamav.net>
> *Sent:* Saturday, April 6, 2019 03:18
> *To:* ClamAV users ML; Mark Allan
> *Cc:* Micah Snyder (micasnyd)
> *Subject:* [External] Re: [clamav-users] Scan very slow
>
>
>
> Regarding slow scan times today (and slow scan times in general), it
> appears that the signatures we generate based on PhishTank’s feed for
> phishing URLs are resulting in very slow load and scan times.
>
>
>
> Today’s daily update saw 7448 new Phishtank signatures (much higher than
> usual) coinciding with the immediate performance drop for load time and
> scan time. One user reported that the load time today on some of his
> slower machines was slow enough to exceed the timeout for service startup (
> https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>
>
>
> In limited testing on my own machine I saw the following change after
> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>
> - Database load time on my laptop went from 75.43203997612 seconds
> down to 14.859203100204468 seconds
> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>
>
>
> After some discussion between the teams that work on ClamAV and ClamAV
> signature content and deployment, we’ve agreed to drop PhishTank signatures
> from the database until we can determine a way to craft Phishtank
> signatures without incurring such a significant performance hit.
>
>
>
> The daily update tomorrow will have the change.
>
>
>
> -Micah
>
>
>
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.
>
>
>
>
>
>
>
> *From: *clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of
> "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net>
> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net>
> *Date: *Friday, April 5, 2019 at 1:08 PM
> *To: *Mark Allan <markjallan@gmail.com>, ClamAV users ML <
> clamav-users@lists.clamav.net>
> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
> *Subject: *Re: [clamav-users] Scan very slow
>
>
>
> Hi Mark,
>
>
>
> Sorry about the delay in responding. I hadn’t looked at my clamav-users
> filter this morning. Just investigating now. Will respond when I know
> more.
>
>
>
> -Micah
>
>
>
> *From: *Mark Allan <markjallan@gmail.com>
> *Date: *Friday, April 5, 2019 at 9:12 AM
> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder
> (micasnyd)" <micasnyd@cisco.com>
> *Subject: *Re: [clamav-users] Scan very slow
>
>
>
> Also CC'ing Micah directly as the mailing list would appear to be offline
> (at least lists.clamav.net isn't responding to http requests anyway)
>
>
>
> It looks like scan times have gone through the roof. As Oya said, they're
> still considerably higher than they were a couple of months ago, but
> today's scan time is insane.
>
>
>
> Yesterday's scan using
>
> 0.101.2:58:25409:1554370140:1:63:48554:328
>
> took 7m 3s
>
>
>
> On the same hardware, scanning the same read-only disk image, with today's
> scan using
>
> 0.101.2:58:25410:1554452941:1:63:48557:328
>
> the scan time has jumped to 26m 15s
>
>
>
> This is the longest it has ever taken to scan this volume (cf my previous
> email of 25th March)
>
>
>
> Is there anything that can be excluded?
>
>
>
> Best regards
>
> Mark
>
>
>
> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
> clamav-users@lists.clamav.net> wrote:
>
> Thanks Oya for the update. We will continue to investigate the signature
> performance issue.
>
> Regards,
> Micah
>
> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
> clamav-users-bounces@lists.clamav.net on behalf of oyamada@promark-inc.com>
> wrote:
>
> Hi Micah
>
> It seems that the scanning slow down issue of this time has been
> solved
> at some level with CVD Update of the other day.
> However, there is still big discrepancy in between the current
> condition and
> the last condition in one month ago.
>
> Date Files Scan time
> 2019/02/15 2550338 08:53:57
> 2019/03/15 2612792 19:22:54
> 2019/03/26 2634489 18:13:56
> 2019/03/27 2637201 18:10:05
>
> We know the improvement of this time is due to the details of CVD,
> because
> we did not make any change on the user's system.
> We are going to try some tuning for scanning.
>
> We like to know if you still have some room to make further improvement
> for this slow down issue.
> Thank you for your help, in advance.
>
> Best regards,
> Oya
>
> On Mon, 25 Mar 2019 15:45:02 +0000
> "Micah Snyder \(micasnyd\) via clamav-users" <
> clamav-users@lists.clamav.net> wrote:
>
> > Hi Mark, all:
> >
> > I’m disappointed to hear that it is still slow for you.
> >
> > We found that the target-type of signatures used for
> PhishTank.Phishing signatures were causing a significant slowdown. We
> have dropped them as of this past Saturday (
> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
> have been re-adding them with more specific scan target types. We’re now
> investigating some other optimizations we can make for the next major
> ClamAV release to improve scan times but at present we don’t have any other
> leads for signatures that may be slowing down scans.
> >
> > Regards,
> > Micah
> >
> >
> > From: clamav-users <clamav-users-bounces@lists.clamav.net> on
> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
> > Date: Monday, March 25, 2019 at 9:37 AM
> > To: ClamAV users ML <clamav-users@lists.clamav.net>
> > Cc: Mark Allan <markjallan@gmail.com>
> > Subject: Re: [clamav-users] Scan very slow
> >
> > Cheers Steve,
> >
> > In the interest of completeness, here's the scan from today (TXT
> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
> improvement in scan time, although at 6m 7s it's still almost twice what it
> used to be.
> >
> > Mark
> >
> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
> steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com>>
> wrote:
> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > > Hi all,
> > >
> > te.
> > >
> > > Hopefully this helps someone to narrow things down a bit.
> > >
> > > Mark
> > >
> >
> > 18/3/19 10m 49s TXT from DNS:
> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
> >
> > Here's the changes for the above update:
> >
> > https://lists.gt.net/clamav/virusdb/75154
> >
> > You can also check sigs quickly per update:
> >
> > https://lists.gt.net/clamav/virusdb/
> >
> >
> >
> > --
> > Cheers,
> >
> > Steve
> > Twitter: @sanesecurity
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> *DISCLAIMER*
>
> The information contained in this email and any attachments are
> confidential. It is intended solely for the individual or entity to whom
> they are addressed. Access to this email by anyone else is unauthorized.
>
> If you are not the intended recipient, any disclosure, copying,
> distribution or any action taken or omitted to be taken in reliance on it,
> is prohibited and may be unlawful. If you have received this communication
> in error, please notify us immediately by responding to this email and then
> delete it from your system.
>
> The Red Flag Group is neither liable for the proper and complete
> transmission of the information contained in this communication nor for any
> delay in its receipt.
>
> Any advice, recommendations or opinion contained within this email or its
> attachments are not to be construed as legal advice.
>
>

An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more.

-Al-

> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Hi Micah,
>
> Sorry to pester you, but have you any update on when the remaining Phishtank signatures will be getting removed? It would be really great to get scan times properly back to normal.
>
> Best regards
> Mark
>
> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
> Mark,
>
>
> Yes, the plan is still to remove the rest of the Phishtank signatures. We wanted to get things back to relative normal and resolve the immediate crisis. We’ll remove the rest of them soon.
>
>
>
> Best,
>
> Micah
>
>
>
> From: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
> Date: Tuesday, April 9, 2019 at 6:26 AM
> To: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
> Cc: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> Subject: Re: [External] Re: [clamav-users] Scan very slow
>
>
>
> The scan times are definitely better than they were - in fact, they're back to how they were before last week's inclusion of the Phishtank signatures. They're still almost double what they used to be though, and as far as I can see, there are still almost 4000 Phishtank signatures in the DB:
>
> $ sigtool --find Phishtank | wc -l
>
> 3968
>
>
>
> Can I request that those ones also be removed please?
>
>
>
> Best regards
>
> Mark
>
>
>
> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
>
> Tim,
>
>
>
> There are a couple of ways for users to drop specific categories of signatures at this time. Sadly, they wouldn’t have helped this last week. These include bytecode signatures, PUA (potentially unwanted applications) signatures, Email.Phishing and HTML.Phishing signatures, and the Safebrowsing database.
>
>
>
> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or Email.Phishing.Phishtank then they could have been disabled with the clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>
>
>
> Maybe a better option would be for us to create a new optional database for phishing signatures. However, the names for the databases are hardcoded into freshclam, so it is non-trivial to add a new database and would require a few changes to ClamAV’s code. We have talked about making the databases easier to add/remove in the future so users can have more categories to enable/disable. In this light, it ties in well with existing plans.
>
>
>
> Of note the Phishtank sigs from Friday’s daily were removed yesterday and scan times should be back to normal.
>
>
>
> Regards,
>
> Micah
>
>
>
> From: Tim Hawkins <tim.hawkins@redflaggroup.com <mailto:tim.hawkins@redflaggroup.com>>
> Date: Friday, April 5, 2019 at 6:06 PM
> To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>, Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
> Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
> Subject: Re: [External] Re: [clamav-users] Scan very slow
>
>
>
> Hi Micah
>
>
> Does clamav partition the database so that signatures that are mainly associated with email scanning can be dropped out for folks only needing filesystems scans, none of our systems use email, and we dont make use of the mailer extension.
>
> Having to load all the email focused signatures could as you have observed impact performance.
>
> Sent from Nine <http://www.9folders.com/>
> From: "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> Sent: Saturday, April 6, 2019 03:18
> To: ClamAV users ML; Mark Allan
> Cc: Micah Snyder (micasnyd)
> Subject: [External] Re: [clamav-users] Scan very slow
>
>
>
> Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times.
>
>
>
> Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time. One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317 <https://bugzilla.clamav.net/show_bug.cgi?id=12317>).
>
>
>
> In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>
> Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds
> Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>
>
> After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit.
>
>
>
> The daily update tomorrow will have the change.
>
>
>
> -Micah
>
>
>
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.
>
>
>
>
>
>
>
> From: clamav-users <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net>> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> Date: Friday, April 5, 2019 at 1:08 PM
> To: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>, ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
> Subject: Re: [clamav-users] Scan very slow
>
>
>
> Hi Mark,
>
>
>
> Sorry about the delay in responding. I hadn’t looked at my clamav-users filter this morning. Just investigating now. Will respond when I know more.
>
>
>
> -Micah
>
>
>
> From: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
> Date: Friday, April 5, 2019 at 9:12 AM
> To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>, "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
> Subject: Re: [clamav-users] Scan very slow
>
>
>
> Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net <http://lists.clamav.net/> isn't responding to http requests anyway)
>
>
>
> It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane.
>
>
>
> Yesterday's scan using
>
> 0.101.2:58:25409:1554370140:1:63:48554:328
>
> took 7m 3s
>
>
>
> On the same hardware, scanning the same read-only disk image, with today's scan using
>
> 0.101.2:58:25410:1554452941:1:63:48557:328
>
> the scan time has jumped to 26m 15s
>
>
>
> This is the longest it has ever taken to scan this volume (cf my previous email of 25th March)
>
>
>
> Is there anything that can be excluded?
>
>
>
> Best regards
>
> Mark
>
>
>
> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>
> Thanks Oya for the update. We will continue to investigate the signature performance issue.
>
> Regards,
> Micah
>
> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net> on behalf of oyamada@promark-inc.com <mailto:oyamada@promark-inc.com>> wrote:
>
> Hi Micah
>
> It seems that the scanning slow down issue of this time has been solved
> at some level with CVD Update of the other day.
> However, there is still big discrepancy in between the current condition and
> the last condition in one month ago.
>
> Date Files Scan time
> 2019/02/15 2550338 08:53:57
> 2019/03/15 2612792 19:22:54
> 2019/03/26 2634489 18:13:56
> 2019/03/27 2637201 18:10:05
>
> We know the improvement of this time is due to the details of CVD, because
> we did not make any change on the user's system.
> We are going to try some tuning for scanning.
>
> We like to know if you still have some room to make further improvement
> for this slow down issue.
> Thank you for your help, in advance.
>
> Best regards,
> Oya
>
> On Mon, 25 Mar 2019 15:45:02 +0000
> "Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>
> > Hi Mark, all:
> >
> > I’m disappointed to hear that it is still slow for you.
> >
> > We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown. We have dropped them as of this past Saturday (https://lists.gt.net/clamav/virusdb/75279 <https://lists.gt.net/clamav/virusdb/75279> ) and in the last two updates have been re-adding them with more specific scan target types. We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans.
> >
> > Regards,
> > Micah
> >
> >
> > From: clamav-users <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net>> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> > Date: Monday, March 25, 2019 at 9:37 AM
> > To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> > Cc: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
> > Subject: Re: [clamav-users] Scan very slow
> >
> > Cheers Steve,
> >
> > In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be.
> >
> > Mark
> >
> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_clamav@sanesecurity.com <mailto:steveb_clamav@sanesecurity.com><mailto:steveb_clamav@sanesecurity.com <mailto:steveb_clamav@sanesecurity.com>>> wrote:
> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > > Hi all,
> > >
> > te.
> > >
> > > Hopefully this helps someone to narrow things down a bit.
> > >
> > > Mark
> > >
> >
> > 18/3/19 10m 49s TXT from DNS:
> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
> >
> > Here's the changes for the above update:
> >
> > https://lists.gt.net/clamav/virusdb/75154 <https://lists.gt.net/clamav/virusdb/75154>
> >
> > You can also check sigs quickly per update:
> >
> > https://lists.gt.net/clamav/virusdb/ <https://lists.gt.net/clamav/virusdb/>
> >
> >
> >
> > --
> > Cheers,
> >
> > Steve
> > Twitter: @sanesecurity
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
> > https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
> >
> > http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>
> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>
> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>
>
> DISCLAIMER
>
> The information contained in this email and any attachments are confidential. It is intended solely for the individual or entity to whom they are addressed. Access to this email by anyone else is unauthorized.
>
> If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system.
>
> The Red Flag Group is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
>
> Any advice, recommendations or opinion contained within this email or its attachments are not to be construed as legal advice.
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>
> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>

Are the "Phish" REPHISH signatures still in the daily or were they removed
as well? Those were causing part of the issue.


--Maarten

On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <
clamav-users@lists.clamav.net> wrote:

> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were
> dropped by daily-25417 on 12 April, and I can't seem to locate any more.
>
> -Al-
>
> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <
> clamav-users@lists.clamav.net> wrote:
>
> Hi Micah,
>
> Sorry to pester you, but have you any update on when the remaining
> Phishtank signatures will be getting removed? It would be really great to
> get scan times properly back to normal.
>
> Best regards
> Mark
>
> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com>
> wrote:
>
>> Mark,
>>
>>
>> Yes, the plan is still to remove the rest of the Phishtank signatures.
>> We wanted to get things back to relative normal and resolve the immediate
>> crisis. We’ll remove the rest of them soon.
>>
>>
>>
>> Best,
>>
>> Micah
>>
>>
>>
>> *From: *Mark Allan <markjallan@gmail.com>
>> *Date: *Tuesday, April 9, 2019 at 6:26 AM
>> *To: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net>
>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> The scan times are definitely better than they were - in fact, they're
>> back to how they were before last week's inclusion of the Phishtank
>> signatures. They're still almost double what they used to be though, and as
>> far as I can see, there are still almost 4000 Phishtank signatures in the
>> DB:
>>
>> $ sigtool --find Phishtank | wc -l
>>
>> 3968
>>
>>
>>
>> Can I request that those ones also be removed please?
>>
>>
>>
>> Best regards
>>
>> Mark
>>
>>
>>
>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com>
>> wrote:
>>
>> Tim,
>>
>>
>>
>> There are a couple of ways for users to drop specific categories of
>> signatures at this time. Sadly, they wouldn’t have helped this last week.
>> These include bytecode signatures, PUA (potentially unwanted applications)
>> signatures, Email.Phishing and HTML.Phishing signatures, and the
>> Safebrowsing database.
>>
>>
>>
>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or
>> Email.Phishing.Phishtank then they could have been disabled with the
>> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>>
>>
>>
>> Maybe a better option would be for us to create a new optional database
>> for phishing signatures. However, the names for the databases are hardcoded
>> into freshclam, so it is non-trivial to add a new database and would
>> require a few changes to ClamAV’s code. We have talked about making the
>> databases easier to add/remove in the future so users can have more
>> categories to enable/disable. In this light, it ties in well with existing
>> plans.
>>
>>
>>
>> Of note the Phishtank sigs from Friday’s daily were removed yesterday and
>> scan times should be back to normal.
>>
>>
>>
>> Regards,
>>
>> Micah
>>
>>
>>
>> *From: *Tim Hawkins <tim.hawkins@redflaggroup.com>
>> *Date: *Friday, April 5, 2019 at 6:06 PM
>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan <
>> markjallan@gmail.com>
>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> Hi Micah
>>
>>
>> Does clamav partition the database so that signatures that are mainly
>> associated with email scanning can be dropped out for folks only needing
>> filesystems scans, none of our systems use email, and we dont make use of
>> the mailer extension.
>>
>> Having to load all the email focused signatures could as you have
>> observed impact performance.
>>
>> Sent from Nine <http://www.9folders.com/>
>> ------------------------------
>>
>> *From:* "Micah Snyder (micasnyd) via clamav-users" <
>> clamav-users@lists.clamav.net>
>> *Sent:* Saturday, April 6, 2019 03:18
>> *To:* ClamAV users ML; Mark Allan
>> *Cc:* Micah Snyder (micasnyd)
>> *Subject:* [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> Regarding slow scan times today (and slow scan times in general), it
>> appears that the signatures we generate based on PhishTank’s feed for
>> phishing URLs are resulting in very slow load and scan times.
>>
>>
>>
>> Today’s daily update saw 7448 new Phishtank signatures (much higher than
>> usual) coinciding with the immediate performance drop for load time and
>> scan time. One user reported that the load time today on some of his
>> slower machines was slow enough to exceed the timeout for service startup (
>> https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>>
>>
>>
>> In limited testing on my own machine I saw the following change after
>> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>>
>> - Database load time on my laptop went from 75.43203997612 seconds
>> down to 14.859203100204468 seconds
>> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>>
>>
>>
>> After some discussion between the teams that work on ClamAV and ClamAV
>> signature content and deployment, we’ve agreed to drop PhishTank signatures
>> from the database until we can determine a way to craft Phishtank
>> signatures without incurring such a significant performance hit.
>>
>>
>>
>> The daily update tomorrow will have the change.
>>
>>
>>
>> -Micah
>>
>>
>>
>>
>> Micah Snyder
>> ClamAV Development
>> Talos
>> Cisco Systems, Inc.
>>
>>
>>
>>
>>
>>
>>
>> *From: *clamav-users <clamav-users-bounces@lists.clamav.net> on behalf
>> of "Micah Snyder (micasnyd) via clamav-users" <
>> clamav-users@lists.clamav.net>
>> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net>
>> *Date: *Friday, April 5, 2019 at 1:08 PM
>> *To: *Mark Allan <markjallan@gmail.com>, ClamAV users ML <
>> clamav-users@lists.clamav.net>
>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>> *Subject: *Re: [clamav-users] Scan very slow
>>
>>
>>
>> Hi Mark,
>>
>>
>>
>> Sorry about the delay in responding. I hadn’t looked at my clamav-users
>> filter this morning. Just investigating now. Will respond when I know
>> more.
>>
>>
>>
>> -Micah
>>
>>
>>
>> *From: *Mark Allan <markjallan@gmail.com>
>> *Date: *Friday, April 5, 2019 at 9:12 AM
>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder
>> (micasnyd)" <micasnyd@cisco.com>
>> *Subject: *Re: [clamav-users] Scan very slow
>>
>>
>>
>> Also CC'ing Micah directly as the mailing list would appear to be offline
>> (at least lists.clamav.net isn't responding to http requests anyway)
>>
>>
>>
>> It looks like scan times have gone through the roof. As Oya said, they're
>> still considerably higher than they were a couple of months ago, but
>> today's scan time is insane.
>>
>>
>>
>> Yesterday's scan using
>>
>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>
>> took 7m 3s
>>
>>
>>
>> On the same hardware, scanning the same read-only disk image, with
>> today's scan using
>>
>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>
>> the scan time has jumped to 26m 15s
>>
>>
>>
>> This is the longest it has ever taken to scan this volume (cf my previous
>> email of 25th March)
>>
>>
>>
>> Is there anything that can be excluded?
>>
>>
>>
>> Best regards
>>
>> Mark
>>
>>
>>
>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
>> clamav-users@lists.clamav.net> wrote:
>>
>> Thanks Oya for the update. We will continue to investigate the signature
>> performance issue.
>>
>> Regards,
>> Micah
>>
>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
>> clamav-users-bounces@lists.clamav.net on behalf of
>> oyamada@promark-inc.com> wrote:
>>
>> Hi Micah
>>
>> It seems that the scanning slow down issue of this time has been
>> solved
>> at some level with CVD Update of the other day.
>> However, there is still big discrepancy in between the current
>> condition and
>> the last condition in one month ago.
>>
>> Date Files Scan time
>> 2019/02/15 2550338 08:53:57
>> 2019/03/15 2612792 19:22:54
>> 2019/03/26 2634489 18:13:56
>> 2019/03/27 2637201 18:10:05
>>
>> We know the improvement of this time is due to the details of CVD,
>> because
>> we did not make any change on the user's system.
>> We are going to try some tuning for scanning.
>>
>> We like to know if you still have some room to make further
>> improvement
>> for this slow down issue.
>> Thank you for your help, in advance.
>>
>> Best regards,
>> Oya
>>
>> On Mon, 25 Mar 2019 15:45:02 +0000
>> "Micah Snyder \(micasnyd\) via clamav-users" <
>> clamav-users@lists.clamav.net> wrote:
>>
>> > Hi Mark, all:
>> >
>> > I’m disappointed to hear that it is still slow for you.
>> >
>> > We found that the target-type of signatures used for
>> PhishTank.Phishing signatures were causing a significant slowdown. We
>> have dropped them as of this past Saturday (
>> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
>> have been re-adding them with more specific scan target types. We’re now
>> investigating some other optimizations we can make for the next major
>> ClamAV release to improve scan times but at present we don’t have any other
>> leads for signatures that may be slowing down scans.
>> >
>> > Regards,
>> > Micah
>> >
>> >
>> > From: clamav-users <clamav-users-bounces@lists.clamav.net> on
>> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
>> > Date: Monday, March 25, 2019 at 9:37 AM
>> > To: ClamAV users ML <clamav-users@lists.clamav.net>
>> > Cc: Mark Allan <markjallan@gmail.com>
>> > Subject: Re: [clamav-users] Scan very slow
>> >
>> > Cheers Steve,
>> >
>> > In the interest of completeness, here's the scan from today (TXT
>> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
>> improvement in scan time, although at 6m 7s it's still almost twice what it
>> used to be.
>> >
>> > Mark
>> >
>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
>> steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com>>
>> wrote:
>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
>> > > Hi all,
>> > >
>> > te.
>> > >
>> > > Hopefully this helps someone to narrow things down a bit.
>> > >
>> > > Mark
>> > >
>> >
>> > 18/3/19 10m 49s TXT from DNS:
>> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>> >
>> > Here's the changes for the above update:
>> >
>> > https://lists.gt.net/clamav/virusdb/75154
>> >
>> > You can also check sigs quickly per update:
>> >
>> > https://lists.gt.net/clamav/virusdb/
>> >
>> >
>> >
>> > --
>> > Cheers,
>> >
>> > Steve
>> > Twitter: @sanesecurity
>> >
>> > _______________________________________________
>> >
>> > clamav-users mailing list
>> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
>> > https://lists.clamav.net/mailman/listinfo/clamav-users
>> >
>> >
>> > Help us build a comprehensive ClamAV guide:
>> > https://github.com/vrtadmin/clamav-faq
>> >
>> > http://www.clamav.net/contact.html#ml
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>> *DISCLAIMER*
>>
>> The information contained in this email and any attachments are
>> confidential. It is intended solely for the individual or entity to whom
>> they are addressed. Access to this email by anyone else is unauthorized.
>>
>> If you are not the intended recipient, any disclosure, copying,
>> distribution or any action taken or omitted to be taken in reliance on it,
>> is prohibited and may be unlawful. If you have received this communication
>> in error, please notify us immediately by responding to this email and then
>> delete it from your system.
>>
>> The Red Flag Group is neither liable for the proper and complete
>> transmission of the information contained in this communication nor for any
>> delay in its receipt.
>>
>> Any advice, recommendations or opinion contained within this email or its
>> attachments are not to be construed as legal advice.
>>
>>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in daily.ldb

-Al-

> On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broekman@gmail.com <mailto:maarten.broekman@gmail.com>> wrote:
>
> Are the "Phish" REPHISH signatures still in the daily or were they removed as well? Those were causing part of the issue.
>
>
> --Maarten
>
> On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more.
>
> -Al-
>
>> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>
>> Hi Micah,
>>
>> Sorry to pester you, but have you any update on when the remaining Phishtank signatures will be getting removed? It would be really great to get scan times properly back to normal.
>>
>> Best regards
>> Mark
>>
>> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
>> Mark,
>>
>>
>> Yes, the plan is still to remove the rest of the Phishtank signatures. We wanted to get things back to relative normal and resolve the immediate crisis. We’ll remove the rest of them soon.
>>
>>
>>
>> Best,
>>
>> Micah
>>
>>
>>
>> From: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>> Date: Tuesday, April 9, 2019 at 6:26 AM
>> To: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>> Cc: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> Subject: Re: [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> The scan times are definitely better than they were - in fact, they're back to how they were before last week's inclusion of the Phishtank signatures. They're still almost double what they used to be though, and as far as I can see, there are still almost 4000 Phishtank signatures in the DB:
>>
>> $ sigtool --find Phishtank | wc -l
>>
>> 3968
>>
>>
>>
>> Can I request that those ones also be removed please?
>>
>>
>>
>> Best regards
>>
>> Mark
>>
>>
>>
>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
>>
>> Tim,
>>
>>
>>
>> There are a couple of ways for users to drop specific categories of signatures at this time. Sadly, they wouldn’t have helped this last week. These include bytecode signatures, PUA (potentially unwanted applications) signatures, Email.Phishing and HTML.Phishing signatures, and the Safebrowsing database.
>>
>>
>>
>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or Email.Phishing.Phishtank then they could have been disabled with the clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>>
>>
>>
>> Maybe a better option would be for us to create a new optional database for phishing signatures. However, the names for the databases are hardcoded into freshclam, so it is non-trivial to add a new database and would require a few changes to ClamAV’s code. We have talked about making the databases easier to add/remove in the future so users can have more categories to enable/disable. In this light, it ties in well with existing plans.
>>
>>
>>
>> Of note the Phishtank sigs from Friday’s daily were removed yesterday and scan times should be back to normal.
>>
>>
>>
>> Regards,
>>
>> Micah
>>
>>
>>
>> From: Tim Hawkins <tim.hawkins@redflaggroup.com <mailto:tim.hawkins@redflaggroup.com>>
>> Date: Friday, April 5, 2019 at 6:06 PM
>> To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>, Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>> Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>> Subject: Re: [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> Hi Micah
>>
>>
>> Does clamav partition the database so that signatures that are mainly associated with email scanning can be dropped out for folks only needing filesystems scans, none of our systems use email, and we dont make use of the mailer extension.
>>
>> Having to load all the email focused signatures could as you have observed impact performance.
>>
>> Sent from Nine <http://www.9folders.com/>
>> From: "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> Sent: Saturday, April 6, 2019 03:18
>> To: ClamAV users ML; Mark Allan
>> Cc: Micah Snyder (micasnyd)
>> Subject: [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times.
>>
>>
>>
>> Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time. One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317 <https://bugzilla.clamav.net/show_bug.cgi?id=12317>).
>>
>>
>>
>> In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>>
>> Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds
>> Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>>
>>
>> After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit.
>>
>>
>>
>> The daily update tomorrow will have the change.
>>
>>
>>
>> -Micah
>>
>>
>>
>>
>> Micah Snyder
>> ClamAV Development
>> Talos
>> Cisco Systems, Inc.
>>
>>
>>
>>
>>
>>
>>
>> From: clamav-users <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net>> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> Date: Friday, April 5, 2019 at 1:08 PM
>> To: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>, ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>> Subject: Re: [clamav-users] Scan very slow
>>
>>
>>
>> Hi Mark,
>>
>>
>>
>> Sorry about the delay in responding. I hadn’t looked at my clamav-users filter this morning. Just investigating now. Will respond when I know more.
>>
>>
>>
>> -Micah
>>
>>
>>
>> From: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>> Date: Friday, April 5, 2019 at 9:12 AM
>> To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>, "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>> Subject: Re: [clamav-users] Scan very slow
>>
>>
>>
>> Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net <http://lists.clamav.net/> isn't responding to http requests anyway)
>>
>>
>>
>> It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane.
>>
>>
>>
>> Yesterday's scan using
>>
>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>
>> took 7m 3s
>>
>>
>>
>> On the same hardware, scanning the same read-only disk image, with today's scan using
>>
>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>
>> the scan time has jumped to 26m 15s
>>
>>
>>
>> This is the longest it has ever taken to scan this volume (cf my previous email of 25th March)
>>
>>
>>
>> Is there anything that can be excluded?
>>
>>
>>
>> Best regards
>>
>> Mark
>>
>>
>>
>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>
>> Thanks Oya for the update. We will continue to investigate the signature performance issue.
>>
>> Regards,
>> Micah
>>
>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net> on behalf of oyamada@promark-inc.com <mailto:oyamada@promark-inc.com>> wrote:
>>
>> Hi Micah
>>
>> It seems that the scanning slow down issue of this time has been solved
>> at some level with CVD Update of the other day.
>> However, there is still big discrepancy in between the current condition and
>> the last condition in one month ago.
>>
>> Date Files Scan time
>> 2019/02/15 2550338 08:53:57
>> 2019/03/15 2612792 19:22:54
>> 2019/03/26 2634489 18:13:56
>> 2019/03/27 2637201 18:10:05
>>
>> We know the improvement of this time is due to the details of CVD, because
>> we did not make any change on the user's system.
>> We are going to try some tuning for scanning.
>>
>> We like to know if you still have some room to make further improvement
>> for this slow down issue.
>> Thank you for your help, in advance.
>>
>> Best regards,
>> Oya
>>
>> On Mon, 25 Mar 2019 15:45:02 +0000
>> "Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>
>> > Hi Mark, all:
>> >
>> > I’m disappointed to hear that it is still slow for you.
>> >
>> > We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown. We have dropped them as of this past Saturday (https://lists.gt.net/clamav/virusdb/75279 <https://lists.gt.net/clamav/virusdb/75279> ) and in the last two updates have been re-adding them with more specific scan target types. We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans.
>> >
>> > Regards,
>> > Micah
>> >
>> >
>> > From: clamav-users <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net>> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> > Date: Monday, March 25, 2019 at 9:37 AM
>> > To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> > Cc: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>> > Subject: Re: [clamav-users] Scan very slow
>> >
>> > Cheers Steve,
>> >
>> > In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be.
>> >
>> > Mark
>> >
>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_clamav@sanesecurity.com <mailto:steveb_clamav@sanesecurity.com><mailto:steveb_clamav@sanesecurity.com <mailto:steveb_clamav@sanesecurity.com>>> wrote:
>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
>> > > Hi all,
>> > >
>> > te.
>> > >
>> > > Hopefully this helps someone to narrow things down a bit.
>> > >
>> > > Mark
>> > >
>> >
>> > 18/3/19 10m 49s TXT from DNS:
>> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>> >
>> > Here's the changes for the above update:
>> >
>> > https://lists.gt.net/clamav/virusdb/75154 <https://lists.gt.net/clamav/virusdb/75154>
>> >
>> > You can also check sigs quickly per update:
>> >
>> > https://lists.gt.net/clamav/virusdb/ <https://lists.gt.net/clamav/virusdb/>
>> >
>> >
>> >
>> > --
>> > Cheers,
>> >
>> > Steve
>> > Twitter: @sanesecurity
>> >
>> > _______________________________________________
>> >
>> > clamav-users mailing list
>> > clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>> > https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>> >
>> >
>> > Help us build a comprehensive ClamAV guide:
>> > https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>> >
>> > http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>
>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>
>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>
>>
>> DISCLAIMER
>>
>> The information contained in this email and any attachments are confidential. It is intended solely for the individual or entity to whom they are addressed. Access to this email by anyone else is unauthorized.
>>
>> If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system.
>>
>> The Red Flag Group is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
>>
>> Any advice, recommendations or opinion contained within this email or its attachments are not to be construed as legal advice.
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>
>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>
> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>
>

Gotcha. Those were slowing the scans down more than the 3000-someodd
PhishTank sigs the last time I tested (Apr 9th).

daily_Phish.ldb ==== Time: 1.612 sec (0 m 1 s)
daily_Phishtank.ldb ==== Time: 0.146 sec (0 m 0 s)

2515 daily_Phish.ldb
3516 daily_Phishtank.ldb


On Wed, Apr 17, 2019 at 7:03 AM Al Varnell via clamav-users <
clamav-users@lists.clamav.net> wrote:

> There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in
> daily.ldb
>
> -Al-
>
> On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broekman@gmail.com>
> wrote:
>
> Are the "Phish" REPHISH signatures still in the daily or were they removed
> as well? Those were causing part of the issue.
>
>
> --Maarten
>
> On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <
> clamav-users@lists.clamav.net> wrote:
>
>> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were
>> dropped by daily-25417 on 12 April, and I can't seem to locate any more.
>>
>> -Al-
>>
>> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <
>> clamav-users@lists.clamav.net> wrote:
>>
>> Hi Micah,
>>
>> Sorry to pester you, but have you any update on when the remaining
>> Phishtank signatures will be getting removed? It would be really great to
>> get scan times properly back to normal.
>>
>> Best regards
>> Mark
>>
>> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com>
>> wrote:
>>
>>> Mark,
>>>
>>>
>>> Yes, the plan is still to remove the rest of the Phishtank signatures.
>>> We wanted to get things back to relative normal and resolve the immediate
>>> crisis. We’ll remove the rest of them soon.
>>>
>>>
>>>
>>> Best,
>>>
>>> Micah
>>>
>>>
>>>
>>> *From: *Mark Allan <markjallan@gmail.com>
>>> *Date: *Tuesday, April 9, 2019 at 6:26 AM
>>> *To: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>>> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net>
>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> The scan times are definitely better than they were - in fact, they're
>>> back to how they were before last week's inclusion of the Phishtank
>>> signatures. They're still almost double what they used to be though, and as
>>> far as I can see, there are still almost 4000 Phishtank signatures in the
>>> DB:
>>>
>>> $ sigtool --find Phishtank | wc -l
>>>
>>> 3968
>>>
>>>
>>>
>>> Can I request that those ones also be removed please?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com>
>>> wrote:
>>>
>>> Tim,
>>>
>>>
>>>
>>> There are a couple of ways for users to drop specific categories of
>>> signatures at this time. Sadly, they wouldn’t have helped this last week.
>>> These include bytecode signatures, PUA (potentially unwanted applications)
>>> signatures, Email.Phishing and HTML.Phishing signatures, and the
>>> Safebrowsing database.
>>>
>>>
>>>
>>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank
>>> or Email.Phishing.Phishtank then they could have been disabled with the
>>> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>>>
>>>
>>>
>>> Maybe a better option would be for us to create a new optional database
>>> for phishing signatures. However, the names for the databases are hardcoded
>>> into freshclam, so it is non-trivial to add a new database and would
>>> require a few changes to ClamAV’s code. We have talked about making the
>>> databases easier to add/remove in the future so users can have more
>>> categories to enable/disable. In this light, it ties in well with existing
>>> plans.
>>>
>>>
>>>
>>> Of note the Phishtank sigs from Friday’s daily were removed yesterday
>>> and scan times should be back to normal.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Micah
>>>
>>>
>>>
>>> *From: *Tim Hawkins <tim.hawkins@redflaggroup.com>
>>> *Date: *Friday, April 5, 2019 at 6:06 PM
>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan <
>>> markjallan@gmail.com>
>>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Micah
>>>
>>>
>>> Does clamav partition the database so that signatures that are mainly
>>> associated with email scanning can be dropped out for folks only needing
>>> filesystems scans, none of our systems use email, and we dont make use of
>>> the mailer extension.
>>>
>>> Having to load all the email focused signatures could as you have
>>> observed impact performance.
>>>
>>> Sent from Nine <http://www.9folders.com/>
>>> ------------------------------
>>>
>>> *From:* "Micah Snyder (micasnyd) via clamav-users" <
>>> clamav-users@lists.clamav.net>
>>> *Sent:* Saturday, April 6, 2019 03:18
>>> *To:* ClamAV users ML; Mark Allan
>>> *Cc:* Micah Snyder (micasnyd)
>>> *Subject:* [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Regarding slow scan times today (and slow scan times in general), it
>>> appears that the signatures we generate based on PhishTank’s feed for
>>> phishing URLs are resulting in very slow load and scan times.
>>>
>>>
>>>
>>> Today’s daily update saw 7448 new Phishtank signatures (much higher than
>>> usual) coinciding with the immediate performance drop for load time and
>>> scan time. One user reported that the load time today on some of his
>>> slower machines was slow enough to exceed the timeout for service startup (
>>> https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>>>
>>>
>>>
>>> In limited testing on my own machine I saw the following change after
>>> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>>>
>>> - Database load time on my laptop went from 75.43203997612 seconds
>>> down to 14.859203100204468 seconds
>>> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>>>
>>>
>>>
>>> After some discussion between the teams that work on ClamAV and ClamAV
>>> signature content and deployment, we’ve agreed to drop PhishTank signatures
>>> from the database until we can determine a way to craft Phishtank
>>> signatures without incurring such a significant performance hit.
>>>
>>>
>>>
>>> The daily update tomorrow will have the change.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>>
>>> Micah Snyder
>>> ClamAV Development
>>> Talos
>>> Cisco Systems, Inc.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From: *clamav-users <clamav-users-bounces@lists.clamav.net> on behalf
>>> of "Micah Snyder (micasnyd) via clamav-users" <
>>> clamav-users@lists.clamav.net>
>>> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net>
>>> *Date: *Friday, April 5, 2019 at 1:08 PM
>>> *To: *Mark Allan <markjallan@gmail.com>, ClamAV users ML <
>>> clamav-users@lists.clamav.net>
>>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>>> *Subject: *Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Mark,
>>>
>>>
>>>
>>> Sorry about the delay in responding. I hadn’t looked at my clamav-users
>>> filter this morning. Just investigating now. Will respond when I know
>>> more.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>> *From: *Mark Allan <markjallan@gmail.com>
>>> *Date: *Friday, April 5, 2019 at 9:12 AM
>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder
>>> (micasnyd)" <micasnyd@cisco.com>
>>> *Subject: *Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Also CC'ing Micah directly as the mailing list would appear to be
>>> offline (at least lists.clamav.net isn't responding to http requests
>>> anyway)
>>>
>>>
>>>
>>> It looks like scan times have gone through the roof. As Oya said,
>>> they're still considerably higher than they were a couple of months ago,
>>> but today's scan time is insane.
>>>
>>>
>>>
>>> Yesterday's scan using
>>>
>>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>>
>>> took 7m 3s
>>>
>>>
>>>
>>> On the same hardware, scanning the same read-only disk image, with
>>> today's scan using
>>>
>>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>>
>>> the scan time has jumped to 26m 15s
>>>
>>>
>>>
>>> This is the longest it has ever taken to scan this volume (cf my
>>> previous email of 25th March)
>>>
>>>
>>>
>>> Is there anything that can be excluded?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
>>> clamav-users@lists.clamav.net> wrote:
>>>
>>> Thanks Oya for the update. We will continue to investigate the
>>> signature performance issue.
>>>
>>> Regards,
>>> Micah
>>>
>>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
>>> clamav-users-bounces@lists.clamav.net on behalf of
>>> oyamada@promark-inc.com> wrote:
>>>
>>> Hi Micah
>>>
>>> It seems that the scanning slow down issue of this time has been
>>> solved
>>> at some level with CVD Update of the other day.
>>> However, there is still big discrepancy in between the current
>>> condition and
>>> the last condition in one month ago.
>>>
>>> Date Files Scan time
>>> 2019/02/15 2550338 08:53:57
>>> 2019/03/15 2612792 19:22:54
>>> 2019/03/26 2634489 18:13:56
>>> 2019/03/27 2637201 18:10:05
>>>
>>> We know the improvement of this time is due to the details of CVD,
>>> because
>>> we did not make any change on the user's system.
>>> We are going to try some tuning for scanning.
>>>
>>> We like to know if you still have some room to make further
>>> improvement
>>> for this slow down issue.
>>> Thank you for your help, in advance.
>>>
>>> Best regards,
>>> Oya
>>>
>>> On Mon, 25 Mar 2019 15:45:02 +0000
>>> "Micah Snyder \(micasnyd\) via clamav-users" <
>>> clamav-users@lists.clamav.net> wrote:
>>>
>>> > Hi Mark, all:
>>> >
>>> > I’m disappointed to hear that it is still slow for you.
>>> >
>>> > We found that the target-type of signatures used for
>>> PhishTank.Phishing signatures were causing a significant slowdown. We
>>> have dropped them as of this past Saturday (
>>> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
>>> have been re-adding them with more specific scan target types. We’re now
>>> investigating some other optimizations we can make for the next major
>>> ClamAV release to improve scan times but at present we don’t have any other
>>> leads for signatures that may be slowing down scans.
>>> >
>>> > Regards,
>>> > Micah
>>> >
>>> >
>>> > From: clamav-users <clamav-users-bounces@lists.clamav.net> on
>>> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
>>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
>>> > Date: Monday, March 25, 2019 at 9:37 AM
>>> > To: ClamAV users ML <clamav-users@lists.clamav.net>
>>> > Cc: Mark Allan <markjallan@gmail.com>
>>> > Subject: Re: [clamav-users] Scan very slow
>>> >
>>> > Cheers Steve,
>>> >
>>> > In the interest of completeness, here's the scan from today (TXT
>>> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
>>> improvement in scan time, although at 6m 7s it's still almost twice what it
>>> used to be.
>>> >
>>> > Mark
>>> >
>>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
>>> steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com>>
>>> wrote:
>>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
>>> > > Hi all,
>>> > >
>>> > te.
>>> > >
>>> > > Hopefully this helps someone to narrow things down a bit.
>>> > >
>>> > > Mark
>>> > >
>>> >
>>> > 18/3/19 10m 49s TXT from DNS:
>>> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>>> >
>>> > Here's the changes for the above update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/75154
>>> >
>>> > You can also check sigs quickly per update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> >
>>> > Steve
>>> > Twitter: @sanesecurity
>>> >
>>> > _______________________________________________
>>> >
>>> > clamav-users mailing list
>>> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net
>>> >
>>> > https://lists.clamav.net/mailman/listinfo/clamav-users
>>> >
>>> >
>>> > Help us build a comprehensive ClamAV guide:
>>> > https://github.com/vrtadmin/clamav-faq
>>> >
>>> > http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> *DISCLAIMER*
>>>
>>> The information contained in this email and any attachments are
>>> confidential. It is intended solely for the individual or entity to whom
>>> they are addressed. Access to this email by anyone else is unauthorized.
>>>
>>> If you are not the intended recipient, any disclosure, copying,
>>> distribution or any action taken or omitted to be taken in reliance on it,
>>> is prohibited and may be unlawful. If you have received this communication
>>> in error, please notify us immediately by responding to this email and then
>>> delete it from your system.
>>>
>>> The Red Flag Group is neither liable for the proper and complete
>>> transmission of the information contained in this communication nor for any
>>> delay in its receipt.
>>>
>>> Any advice, recommendations or opinion contained within this email or
>>> its attachments are not to be construed as legal advice.
>>>
>>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

I do still have an interest in dropping the Phish.Phishing.REPHISH signatures. I will look into dropping these as well.

We identified the issue causing the PhishTank and Phish signatures to run slowly and how to make them run quickly, but have yet to decide whether or not to reintroduce the signatures as a new database, or perhaps under Email.Phishing or HTML.Phishing so that they can be enabled/disabled with clamd and clamscan configuration options.

I’ll try to see that this is addressed sooner rather than later.

Regards,
Micah

From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Al Varnell via clamav-users <clamav-users@lists.clamav.net>
Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
Date: Wednesday, April 17, 2019 at 7:03 AM
To: "clamav-users@lists.clamav.net" <clamav-users@lists.clamav.net>
Cc: Al Varnell <alvarnell@mac.com>
Subject: Re: [clamav-users] [External] Re: Scan very slow

There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in daily.ldb

-Al-


On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broekman@gmail.com<mailto:maarten.broekman@gmail.com>> wrote:

Are the "Phish" REPHISH signatures still in the daily or were they removed as well? Those were causing part of the issue.


--Maarten

On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more.

-Al-


On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

Hi Micah,

Sorry to pester you, but have you any update on when the remaining Phishtank signatures will be getting removed? It would be really great to get scan times properly back to normal.

Best regards
Mark

On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com<mailto:micasnyd@cisco.com>> wrote:
Mark,

Yes, the plan is still to remove the rest of the Phishtank signatures. We wanted to get things back to relative normal and resolve the immediate crisis. We’ll remove the rest of them soon.

Best,
Micah

From: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
Date: Tuesday, April 9, 2019 at 6:26 AM
To: "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Cc: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Subject: Re: [External] Re: [clamav-users] Scan very slow

The scan times are definitely better than they were - in fact, they're back to how they were before last week's inclusion of the Phishtank signatures. They're still almost double what they used to be though, and as far as I can see, there are still almost 4000 Phishtank signatures in the DB:
$ sigtool --find Phishtank | wc -l
3968

Can I request that those ones also be removed please?

Best regards
Mark

On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com<mailto:micasnyd@cisco.com>> wrote:
Tim,

There are a couple of ways for users to drop specific categories of signatures at this time. Sadly, they wouldn’t have helped this last week. These include bytecode signatures, PUA (potentially unwanted applications) signatures, Email.Phishing and HTML.Phishing signatures, and the Safebrowsing database.

If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or Email.Phishing.Phishtank then they could have been disabled with the clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).

Maybe a better option would be for us to create a new optional database for phishing signatures. However, the names for the databases are hardcoded into freshclam, so it is non-trivial to add a new database and would require a few changes to ClamAV’s code. We have talked about making the databases easier to add/remove in the future so users can have more categories to enable/disable. In this light, it ties in well with existing plans.

Of note the Phishtank sigs from Friday’s daily were removed yesterday and scan times should be back to normal.

Regards,
Micah

From: Tim Hawkins <tim.hawkins@redflaggroup.com<mailto:tim.hawkins@redflaggroup.com>>
Date: Friday, April 5, 2019 at 6:06 PM
To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>, Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Subject: Re: [External] Re: [clamav-users] Scan very slow

Hi Micah

Does clamav partition the database so that signatures that are mainly associated with email scanning can be dropped out for folks only needing filesystems scans, none of our systems use email, and we dont make use of the mailer extension.

Having to load all the email focused signatures could as you have observed impact performance.
Sent from Nine<http://www.9folders.com/>
________________________________
From: "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Sent: Saturday, April 6, 2019 03:18
To: ClamAV users ML; Mark Allan
Cc: Micah Snyder (micasnyd)
Subject: [External] Re: [clamav-users] Scan very slow

Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times.

Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time. One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317).

In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:

* Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds
* Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.

After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit.

The daily update tomorrow will have the change.

-Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.



From: clamav-users <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net>> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Date: Friday, April 5, 2019 at 1:08 PM
To: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>, ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Subject: Re: [clamav-users] Scan very slow

Hi Mark,

Sorry about the delay in responding. I hadn’t looked at my clamav-users filter this morning. Just investigating now. Will respond when I know more.

-Micah

From: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
Date: Friday, April 5, 2019 at 9:12 AM
To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>, "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Subject: Re: [clamav-users] Scan very slow

Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net<http://lists.clamav.net/> isn't responding to http requests anyway)

It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane.

Yesterday's scan using
0.101.2:58:25409:1554370140:1:63:48554:328
took 7m 3s

On the same hardware, scanning the same read-only disk image, with today's scan using
0.101.2:58:25410:1554452941:1:63:48557:328
the scan time has jumped to 26m 15s

This is the longest it has ever taken to scan this volume (cf my previous email of 25th March)

Is there anything that can be excluded?

Best regards
Mark

On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
Thanks Oya for the update. We will continue to investigate the signature performance issue.

Regards,
Micah

On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net> on behalf of oyamada@promark-inc.com<mailto:oyamada@promark-inc.com>> wrote:

Hi Micah

It seems that the scanning slow down issue of this time has been solved
at some level with CVD Update of the other day.
However, there is still big discrepancy in between the current condition and
the last condition in one month ago.

Date Files Scan time
2019/02/15 2550338 08:53:57
2019/03/15 2612792 19:22:54
2019/03/26 2634489 18:13:56
2019/03/27 2637201 18:10:05

We know the improvement of this time is due to the details of CVD, because
we did not make any change on the user's system.
We are going to try some tuning for scanning.

We like to know if you still have some room to make further improvement
for this slow down issue.
Thank you for your help, in advance.

Best regards,
Oya

On Mon, 25 Mar 2019 15:45:02 +0000
"Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

> Hi Mark, all:
>
> I’m disappointed to hear that it is still slow for you.
>
> We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown. We have dropped them as of this past Saturday (https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates have been re-adding them with more specific scan target types. We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans.
>
> Regards,
> Micah
>
>
> From: clamav-users <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net>> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Date: Monday, March 25, 2019 at 9:37 AM
> To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Cc: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
> Subject: Re: [clamav-users] Scan very slow
>
> Cheers Steve,
>
> In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be.
>
> Mark
>
> On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com><mailto:steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com>>> wrote:
> On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > Hi all,
> >
> te.
> >
> > Hopefully this helps someone to narrow things down a bit.
> >
> > Mark
> >
>
> 18/3/19 10m 49s TXT from DNS:
> 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>
> Here's the changes for the above update:
>
> https://lists.gt.net/clamav/virusdb/75154
>
> You can also check sigs quickly per update:
>
> https://lists.gt.net/clamav/virusdb/
>
>
>
> --
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


DISCLAIMER
The information contained in this email and any attachments are confidential. It is intended solely for the individual or entity to whom they are addressed. Access to this email by anyone else is unauthorized.
If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system.
The Red Flag Group is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Any advice, recommendations or opinion contained within this email or its attachments are not to be construed as legal advice.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Looks like all Phish.Phishing.REPHISH_ID_... signatures were dropped by daily-25423 today.

-Al-

> On Apr 17, 2019, at 04:02, Al Varnell <alvarnell@mac.com> wrote:
>
> There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in daily.ldb
>
> -Al-
>
>> On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broekman@gmail.com <mailto:maarten.broekman@gmail.com>> wrote:
>>
>> Are the "Phish" REPHISH signatures still in the daily or were they removed as well? Those were causing part of the issue.
>>
>>
>> --Maarten
>>
>> On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more.
>>
>> -Al-
>>
>>> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>>
>>> Hi Micah,
>>>
>>> Sorry to pester you, but have you any update on when the remaining Phishtank signatures will be getting removed? It would be really great to get scan times properly back to normal.
>>>
>>> Best regards
>>> Mark
>>>
>>> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
>>> Mark,
>>>
>>>
>>> Yes, the plan is still to remove the rest of the Phishtank signatures. We wanted to get things back to relative normal and resolve the immediate crisis. We’ll remove the rest of them soon.
>>>
>>>
>>>
>>> Best,
>>>
>>> Micah
>>>
>>>
>>>
>>> From: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>>> Date: Tuesday, April 9, 2019 at 6:26 AM
>>> To: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>>> Cc: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> Subject: Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> The scan times are definitely better than they were - in fact, they're back to how they were before last week's inclusion of the Phishtank signatures. They're still almost double what they used to be though, and as far as I can see, there are still almost 4000 Phishtank signatures in the DB:
>>>
>>> $ sigtool --find Phishtank | wc -l
>>>
>>> 3968
>>>
>>>
>>>
>>> Can I request that those ones also be removed please?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
>>>
>>> Tim,
>>>
>>>
>>>
>>> There are a couple of ways for users to drop specific categories of signatures at this time. Sadly, they wouldn’t have helped this last week. These include bytecode signatures, PUA (potentially unwanted applications) signatures, Email.Phishing and HTML.Phishing signatures, and the Safebrowsing database.
>>>
>>>
>>>
>>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or Email.Phishing.Phishtank then they could have been disabled with the clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>>>
>>>
>>>
>>> Maybe a better option would be for us to create a new optional database for phishing signatures. However, the names for the databases are hardcoded into freshclam, so it is non-trivial to add a new database and would require a few changes to ClamAV’s code. We have talked about making the databases easier to add/remove in the future so users can have more categories to enable/disable. In this light, it ties in well with existing plans.
>>>
>>>
>>>
>>> Of note the Phishtank sigs from Friday’s daily were removed yesterday and scan times should be back to normal.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Micah
>>>
>>>
>>>
>>> From: Tim Hawkins <tim.hawkins@redflaggroup.com <mailto:tim.hawkins@redflaggroup.com>>
>>> Date: Friday, April 5, 2019 at 6:06 PM
>>> To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>, Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>>> Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>>> Subject: Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Micah
>>>
>>>
>>> Does clamav partition the database so that signatures that are mainly associated with email scanning can be dropped out for folks only needing filesystems scans, none of our systems use email, and we dont make use of the mailer extension.
>>>
>>> Having to load all the email focused signatures could as you have observed impact performance.
>>>
>>> Sent from Nine <http://www.9folders.com/>
>>> From: "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> Sent: Saturday, April 6, 2019 03:18
>>> To: ClamAV users ML; Mark Allan
>>> Cc: Micah Snyder (micasnyd)
>>> Subject: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times.
>>>
>>>
>>>
>>> Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time. One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317 <https://bugzilla.clamav.net/show_bug.cgi?id=12317>).
>>>
>>>
>>>
>>> In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>>>
>>> Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds
>>> Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>>>
>>>
>>> After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit.
>>>
>>>
>>>
>>> The daily update tomorrow will have the change.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>>
>>> Micah Snyder
>>> ClamAV Development
>>> Talos
>>> Cisco Systems, Inc.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: clamav-users <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net>> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> Date: Friday, April 5, 2019 at 1:08 PM
>>> To: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>, ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>>> Subject: Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Mark,
>>>
>>>
>>>
>>> Sorry about the delay in responding. I hadn’t looked at my clamav-users filter this morning. Just investigating now. Will respond when I know more.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>> From: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>>> Date: Friday, April 5, 2019 at 9:12 AM
>>> To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>, "Micah Snyder (micasnyd)" <micasnyd@cisco.com <mailto:micasnyd@cisco.com>>
>>> Subject: Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net <http://lists.clamav.net/> isn't responding to http requests anyway)
>>>
>>>
>>>
>>> It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane.
>>>
>>>
>>>
>>> Yesterday's scan using
>>>
>>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>>
>>> took 7m 3s
>>>
>>>
>>>
>>> On the same hardware, scanning the same read-only disk image, with today's scan using
>>>
>>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>>
>>> the scan time has jumped to 26m 15s
>>>
>>>
>>>
>>> This is the longest it has ever taken to scan this volume (cf my previous email of 25th March)
>>>
>>>
>>>
>>> Is there anything that can be excluded?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>>
>>> Thanks Oya for the update. We will continue to investigate the signature performance issue.
>>>
>>> Regards,
>>> Micah
>>>
>>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net> on behalf of oyamada@promark-inc.com <mailto:oyamada@promark-inc.com>> wrote:
>>>
>>> Hi Micah
>>>
>>> It seems that the scanning slow down issue of this time has been solved
>>> at some level with CVD Update of the other day.
>>> However, there is still big discrepancy in between the current condition and
>>> the last condition in one month ago.
>>>
>>> Date Files Scan time
>>> 2019/02/15 2550338 08:53:57
>>> 2019/03/15 2612792 19:22:54
>>> 2019/03/26 2634489 18:13:56
>>> 2019/03/27 2637201 18:10:05
>>>
>>> We know the improvement of this time is due to the details of CVD, because
>>> we did not make any change on the user's system.
>>> We are going to try some tuning for scanning.
>>>
>>> We like to know if you still have some room to make further improvement
>>> for this slow down issue.
>>> Thank you for your help, in advance.
>>>
>>> Best regards,
>>> Oya
>>>
>>> On Mon, 25 Mar 2019 15:45:02 +0000
>>> "Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote:
>>>
>>> > Hi Mark, all:
>>> >
>>> > I’m disappointed to hear that it is still slow for you.
>>> >
>>> > We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown. We have dropped them as of this past Saturday (https://lists.gt.net/clamav/virusdb/75279 <https://lists.gt.net/clamav/virusdb/75279> ) and in the last two updates have been re-adding them with more specific scan target types. We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans.
>>> >
>>> > Regards,
>>> > Micah
>>> >
>>> >
>>> > From: clamav-users <clamav-users-bounces@lists.clamav.net <mailto:clamav-users-bounces@lists.clamav.net>> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> > Date: Monday, March 25, 2019 at 9:37 AM
>>> > To: ClamAV users ML <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> > Cc: Mark Allan <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>>> > Subject: Re: [clamav-users] Scan very slow
>>> >
>>> > Cheers Steve,
>>> >
>>> > In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be.
>>> >
>>> > Mark
>>> >
>>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_clamav@sanesecurity.com <mailto:steveb_clamav@sanesecurity.com><mailto:steveb_clamav@sanesecurity.com <mailto:steveb_clamav@sanesecurity.com>>> wrote:
>>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
>>> > > Hi all,
>>> > >
>>> > te.
>>> > >
>>> > > Hopefully this helps someone to narrow things down a bit.
>>> > >
>>> > > Mark
>>> > >
>>> >
>>> > 18/3/19 10m 49s TXT from DNS:
>>> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>>> >
>>> > Here's the changes for the above update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/75154 <https://lists.gt.net/clamav/virusdb/75154>
>>> >
>>> > You can also check sigs quickly per update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/ <https://lists.gt.net/clamav/virusdb/>
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> >
>>> > Steve
>>> > Twitter: @sanesecurity
>>> >
>>> > _______________________________________________
>>> >
>>> > clamav-users mailing list
>>> > clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>>
>>> > https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>> >
>>> >
>>> > Help us build a comprehensive ClamAV guide:
>>> > https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>> >
>>> > http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>>
>>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>>
>>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>>
>>>
>>> DISCLAIMER
>>>
>>> The information contained in this email and any attachments are confidential. It is intended solely for the individual or entity to whom they are addressed. Access to this email by anyone else is unauthorized.
>>>
>>> If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system.
>>>
>>> The Red Flag Group is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
>>>
>>> Any advice, recommendations or opinion contained within this email or its attachments are not to be construed as legal advice.
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>>
>>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>
>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>
>>
>

Thanks Al and thanks to the ClamAV team for the quick work on this. 
Love ClamAV.

Here's some some data on one system about the times for freshclam to
complete:

25409 Freshclam took 3:30s
25410 Freshclam took 9:27s
25422 Freshclam took 9:17s
25423 Freshclam took 1:53s

Here's some of the snippets for a few data points above:

Thu Apr  4 04:01:09 2019 -> *Current working dir is
/usr/local/clamav/share/clamav
Thu Apr  4 04:01:09 2019 -> *Max retries == 3
Thu Apr  4 04:01:09 2019 -> ClamAV update process started at Thu Apr  4
04:01:09 2019
Thu Apr  4 04:01:09 2019 -> *Using IPv6 aware code
Thu Apr  4 04:01:09 2019 -> *Querying current.cvd.clamav.net
Thu Apr  4 04:01:09 2019 -> *TTL: 1800
Thu Apr  4 04:01:09 2019 -> *Software version from DNS: 0.101.2
Thu Apr  4 04:01:09 2019 -> ^Your ClamAV installation is OUTDATED!
Thu Apr  4 04:01:09 2019 -> ^Local version: 0.100.2 Recommended version:
0.101.2
Thu Apr  4 04:01:09 2019 -> DON'T PANIC! Read
https://www.clamav.net/documents/upgrading-clamav
Thu Apr  4 04:01:09 2019 -> *main.cvd version from DNS: 58
Thu Apr  4 04:01:09 2019 -> main.cld is up to date (version: 58, sigs:
4566249, f-level: 60, builder: sigmgr)
Thu Apr  4 04:01:09 2019 -> *daily.cvd version from DNS: 25409
Thu Apr  4 04:01:16 2019 -> *Retrieving
http://db.US.clamav.net/daily-25409.cdiff                                                  

Thu Apr  4 04:01:16 2019 -> *Trying to download
http://db.US.clamav.net/daily-25409.cdiff (IP: 104.16.219.84)
Thu Apr  4 04:01:16 2019 -> Downloading daily-25409.cdiff [100%]
Thu Apr  4 04:01:17 2019 -> *cdiff_apply: Parsed 2477 lines and executed
2477 commands
Thu Apr  4 04:01:18 2019 -> *Loading signatures from daily.cld
Thu Apr  4 04:03:21 2019 -> *Properly loaded 1543509 signatures from new
daily.cld
Thu Apr  4 04:03:22 2019 -> daily.cld updated (version: 25409, sigs:
1543509, f-level: 63, builder: raynman)
Thu Apr  4 04:03:22 2019 -> *Querying
daily.25409.93.1.0.6810DB54.ping.clamav.net
Thu Apr  4 04:03:22 2019 -> *bytecode.cvd version from DNS: 328
Thu Apr  4 04:03:22 2019 -> bytecode.cld is up to date (version: 328,
sigs: 94, f-level: 63, builder: neo)
Thu Apr  4 04:03:39 2019 -> Database updated (6109852 signatures) from
db.US.clamav.net (IP: 104.16.219.84)
Thu Apr  4 04:03:39 2019 -> Clamd successfully notified about the update.



Fri Apr  5 05:01:10 2019 -> *Current working dir is
/usr/local/clamav/share/clamav
Fri Apr  5 05:01:10 2019 -> *Max retries == 3
Fri Apr  5 05:01:10 2019 -> ClamAV update process started at Fri Apr  5
05:01:10 2019
Fri Apr  5 05:01:10 2019 -> *Using IPv6 aware code
Fri Apr  5 05:01:10 2019 -> *Querying current.cvd.clamav.net
Fri Apr  5 05:01:10 2019 -> *TTL: 1800
Fri Apr  5 05:01:10 2019 -> *Software version from DNS: 0.101.2
Fri Apr  5 05:01:10 2019 -> ^Your ClamAV installation is OUTDATED!
Fri Apr  5 05:01:10 2019 -> ^Local version: 0.100.2 Recommended version:
0.101.2
Fri Apr  5 05:01:10 2019 -> DON'T PANIC! Read
https://www.clamav.net/documents/upgrading-clamav
Fri Apr  5 05:01:10 2019 -> *main.cvd version from DNS: 58
Fri Apr  5 05:01:10 2019 -> main.cld is up to date (version: 58, sigs:
4566249, f-level: 60, builder: sigmgr)
Fri Apr  5 05:01:10 2019 -> *daily.cvd version from DNS: 25410
Fri Apr  5 05:01:15 2019 -> *Retrieving
http://db.US.clamav.net/daily-25410.cdiff                                                  

Fri Apr  5 05:01:15 2019 -> *Trying to download
http://db.US.clamav.net/daily-25410.cdiff (IP: 104.16.219.84)
Fri Apr  5 05:01:16 2019 -> Downloading daily-25410.cdiff [100%]
Fri Apr  5 05:01:22 2019 -> *cdiff_apply: Parsed 9065 lines and executed
9065 commands
Fri Apr  5 05:01:24 2019 -> *Loading signatures from daily.cld
Fri Apr  5 05:10:19 2019 -> *Properly loaded 1552552 signatures from new
daily.cld
Fri Apr  5 05:10:20 2019 -> daily.cld updated (version: 25410, sigs:
1552552, f-level: 63, builder: raynman)
Fri Apr  5 05:10:20 2019 -> *Querying
daily.25410.93.1.0.6810DB54.ping.clamav.net
Fri Apr  5 05:10:22 2019 -> *bytecode.cvd version from DNS: 328
Fri Apr  5 05:10:22 2019 -> bytecode.cld is up to date (version: 328,
sigs: 94, f-level: 63, builder: neo)
Fri Apr  5 05:10:37 2019 -> Database updated (6118895 signatures) from
db.US.clamav.net (IP: 104.16.219.84)
Fri Apr  5 05:10:37 2019 -> Clamd successfully notified about the update.


Thu Apr 18 05:01:37 2019 -> *Current working dir is
/usr/local/clamav/share/clamav
Thu Apr 18 05:01:37 2019 -> *Max retries == 3
Thu Apr 18 05:01:37 2019 -> ClamAV update process started at Thu Apr 18
05:01:37 2019
Thu Apr 18 05:01:37 2019 -> *Using IPv6 aware code
Thu Apr 18 05:01:37 2019 -> *Querying current.cvd.clamav.net
Thu Apr 18 05:01:37 2019 -> *TTL: 1800
Thu Apr 18 05:01:37 2019 -> *Software version from DNS: 0.101.2
Thu Apr 18 05:01:37 2019 -> ^Your ClamAV installation is OUTDATED!
Thu Apr 18 05:01:37 2019 -> ^Local version: 0.100.3 Recommended version:
0.101.2
Thu Apr 18 05:01:37 2019 -> DON'T PANIC! Read
https://www.clamav.net/documents/upgrading-clamav
Thu Apr 18 05:01:37 2019 -> *main.cvd version from DNS: 58
Thu Apr 18 05:01:37 2019 -> main.cvd is up to date (version: 58, sigs:
4566249, f-level: 60, builder: sigmgr)
Thu Apr 18 05:01:37 2019 -> *daily.cvd version from DNS: 25423
Thu Apr 18 05:01:47 2019 -> *Retrieving
http://db.US.clamav.net/daily-25423.cdiff                                                  

Thu Apr 18 05:01:47 2019 -> *Trying to download
http://db.US.clamav.net/daily-25423.cdiff (IP: 104.16.219.84)
Thu Apr 18 05:01:48 2019 -> Downloading daily-25423.cdiff [100%]
Thu Apr 18 05:01:48 2019 -> *cdiff_apply: Parsed 4849 lines and executed
4849 commands
Thu Apr 18 05:01:51 2019 -> *Loading signatures from daily.cld
Thu Apr 18 05:03:13 2019 -> *Properly loaded 1551908 signatures from new
daily.cld
Thu Apr 18 05:03:14 2019 -> daily.cld updated (version: 25423, sigs:
1551908, f-level: 63, builder: raynman)
Thu Apr 18 05:03:14 2019 -> *Querying
daily.25423.93.1.0.6810DB54.ping.clamav.net
Thu Apr 18 05:03:14 2019 -> *bytecode.cvd version from DNS: 328
Thu Apr 18 05:03:14 2019 -> bytecode.cvd is up to date (version: 328,
sigs: 94, f-level: 63, builder: neo)
Thu Apr 18 05:03:30 2019 -> Database updated (6118251 signatures) from
db.US.clamav.net (IP: 104.16.219.84)
Thu Apr 18 05:03:30 2019 -> Clamd successfully notified about the update.

Regards,
KAM

On 4/18/2019 4:45 AM, Al Varnell via clamav-users wrote:
> Looks like all Phish.Phishing.REPHISH_ID_... signatures were dropped
> by daily-25423 today.
>
> -Al-
>
>> On Apr 17, 2019, at 04:02, Al Varnell <alvarnell@mac.com
>> <mailto:alvarnell@mac.com>> wrote:
>>
>> There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in
>> daily.ldb
>>
>> -Al-
>>
>>> On Apr 17, 2019, at 03:36, Maarten Broekman
>>> <maarten.broekman@gmail.com <mailto:maarten.broekman@gmail.com>> wrote:
>>>
>>> Are the "Phish" REPHISH signatures still in the daily or were they
>>> removed as well? Those were causing part of the issue.
>>>
>>>
>>> --Maarten
>>>
>>> On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users
>>> <clamav-users@lists.clamav.net
>>> <mailto:clamav-users@lists.clamav.net>> wrote:
>>>
>>> An additional
>>> 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped
>>> by daily-25417 on 12 April, and I can't seem to locate any more.
>>>
>>> -Al-
>>>
>>>> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>> wrote:
>>>>
>>>> Hi Micah,
>>>>
>>>> Sorry to pester you, but have you any update on when the
>>>> remaining Phishtank signatures will be getting removed? It
>>>> would be really great to get scan times properly back to normal.
>>>>
>>>> Best regards
>>>> Mark
>>>>
>>>> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd)
>>>> <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
>>>>
>>>> Mark,
>>>>
>>>>
>>>> Yes, the plan is still to remove the rest of the Phishtank
>>>> signatures.  We wanted to get things back to relative
>>>> normal and resolve the immediate crisis.  We’ll remove the
>>>> rest of them soon.
>>>>
>>>>  
>>>>
>>>> Best,
>>>>
>>>> Micah  
>>>>
>>>>  
>>>>
>>>> *From: *Mark Allan <markjallan@gmail.com
>>>> <mailto:markjallan@gmail.com>>
>>>> *Date: *Tuesday, April 9, 2019 at 6:26 AM
>>>> *To: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com
>>>> <mailto:micasnyd@cisco.com>>
>>>> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>>
>>>>  
>>>>
>>>> The scan times are definitely better than they were - in
>>>> fact, they're back to how they were before last week's
>>>> inclusion of the Phishtank signatures. They're still almost
>>>> double what they used to be though, and as far as I can
>>>> see, there are still almost 4000 Phishtank signatures in
>>>> the DB: 
>>>>
>>>> $ sigtool --find Phishtank | wc -l
>>>>
>>>>     3968
>>>>
>>>>  
>>>>
>>>> Can I request that those ones also be removed please?
>>>>
>>>>  
>>>>
>>>> Best regards
>>>>
>>>> Mark 
>>>>
>>>>  
>>>>
>>>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd)
>>>> <micasnyd@cisco.com <mailto:micasnyd@cisco.com>> wrote:
>>>>
>>>> Tim,
>>>>
>>>>  
>>>>
>>>> There are a couple of ways for users to drop specific
>>>> categories of signatures at this time.  Sadly, they
>>>> wouldn’t have helped this last week.  These include
>>>> bytecode signatures, PUA (potentially unwanted
>>>> applications) signatures, Email.Phishing and
>>>> HTML.Phishing signatures, and the Safebrowsing database. 
>>>>
>>>>  
>>>>
>>>> If we had named the Phishtank.Phishing sigs to
>>>> HTML.Phishing.Phishtank or Email.Phishing.Phishtank
>>>> then they could have been disabled with the clamscan
>>>> option `--phishing-sigs=no` (clamd.conf:
>>>> `PhishingSignatures no`).
>>>>
>>>>  
>>>>
>>>> Maybe a better option would be for us to create a new
>>>> optional database for phishing signatures. However, the
>>>> names for the databases are hardcoded into freshclam,
>>>> so it is non-trivial to add a new database and would
>>>> require a few changes to ClamAV’s code. We have talked
>>>> about making the databases easier to add/remove in the
>>>> future so users can have more categories to
>>>> enable/disable. In this light, it ties in well with
>>>> existing plans.
>>>>
>>>>  
>>>>
>>>> Of note the Phishtank sigs from Friday’s daily were
>>>> removed yesterday and scan times should be back to normal.
>>>>
>>>>  
>>>>
>>>> Regards,
>>>>
>>>> Micah
>>>>
>>>>  
>>>>
>>>> *From: *Tim Hawkins <tim.hawkins@redflaggroup.com
>>>> <mailto:tim.hawkins@redflaggroup.com>>
>>>> *Date: *Friday, April 5, 2019 at 6:06 PM
>>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>, Mark Allan
>>>> <markjallan@gmail.com <mailto:markjallan@gmail.com>>
>>>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com
>>>> <mailto:micasnyd@cisco.com>>
>>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>>
>>>>  
>>>>
>>>> Hi Micah
>>>>
>>>>
>>>> Does clamav partition the database so that signatures
>>>> that are mainly associated with email scanning can be
>>>> dropped out for folks only needing filesystems scans,
>>>>  none of our systems use email, and we dont make use of
>>>> the mailer extension. 
>>>>
>>>> Having to load all the email focused signatures could
>>>> as you have observed impact performance.
>>>>
>>>> Sent from Nine <http://www.9folders.com/>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> *From:* "Micah Snyder (micasnyd) via clamav-users"
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>> *Sent:* Saturday, April 6, 2019 03:18
>>>> *To:* ClamAV users ML; Mark Allan
>>>> *Cc:* Micah Snyder (micasnyd)
>>>> *Subject:* [External] Re: [clamav-users] Scan very slow
>>>>
>>>>  
>>>>
>>>> Regarding slow scan times today (and slow scan times in
>>>> general), it appears that the signatures we generate
>>>> based on PhishTank’s feed for phishing URLs are
>>>> resulting in very slow load and scan times.
>>>>
>>>>  
>>>>
>>>> Today’s daily update saw 7448 new Phishtank signatures
>>>> (much higher than usual) coinciding with the immediate
>>>> performance drop for load time and scan time.  One user
>>>> reported that the load time today on some of his slower
>>>> machines was slow enough to exceed the timeout for
>>>> service startup
>>>> (https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>>>>
>>>>  
>>>>
>>>> In limited testing on my own machine I saw the
>>>> following change after dropping the Phishtank.Phishing
>>>> signatures from daily.cvd’s daily.ldb file:
>>>>
>>>> * Database load time on my laptop went from
>>>> 75.43203997612 seconds down to 14.859203100204468
>>>> seconds
>>>> * Scan time (for an arbitrary pdf) went from 1.798
>>>> sec to 0.644 sec.
>>>>
>>>>  
>>>>
>>>> After some discussion between the teams that work on
>>>> ClamAV and ClamAV signature content and deployment,
>>>> we’ve agreed to drop PhishTank signatures from the
>>>> database until we can determine a way to craft
>>>> Phishtank signatures without incurring such a
>>>> significant performance hit.
>>>>
>>>>  
>>>>
>>>> The daily update tomorrow will have the change.
>>>>
>>>>  
>>>>
>>>> -Micah
>>>>
>>>>  
>>>>
>>>>
>>>> Micah Snyder
>>>> ClamAV Development
>>>> Talos
>>>> Cisco Systems, Inc.
>>>>
>>>>  
>>>>
>>>>  
>>>>
>>>>  
>>>>
>>>> *From: *clamav-users
>>>> <clamav-users-bounces@lists.clamav.net
>>>> <mailto:clamav-users-bounces@lists.clamav.net>> on
>>>> behalf of "Micah Snyder (micasnyd) via clamav-users"
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>> *Reply-To: *ClamAV users ML
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>> *Date: *Friday, April 5, 2019 at 1:08 PM
>>>> *To: *Mark Allan <markjallan@gmail.com
>>>> <mailto:markjallan@gmail.com>>, ClamAV users ML
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com
>>>> <mailto:micasnyd@cisco.com>>
>>>> *Subject: *Re: [clamav-users] Scan very slow
>>>>
>>>>  
>>>>
>>>> Hi Mark,
>>>>
>>>>  
>>>>
>>>> Sorry about the delay in responding.  I hadn’t looked
>>>> at my clamav-users filter this morning.  Just
>>>> investigating now.  Will respond when I know more.
>>>>
>>>>  
>>>>
>>>> -Micah
>>>>
>>>>  
>>>>
>>>> *From: *Mark Allan <markjallan@gmail.com
>>>> <mailto:markjallan@gmail.com>>
>>>> *Date: *Friday, April 5, 2019 at 9:12 AM
>>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>, "Micah Snyder
>>>> (micasnyd)" <micasnyd@cisco.com
>>>> <mailto:micasnyd@cisco.com>>
>>>> *Subject: *Re: [clamav-users] Scan very slow
>>>>
>>>>  
>>>>
>>>> Also CC'ing Micah directly as the mailing list would
>>>> appear to be offline (at least lists.clamav.net
>>>> <http://lists.clamav.net/> isn't responding to http
>>>> requests anyway)
>>>>
>>>>  
>>>>
>>>> It looks like scan times have gone through the roof. As
>>>> Oya said, they're still considerably higher than they
>>>> were a couple of months ago, but today's scan time is
>>>> insane.
>>>>
>>>>  
>>>>
>>>> Yesterday's scan using
>>>>
>>>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>>>
>>>> took 7m 3s
>>>>
>>>>  
>>>>
>>>> On the same hardware, scanning the same read-only disk
>>>> image, with today's scan using
>>>>
>>>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>>>
>>>> the scan time has jumped to 26m 15s
>>>>
>>>>  
>>>>
>>>> This is the longest it has ever taken to scan this
>>>> volume (cf my previous email of 25th March)
>>>>
>>>>  
>>>>
>>>> Is there anything that can be excluded?
>>>>
>>>>  
>>>>
>>>> Best regards
>>>>
>>>> Mark
>>>>
>>>>  
>>>>
>>>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd)
>>>> via clamav-users <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>> wrote:
>>>>
>>>> Thanks Oya for the update.  We will continue to
>>>> investigate the signature performance issue. 
>>>>
>>>> Regards,
>>>> Micah
>>>>
>>>> On 3/28/19, 9:50 AM, "clamav-users on behalf of
>>>> Tsutomu Oyamada"
>>>> <clamav-users-bounces@lists.clamav.net
>>>> <mailto:clamav-users-bounces@lists.clamav.net> on
>>>> behalf of oyamada@promark-inc.com
>>>> <mailto:oyamada@promark-inc.com>> wrote:
>>>>
>>>>     Hi Micah
>>>>
>>>>     It seems that the  scanning slow down issue of
>>>> this time has been solved
>>>>     at some level with CVD Update of the other day.
>>>>     However, there is still big discrepancy in
>>>> between the current condition and
>>>>     the last condition in one month ago.
>>>>
>>>>     Date                Files               Scan time
>>>>     2019/02/15  2550338         08:53:57
>>>>     2019/03/15  2612792         19:22:54
>>>>     2019/03/26  2634489         18:13:56
>>>>     2019/03/27  2637201         18:10:05
>>>>
>>>>     We know the improvement of this time is due to
>>>> the details of CVD, because
>>>>     we did not make any change on the user's system.
>>>>     We are going to try some tuning for scanning.
>>>>
>>>>     We like to know if you still have some room to
>>>> make further improvement
>>>>     for this slow down issue.
>>>>     Thank you for your help, in advance.
>>>>
>>>>     Best regards,
>>>>     Oya
>>>>
>>>>     On Mon, 25 Mar 2019 15:45:02 +0000
>>>>     "Micah Snyder \(micasnyd\) via clamav-users"
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>> wrote:
>>>>
>>>>     > Hi Mark, all:
>>>>     > 
>>>>     > I’m disappointed to hear that it is still
>>>> slow for you.
>>>>     > 
>>>>     > We found that the target-type of signatures
>>>> used for PhishTank.Phishing signatures were causing
>>>> a significant slowdown.   We have dropped them as
>>>> of this past Saturday
>>>> (https://lists.gt.net/clamav/virusdb/75279 ) and in
>>>> the last two updates have been re-adding them with
>>>> more specific scan target types.  We’re now
>>>> investigating some other optimizations we can make
>>>> for the next major ClamAV release to improve scan
>>>> times but at present we don’t have any other leads
>>>> for signatures that may be slowing down scans.
>>>>     > 
>>>>     > Regards,
>>>>     > Micah
>>>>     > 
>>>>     > 
>>>>     > From: clamav-users
>>>> <clamav-users-bounces@lists.clamav.net
>>>> <mailto:clamav-users-bounces@lists.clamav.net>> on
>>>> behalf of Mark Allan via clamav-users
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>>     > Reply-To: ClamAV users ML
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>>     > Date: Monday, March 25, 2019 at 9:37 AM
>>>>     > To: ClamAV users ML
>>>> <clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>>     > Cc: Mark Allan <markjallan@gmail.com
>>>> <mailto:markjallan@gmail.com>>
>>>>     > Subject: Re: [clamav-users] Scan very slow
>>>>     > 
>>>>     > Cheers Steve,
>>>>     > 
>>>>     > In the interest of completeness, here's the
>>>> scan from today (TXT from DNS:
>>>> 0.101.1:58:25399:1553509741:1:63:48528:328) showing
>>>> a marked improvement in scan time, although at 6m
>>>> 7s it's still almost twice what it used to be.
>>>>     > 
>>>>     > Mark
>>>>     > 
>>>>     > On Mon, 25 Mar 2019 at 12:56, Steve Basford
>>>> <steveb_clamav@sanesecurity.com
>>>> <mailto:steveb_clamav@sanesecurity.com><mailto:steveb_clamav@sanesecurity.com
>>>> <mailto:steveb_clamav@sanesecurity.com>>> wrote:
>>>>     > On 2019-03-25 10:52, Mark Allan via
>>>> clamav-users wrote:
>>>>     > > Hi all,
>>>>     > >
>>>>     > te.
>>>>     > >
>>>>     > > Hopefully this helps someone to narrow
>>>> things down a bit.
>>>>     > >
>>>>     > > Mark
>>>>     > >
>>>>     > 
>>>>     > 18/3/19         10m 49s         TXT from DNS:
>>>>     > 0.101.1:58:25392:1552904941:1:63:48507:328   
>>>>   ***
>>>>     > 
>>>>     > Here's the changes for the above update:
>>>>     > 
>>>>     > https://lists.gt.net/clamav/virusdb/75154
>>>>     > 
>>>>     > You can also check sigs quickly per update:
>>>>     > 
>>>>     > https://lists.gt.net/clamav/virusdb/
>>>>     > 
>>>>     > 
>>>>     > 
>>>>     > --
>>>>     > Cheers,
>>>>     > 
>>>>     > Steve
>>>>     > Twitter: @sanesecurity
>>>>     > 
>>>>     > _______________________________________________
>>>>     > 
>>>>     > clamav-users mailing list
>>>>     > clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>>
>>>>  
>>>>   > https://lists.clamav.net/mailman/listinfo/clamav-users
>>>>     > 
>>>>     > 
>>>>     > Help us build a comprehensive ClamAV guide:
>>>>     > https://github.com/vrtadmin/clamav-faq
>>>>     > 
>>>>     > http://www.clamav.net/contact.html#ml
>>>>
>>>>
>>>>
>>>>     _______________________________________________
>>>>
>>>>     clamav-users mailing list
>>>>     clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>
>>>>  
>>>>   https://lists.clamav.net/mailman/listinfo/clamav-users
>>>>
>>>>
>>>>     Help us build a comprehensive ClamAV guide:
>>>>     https://github.com/vrtadmin/clamav-faq
>>>>
>>>>     http://www.clamav.net/contact.html#ml
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>>> clamav-users mailing list
>>>> clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>
>>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>>
>>>>
>>>> Help us build a comprehensive ClamAV guide:
>>>> https://github.com/vrtadmin/clamav-faq
>>>>
>>>> http://www.clamav.net/contact.html#ml
>>>>
>>>>
>>>>
>>>> *DISCLAIMER*
>>>>
>>>> The information contained in this email and any
>>>> attachments are confidential. It is intended solely for
>>>> the individual or entity to whom they are addressed.
>>>> Access to this email by anyone else is unauthorized.
>>>>
>>>> If you are not the intended recipient, any disclosure,
>>>> copying, distribution or any action taken or omitted to
>>>> be taken in reliance on it, is prohibited and may be
>>>> unlawful. If you have received this communication in
>>>> error, please notify us immediately by responding to
>>>> this email and then delete it from your system.
>>>>
>>>> The Red Flag Group is neither liable for the proper and
>>>> complete transmission of the information contained in
>>>> this communication nor for any delay in its receipt.
>>>>
>>>> Any advice, recommendations or opinion contained within
>>>> this email or its attachments are not to be construed
>>>> as legal advice.
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>>> clamav-users mailing list
>>>> clamav-users@lists.clamav.net
>>>> <mailto:clamav-users@lists.clamav.net>
>>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>>
>>>>
>>>> Help us build a comprehensive ClamAV guide:
>>>> https://github.com/vrtadmin/clamav-faq
>>>>
>>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml


--
*Kevin A. McGrail*
CEO Emeritus

Peregrine Computer Consultants Corporation
10311 Cascade Lane
Fairfax, VA 22032

http://www.pccc.com/

703-359-9700 / 800-823-8402 (Toll-Free)
703-798-0171 (wireless)
KMcGrail@PCCC.com <mailto:kmcgrail@pccc.com>

https://www.linkedin.com/in/kmcgrail

Fantastic! I can also confirm that scan times are back to normal now -
more-or-less back to what they were in early February.

The time for one of our FP test volumes which I've been referencing in this
thread is back down to 3m 30s, and the total time for our *full* FP test is
back down from several hours to just 47 minutes.

Thank you!
Mark

On Thu, 18 Apr 2019 at 09:46, Al Varnell via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Looks like all Phish.Phishing.REPHISH_ID_... signatures were dropped by
> daily-25423 today.
>
> -Al-
>
> On Apr 17, 2019, at 04:02, Al Varnell <alvarnell@mac.com> wrote:
>
> There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in
> daily.ldb
>
> -Al-
>
> On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broekman@gmail.com>
> wrote:
>
> Are the "Phish" REPHISH signatures still in the daily or were they removed
> as well? Those were causing part of the issue.
>
>
> --Maarten
>
> On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <
> clamav-users@lists.clamav.net> wrote:
>
>> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were
>> dropped by daily-25417 on 12 April, and I can't seem to locate any more.
>>
>> -Al-
>>
>> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <
>> clamav-users@lists.clamav.net> wrote:
>>
>> Hi Micah,
>>
>> Sorry to pester you, but have you any update on when the remaining
>> Phishtank signatures will be getting removed? It would be really great to
>> get scan times properly back to normal.
>>
>> Best regards
>> Mark
>>
>> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com>
>> wrote:
>>
>>> Mark,
>>>
>>>
>>> Yes, the plan is still to remove the rest of the Phishtank signatures.
>>> We wanted to get things back to relative normal and resolve the immediate
>>> crisis. We’ll remove the rest of them soon.
>>>
>>>
>>>
>>> Best,
>>>
>>> Micah
>>>
>>>
>>>
>>> *From: *Mark Allan <markjallan@gmail.com>
>>> *Date: *Tuesday, April 9, 2019 at 6:26 AM
>>> *To: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>>> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net>
>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> The scan times are definitely better than they were - in fact, they're
>>> back to how they were before last week's inclusion of the Phishtank
>>> signatures. They're still almost double what they used to be though, and as
>>> far as I can see, there are still almost 4000 Phishtank signatures in the
>>> DB:
>>>
>>> $ sigtool --find Phishtank | wc -l
>>>
>>> 3968
>>>
>>>
>>>
>>> Can I request that those ones also be removed please?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com>
>>> wrote:
>>>
>>> Tim,
>>>
>>>
>>>
>>> There are a couple of ways for users to drop specific categories of
>>> signatures at this time. Sadly, they wouldn’t have helped this last week.
>>> These include bytecode signatures, PUA (potentially unwanted applications)
>>> signatures, Email.Phishing and HTML.Phishing signatures, and the
>>> Safebrowsing database.
>>>
>>>
>>>
>>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank
>>> or Email.Phishing.Phishtank then they could have been disabled with the
>>> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>>>
>>>
>>>
>>> Maybe a better option would be for us to create a new optional database
>>> for phishing signatures. However, the names for the databases are hardcoded
>>> into freshclam, so it is non-trivial to add a new database and would
>>> require a few changes to ClamAV’s code. We have talked about making the
>>> databases easier to add/remove in the future so users can have more
>>> categories to enable/disable. In this light, it ties in well with existing
>>> plans.
>>>
>>>
>>>
>>> Of note the Phishtank sigs from Friday’s daily were removed yesterday
>>> and scan times should be back to normal.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Micah
>>>
>>>
>>>
>>> *From: *Tim Hawkins <tim.hawkins@redflaggroup.com>
>>> *Date: *Friday, April 5, 2019 at 6:06 PM
>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan <
>>> markjallan@gmail.com>
>>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Micah
>>>
>>>
>>> Does clamav partition the database so that signatures that are mainly
>>> associated with email scanning can be dropped out for folks only needing
>>> filesystems scans, none of our systems use email, and we dont make use of
>>> the mailer extension.
>>>
>>> Having to load all the email focused signatures could as you have
>>> observed impact performance.
>>>
>>> Sent from Nine <http://www.9folders.com/>
>>> ------------------------------
>>>
>>> *From:* "Micah Snyder (micasnyd) via clamav-users" <
>>> clamav-users@lists.clamav.net>
>>> *Sent:* Saturday, April 6, 2019 03:18
>>> *To:* ClamAV users ML; Mark Allan
>>> *Cc:* Micah Snyder (micasnyd)
>>> *Subject:* [External] Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Regarding slow scan times today (and slow scan times in general), it
>>> appears that the signatures we generate based on PhishTank’s feed for
>>> phishing URLs are resulting in very slow load and scan times.
>>>
>>>
>>>
>>> Today’s daily update saw 7448 new Phishtank signatures (much higher than
>>> usual) coinciding with the immediate performance drop for load time and
>>> scan time. One user reported that the load time today on some of his
>>> slower machines was slow enough to exceed the timeout for service startup (
>>> https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>>>
>>>
>>>
>>> In limited testing on my own machine I saw the following change after
>>> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>>>
>>> - Database load time on my laptop went from 75.43203997612 seconds
>>> down to 14.859203100204468 seconds
>>> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>>>
>>>
>>>
>>> After some discussion between the teams that work on ClamAV and ClamAV
>>> signature content and deployment, we’ve agreed to drop PhishTank signatures
>>> from the database until we can determine a way to craft Phishtank
>>> signatures without incurring such a significant performance hit.
>>>
>>>
>>>
>>> The daily update tomorrow will have the change.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>>
>>> Micah Snyder
>>> ClamAV Development
>>> Talos
>>> Cisco Systems, Inc.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From: *clamav-users <clamav-users-bounces@lists.clamav.net> on behalf
>>> of "Micah Snyder (micasnyd) via clamav-users" <
>>> clamav-users@lists.clamav.net>
>>> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net>
>>> *Date: *Friday, April 5, 2019 at 1:08 PM
>>> *To: *Mark Allan <markjallan@gmail.com>, ClamAV users ML <
>>> clamav-users@lists.clamav.net>
>>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd@cisco.com>
>>> *Subject: *Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Hi Mark,
>>>
>>>
>>>
>>> Sorry about the delay in responding. I hadn’t looked at my clamav-users
>>> filter this morning. Just investigating now. Will respond when I know
>>> more.
>>>
>>>
>>>
>>> -Micah
>>>
>>>
>>>
>>> *From: *Mark Allan <markjallan@gmail.com>
>>> *Date: *Friday, April 5, 2019 at 9:12 AM
>>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder
>>> (micasnyd)" <micasnyd@cisco.com>
>>> *Subject: *Re: [clamav-users] Scan very slow
>>>
>>>
>>>
>>> Also CC'ing Micah directly as the mailing list would appear to be
>>> offline (at least lists.clamav.net isn't responding to http requests
>>> anyway)
>>>
>>>
>>>
>>> It looks like scan times have gone through the roof. As Oya said,
>>> they're still considerably higher than they were a couple of months ago,
>>> but today's scan time is insane.
>>>
>>>
>>>
>>> Yesterday's scan using
>>>
>>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>>
>>> took 7m 3s
>>>
>>>
>>>
>>> On the same hardware, scanning the same read-only disk image, with
>>> today's scan using
>>>
>>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>>
>>> the scan time has jumped to 26m 15s
>>>
>>>
>>>
>>> This is the longest it has ever taken to scan this volume (cf my
>>> previous email of 25th March)
>>>
>>>
>>>
>>> Is there anything that can be excluded?
>>>
>>>
>>>
>>> Best regards
>>>
>>> Mark
>>>
>>>
>>>
>>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
>>> clamav-users@lists.clamav.net> wrote:
>>>
>>> Thanks Oya for the update. We will continue to investigate the
>>> signature performance issue.
>>>
>>> Regards,
>>> Micah
>>>
>>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
>>> clamav-users-bounces@lists.clamav.net on behalf of
>>> oyamada@promark-inc.com> wrote:
>>>
>>> Hi Micah
>>>
>>> It seems that the scanning slow down issue of this time has been
>>> solved
>>> at some level with CVD Update of the other day.
>>> However, there is still big discrepancy in between the current
>>> condition and
>>> the last condition in one month ago.
>>>
>>> Date Files Scan time
>>> 2019/02/15 2550338 08:53:57
>>> 2019/03/15 2612792 19:22:54
>>> 2019/03/26 2634489 18:13:56
>>> 2019/03/27 2637201 18:10:05
>>>
>>> We know the improvement of this time is due to the details of CVD,
>>> because
>>> we did not make any change on the user's system.
>>> We are going to try some tuning for scanning.
>>>
>>> We like to know if you still have some room to make further
>>> improvement
>>> for this slow down issue.
>>> Thank you for your help, in advance.
>>>
>>> Best regards,
>>> Oya
>>>
>>> On Mon, 25 Mar 2019 15:45:02 +0000
>>> "Micah Snyder \(micasnyd\) via clamav-users" <
>>> clamav-users@lists.clamav.net> wrote:
>>>
>>> > Hi Mark, all:
>>> >
>>> > I’m disappointed to hear that it is still slow for you.
>>> >
>>> > We found that the target-type of signatures used for
>>> PhishTank.Phishing signatures were causing a significant slowdown. We
>>> have dropped them as of this past Saturday (
>>> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
>>> have been re-adding them with more specific scan target types. We’re now
>>> investigating some other optimizations we can make for the next major
>>> ClamAV release to improve scan times but at present we don’t have any other
>>> leads for signatures that may be slowing down scans.
>>> >
>>> > Regards,
>>> > Micah
>>> >
>>> >
>>> > From: clamav-users <clamav-users-bounces@lists.clamav.net> on
>>> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
>>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
>>> > Date: Monday, March 25, 2019 at 9:37 AM
>>> > To: ClamAV users ML <clamav-users@lists.clamav.net>
>>> > Cc: Mark Allan <markjallan@gmail.com>
>>> > Subject: Re: [clamav-users] Scan very slow
>>> >
>>> > Cheers Steve,
>>> >
>>> > In the interest of completeness, here's the scan from today (TXT
>>> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
>>> improvement in scan time, although at 6m 7s it's still almost twice what it
>>> used to be.
>>> >
>>> > Mark
>>> >
>>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
>>> steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com>>
>>> wrote:
>>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
>>> > > Hi all,
>>> > >
>>> > te.
>>> > >
>>> > > Hopefully this helps someone to narrow things down a bit.
>>> > >
>>> > > Mark
>>> > >
>>> >
>>> > 18/3/19 10m 49s TXT from DNS:
>>> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>>> >
>>> > Here's the changes for the above update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/75154
>>> >
>>> > You can also check sigs quickly per update:
>>> >
>>> > https://lists.gt.net/clamav/virusdb/
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> >
>>> > Steve
>>> > Twitter: @sanesecurity
>>> >
>>> > _______________________________________________
>>> >
>>> > clamav-users mailing list
>>> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net
>>> >
>>> > https://lists.clamav.net/mailman/listinfo/clamav-users
>>> >
>>> >
>>> > Help us build a comprehensive ClamAV guide:
>>> > https://github.com/vrtadmin/clamav-faq
>>> >
>>> > http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users@lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>> *DISCLAIMER*
>>>
>>> The information contained in this email and any attachments are
>>> confidential. It is intended solely for the individual or entity to whom
>>> they are addressed. Access to this email by anyone else is unauthorized.
>>>
>>> If you are not the intended recipient, any disclosure, copying,
>>> distribution or any action taken or omitted to be taken in reliance on it,
>>> is prohibited and may be unlawful. If you have received this communication
>>> in error, please notify us immediately by responding to this email and then
>>> delete it from your system.
>>>
>>> The Red Flag Group is neither liable for the proper and complete
>>> transmission of the information contained in this communication nor for any
>>> delay in its receipt.
>>>
>>> Any advice, recommendations or opinion contained within this email or
>>> its attachments are not to be construed as legal advice.
>>>
>>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>
>
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

Mark, Kevin,

I’m glad to hear that your load and scan times are back down to reasonable levels.

We’ll continue to investigate the best, safest way to add the phishing detection in a way that is fast and optional.

Regards,
Micah

From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
Date: Thursday, April 18, 2019 at 6:09 AM
To: ClamAV users ML <clamav-users@lists.clamav.net>
Cc: Mark Allan <markjallan@gmail.com>
Subject: Re: [clamav-users] [External] Re: Scan very slow

Fantastic! I can also confirm that scan times are back to normal now - more-or-less back to what they were in early February.

The time for one of our FP test volumes which I've been referencing in this thread is back down to 3m 30s, and the total time for our full FP test is back down from several hours to just 47 minutes.

Thank you!
Mark

On Thu, 18 Apr 2019 at 09:46, Al Varnell via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
Looks like all Phish.Phishing.REPHISH_ID_... signatures were dropped by daily-25423 today.

-Al-


On Apr 17, 2019, at 04:02, Al Varnell <alvarnell@mac.com<mailto:alvarnell@mac.com>> wrote:

There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in daily.ldb

-Al-


On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broekman@gmail.com<mailto:maarten.broekman@gmail.com>> wrote:

Are the "Phish" REPHISH signatures still in the daily or were they removed as well? Those were causing part of the issue.


--Maarten

On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more.

-Al-


On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

Hi Micah,

Sorry to pester you, but have you any update on when the remaining Phishtank signatures will be getting removed? It would be really great to get scan times properly back to normal.

Best regards
Mark

On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd@cisco.com<mailto:micasnyd@cisco.com>> wrote:
Mark,

Yes, the plan is still to remove the rest of the Phishtank signatures. We wanted to get things back to relative normal and resolve the immediate crisis. We’ll remove the rest of them soon.

Best,
Micah

From: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
Date: Tuesday, April 9, 2019 at 6:26 AM
To: "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Cc: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Subject: Re: [External] Re: [clamav-users] Scan very slow

The scan times are definitely better than they were - in fact, they're back to how they were before last week's inclusion of the Phishtank signatures. They're still almost double what they used to be though, and as far as I can see, there are still almost 4000 Phishtank signatures in the DB:
$ sigtool --find Phishtank | wc -l
3968

Can I request that those ones also be removed please?

Best regards
Mark

On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd@cisco.com<mailto:micasnyd@cisco.com>> wrote:
Tim,

There are a couple of ways for users to drop specific categories of signatures at this time. Sadly, they wouldn’t have helped this last week. These include bytecode signatures, PUA (potentially unwanted applications) signatures, Email.Phishing and HTML.Phishing signatures, and the Safebrowsing database.

If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or Email.Phishing.Phishtank then they could have been disabled with the clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).

Maybe a better option would be for us to create a new optional database for phishing signatures. However, the names for the databases are hardcoded into freshclam, so it is non-trivial to add a new database and would require a few changes to ClamAV’s code. We have talked about making the databases easier to add/remove in the future so users can have more categories to enable/disable. In this light, it ties in well with existing plans.

Of note the Phishtank sigs from Friday’s daily were removed yesterday and scan times should be back to normal.

Regards,
Micah

From: Tim Hawkins <tim.hawkins@redflaggroup.com<mailto:tim.hawkins@redflaggroup.com>>
Date: Friday, April 5, 2019 at 6:06 PM
To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>, Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Subject: Re: [External] Re: [clamav-users] Scan very slow

Hi Micah

Does clamav partition the database so that signatures that are mainly associated with email scanning can be dropped out for folks only needing filesystems scans, none of our systems use email, and we dont make use of the mailer extension.

Having to load all the email focused signatures could as you have observed impact performance.
Sent from Nine<http://www.9folders.com/>
________________________________
From: "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Sent: Saturday, April 6, 2019 03:18
To: ClamAV users ML; Mark Allan
Cc: Micah Snyder (micasnyd)
Subject: [External] Re: [clamav-users] Scan very slow

Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times.

Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time. One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317).

In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:

* Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds
* Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.

After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit.

The daily update tomorrow will have the change.

-Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.



From: clamav-users <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net>> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Date: Friday, April 5, 2019 at 1:08 PM
To: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>, ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Subject: Re: [clamav-users] Scan very slow

Hi Mark,

Sorry about the delay in responding. I hadn’t looked at my clamav-users filter this morning. Just investigating now. Will respond when I know more.

-Micah

From: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
Date: Friday, April 5, 2019 at 9:12 AM
To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>, "Micah Snyder (micasnyd)" <micasnyd@cisco.com<mailto:micasnyd@cisco.com>>
Subject: Re: [clamav-users] Scan very slow

Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net<http://lists.clamav.net/> isn't responding to http requests anyway)

It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane.

Yesterday's scan using
0.101.2:58:25409:1554370140:1:63:48554:328
took 7m 3s

On the same hardware, scanning the same read-only disk image, with today's scan using
0.101.2:58:25410:1554452941:1:63:48557:328
the scan time has jumped to 26m 15s

This is the longest it has ever taken to scan this volume (cf my previous email of 25th March)

Is there anything that can be excluded?

Best regards
Mark

On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:
Thanks Oya for the update. We will continue to investigate the signature performance issue.

Regards,
Micah

On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net> on behalf of oyamada@promark-inc.com<mailto:oyamada@promark-inc.com>> wrote:

Hi Micah

It seems that the scanning slow down issue of this time has been solved
at some level with CVD Update of the other day.
However, there is still big discrepancy in between the current condition and
the last condition in one month ago.

Date Files Scan time
2019/02/15 2550338 08:53:57
2019/03/15 2612792 19:22:54
2019/03/26 2634489 18:13:56
2019/03/27 2637201 18:10:05

We know the improvement of this time is due to the details of CVD, because
we did not make any change on the user's system.
We are going to try some tuning for scanning.

We like to know if you still have some room to make further improvement
for this slow down issue.
Thank you for your help, in advance.

Best regards,
Oya

On Mon, 25 Mar 2019 15:45:02 +0000
"Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> wrote:

> Hi Mark, all:
>
> I’m disappointed to hear that it is still slow for you.
>
> We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown. We have dropped them as of this past Saturday (https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates have been re-adding them with more specific scan target types. We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans.
>
> Regards,
> Micah
>
>
> From: clamav-users <clamav-users-bounces@lists.clamav.net<mailto:clamav-users-bounces@lists.clamav.net>> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Date: Monday, March 25, 2019 at 9:37 AM
> To: ClamAV users ML <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> Cc: Mark Allan <markjallan@gmail.com<mailto:markjallan@gmail.com>>
> Subject: Re: [clamav-users] Scan very slow
>
> Cheers Steve,
>
> In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be.
>
> Mark
>
> On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com><mailto:steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com>>> wrote:
> On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > Hi all,
> >
> te.
> >
> > Hopefully this helps someone to narrow things down a bit.
> >
> > Mark
> >
>
> 18/3/19 10m 49s TXT from DNS:
> 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>
> Here's the changes for the above update:
>
> https://lists.gt.net/clamav/virusdb/75154
>
> You can also check sigs quickly per update:
>
> https://lists.gt.net/clamav/virusdb/
>
>
>
> --
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>>
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


DISCLAIMER

The information contained in this email and any attachments are confidential. It is intended solely for the individual or entity to whom they are addressed. Access to this email by anyone else is unauthorized.

If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system.

The Red Flag Group is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.

Any advice, recommendations or opinion contained within this email or its attachments are not to be construed as legal advice.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml





_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

As of daily-25458, we've updated the Email.Phishing.VOF2 signatures such
that they should have better performance when scanning larger email files.

Specifically, the signatures each had a PCRE component that began by
looking for the string 'filename', and as it turns out, the PCRE library
will begin evaluating the regex more thoroughly each time the first
character in the regex is encountered in a file being scanned. It also
turns out that RTF files, which get embedded in emails as plain text, can
consist of a surprisingly large number of f's. In an email we were testing
with that had an embedded RTF file, the email was ~13 million bytes in
size, and ~10 million of those were the letter f! We modified the regex to
begin by looking for a semicolon, which is much less common in RTF files
and is not in the base64 character set.

Please let us know if you encounter any other cases of unreasonably slow
scan times, and we will do our best to investigate. Thank you!

-Andrew

Andrew Williams
Malware Research Team
Cisco Talos

On Wed, Apr 10, 2019 at 8:57 PM Micah Snyder (micasnyd) via clamav-users <
clamav-users@lists.clamav.net> wrote:

> JME,
>
> As you've pointed out, it appears that some signatures containing a PCRE
> regex components are responsible for slow scan times on larger email files.
>
> I did a bunch of profiling similar to what Maarten did earlier in order to
> narrow it down. I found that Email.Phishing.VOF2 signatures are performing
> slower with the eml sample you sent me. Email.Phishing.VOF2 signatures
> contain a PCRE regex component to alert on email attachments with specific
> names. Now that we've determined which signatures are performing slowly in
> these cases, I am hopeful that we will be able to optimize the
> Email.Phishing.VOF2 signatures to improve performance.
>
> I will note that your idea to lower the PCRERecMatchLimit setting to 1
> will effectively neuter all signatures that rely on regexes and so I can't
> recommend this.
>
> Regards,
> Micah
>
>
> ?On 4/10/19, 12:36 PM, "clamav-users on behalf of JME via clamav-users" <
> clamav-users-bounces@lists.clamav.net on behalf of
> clamav-users@lists.clamav.net> wrote:
>
> Helo,
>
> I managed to significantly reduce the problems of very long analysis,
> more than 400sec on some emails. Not by disabling PhishingSignatures that
> did not work. But putting: PCRERecMatchLimit to 1.
> The PCRE analyzes are thus bypassed, but SafeBrawsing and the other
> scans continue to work. Is it a mistake to precede as well?
>
> Regards,
> JME
>
> -----Message d'origine-----
> De : clamav-users <clamav-users-bounces@lists.clamav.net> De la part
> de Brent Clark via clamav-users
> Envoyé : mercredi 10 avril 2019 12:33
> À : ClamAV users ML <clamav-users@lists.clamav.net>
> Cc : Brent Clark <brentgclarklist@gmail.com>
> Objet : Re: [clamav-users] [External] Re: Scan very slow
>
> Thanks for doing this.
>
> What Im getting out of your feedback is that maybe you guys need to
> look to implementing or relooking at your CI process(es).
>
> Before pushing a commit, your CI can run the same test(s) and alert on
> slow or long running scans.
>
> All this can be automated and report on issues.
>
> I highly recommend to doing this, I dont think you guys realise how
> many systems are running and dependent on Clamav. Might be a good time to
> too remind the community and ask to support and donate for the project.
>
> HTH
>
> Regards
> Brent
>
> On 2019/04/09 17:58, Maarten Broekman via clamav-users wrote:
> > Clearly the latest daily.cvd is performing better, but the remaining
> > "Phishtank" sigs are _not_ a majority of the slowness.
> >
> > I unpacked the current (?) cvd (ClamAV-VDB:09 Apr 2019 03-53
> > -0400:25414:1548262:63:X:X:raynman:1554796413) and then ran a test
> > scan with each part to see what the load times looked like:
> >
> > daily.cdb ==== Time: 0.007 sec (0 m 0 s)
> > daily.cfg ==== Time: 0.004 sec (0 m 0 s)
> > daily.crb ==== Time: 0.006 sec (0 m 0 s)
> > *daily.cvd ==== Time: 11.384 sec (0 m 11 s)*
> > daily.fp ==== Time: 0.009 sec (0 m 0 s)
> > daily.ftm ==== Time: 0.005 sec (0 m 0 s)
> > daily.hdb ==== Time: 0.303 sec (0 m 0 s)
> > daily.hdu ==== Time: 0.006 sec (0 m 0 s)
> > daily.hsb ==== Time: 1.093 sec (0 m 1 s)
> > daily.hsu ==== Time: 0.005 sec (0 m 0 s)
> > daily.idb ==== Time: 0.006 sec (0 m 0 s)
> > *daily.ldb ==== Time: 5.563 sec (0 m 5 s)
> > *
> > daily.ldu ==== Time: 0.005 sec (0 m 0 s)
> > daily.mdb ==== Time: 0.061 sec (0 m 0 s)
> > daily.mdu ==== Time: 0.007 sec (0 m 0 s)
> > daily.msb ==== Time: 0.005 sec (0 m 0 s)
> > daily.msu ==== Time: 0.005 sec (0 m 0 s)
> > daily.ndb ==== Time: 0.017 sec (0 m 0 s)
> > daily.ndu ==== Time: 0.005 sec (0 m 0 s)
> > daily.pdb ==== Time: 0.010 sec (0 m 0 s)
> > daily.sfp ==== Time: 0.006 sec (0 m 0 s)
> > daily.wdb ==== Time: 0.014 sec (0 m 0 s)
> >
> > So, half the run time of a clamscan is from the daily.ldb. To break
> it
> > down farther, I split the daily.ldb into "daily_<virus>.ldb" where
> > <virus> is the first part of the dot-separated signature name.
> >
> > daily_Andr.ldb ==== Time: 0.008 sec (0 m 0 s)
> > daily_Archive.ldb ==== Time: 0.009 sec (0 m 0 s)
> > daily_Asp.ldb ==== Time: 0.004 sec (0 m 0 s)
> > daily_Doc.ldb ==== Time: 0.116 sec (0 m 0 s)
> > daily_Eicar-Test-Signature.ldb ==== Time: 0.009 sec (0 m 0 s)
> > daily_Email.ldb ==== Time: 0.014 sec (0 m 0 s)
> > daily_Emf.ldb ==== Time: 0.007 sec (0 m 0 s)
> > daily_Heuristics.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Html.ldb ==== Time: 0.010 sec (0 m 0 s)
> > daily_Hwp.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Img.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Ios.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Java.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Js.ldb ==== Time: 0.007 sec (0 m 0 s)
> > daily_Legacy.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Lnk.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Mp4.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Multios.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Osx.ldb ==== Time: 0.008 sec (0 m 0 s)
> > daily_Pdf.ldb ==== Time: 0.007 sec (0 m 0 s)
> > *daily_Phish.ldb ==== Time: 1.612 sec (0 m 1 s)*
> > daily_Phishtank.ldb ==== Time: 0.146 sec (0 m 0 s)
> > daily_Php.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Ppt.ldb ==== Time: 0.007 sec (0 m 0 s)
> > daily_Py.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Rtf.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Svg.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Swf.ldb ==== Time: 0.007 sec (0 m 0 s)
> > daily_Ttf.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Txt.ldb ==== Time: 0.009 sec (0 m 0 s)
> > daily_Unix.ldb ==== Time: 0.008 sec (0 m 0 s)
> > daily_Vbs.ldb ==== Time: 0.009 sec (0 m 0 s)
> > *daily_Win.ldb ==== Time: 3.391 sec (0 m 3 s)*
> > daily_Xls.ldb ==== Time: 0.009 sec (0 m 0 s)
> > daily_Xml.ldb ==== Time: 0.007 sec (0 m 0 s)
> >
> >
> > "Phish.", not "Phishtank.", and "Win." are the longest run times.
> > Looking at the /number/ of signatures in each, the 'Phish.'
> signatures
> > are taking a disproportionate amount of time to load compared to the
> > other signatures:
> >
> > 216 daily_Andr.ldb
> > 3 daily_Archive.ldb
> > 1 daily_Asp.ldb
> > 2096 daily_Doc.ldb
> > 1 daily_Eicar-Test-Signature.ldb
> > 1017 daily_Email.ldb
> > 2 daily_Emf.ldb
> > 5 daily_Heuristics.ldb
> > 250 daily_Html.ldb
> > 1 daily_Hwp.ldb
> > 15 daily_Img.ldb
> > 6 daily_Ios.ldb
> > 16 daily_Java.ldb
> > 69 daily_Js.ldb
> > 27 daily_Legacy.ldb
> > 9 daily_Lnk.ldb
> > 1 daily_Mp4.ldb
> > 9 daily_Multios.ldb
> > 175 daily_Osx.ldb
> > 132 daily_Pdf.ldb
> > 2515 daily_Phish.ldb
> > 3516 daily_Phishtank.ldb
> > 18 daily_Php.ldb
> > 5 daily_Ppt.ldb
> > 3 daily_Py.ldb
> > 28 daily_Rtf.ldb
> > 1 daily_Svg.ldb
> > 103 daily_Swf.ldb
> > 2 daily_Ttf.ldb
> > 140 daily_Txt.ldb
> > 222 daily_Unix.ldb
> > 21 daily_Vbs.ldb
> > 43928 daily_Win.ldb
> > 165 daily_Xls.ldb
> > 8 daily_Xml.ldb
> >
> >
> > From the look of it, "Phish." has those REPHISH signatures. Those
> > signatures seem to be looking at any file (Target 0) and have
> > subsignatures that are combined to match depending on which filetype
> > they are 'looking' for (so, href for HTML files, %PDF, Subtype, and
> > URI objects for PDFs, etc) as opposed to the remaining Phishtank
> sigs
> > which seem to have a separate signature depending on the target type.
> >
> > Breaking up daily_Win into it's constituent sub-parts doesn't reveal
> > any particular culprit from just a simple scan timing though...
> >
> > daily_Win.Adware.ldb ==== Time: 0.013 sec (0 m 0 s)
> > daily_Win.Coinminer.ldb ==== Time: 0.009 sec (0 m 0 s)
> > daily_Win.Downloader.ldb ==== Time: 0.035 sec (0 m 0 s)
> > daily_Win.Dropper.ldb ==== Time: 0.240 sec (0 m 0 s)
> > daily_Win.Exploit.ldb ==== Time: 0.016 sec (0 m 0 s)
> > daily_Win.Ircbot.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Win.Keylogger.ldb ==== Time: 0.010 sec (0 m 0 s)
> > daily_Win.ldb ==== Time: 3.418 sec (0 m 3 s)
> > daily_Win.Macro.ldb ==== Time: 0.009 sec (0 m 0 s)
> > *daily_Win.Malware.ldb ==== Time: 0.731 sec (0 m 0 s)*
> > daily_Win.Packed.ldb ==== Time: 0.131 sec (0 m 0 s)
> > daily_Win.Packer.ldb ==== Time: 0.008 sec (0 m 0 s)
> > daily_Win.Phishing.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Win.Proxy.ldb ==== Time: 0.005 sec (0 m 0 s)
> > daily_Win.Ransomware.ldb ==== Time: 0.019 sec (0 m 0 s)
> > daily_Win.Spyware.ldb ==== Time: 0.006 sec (0 m 0 s)
> > daily_Win.Tool.ldb ==== Time: 0.008 sec (0 m 0 s)
> > *daily_Win.Trojan.ldb ==== Time: 0.582 sec (0 m 0 s)*
> > daily_Win.Virus.ldb ==== Time: 0.059 sec (0 m 0 s)
> > daily_Win.Worm.ldb ==== Time: 0.030 sec (0 m 0 s)
> >
> > 158 daily_Win.Adware.ldb
> > 14 daily_Win.Coinminer.ldb
> > 561 daily_Win.Downloader.ldb
> > 8084 daily_Win.Dropper.ldb
> > 216 daily_Win.Exploit.ldb
> > 9 daily_Win.Ircbot.ldb
> > 193 daily_Win.Keylogger.ldb
> > 43928 daily_Win.ldb
> > 1 daily_Win.Macro.ldb
> > * 14820 daily_Win.Malware.ldb*
> > 4209 daily_Win.Packed.ldb
> > 20 daily_Win.Packer.ldb
> > 2 daily_Win.Phishing.ldb
> > 4 daily_Win.Proxy.ldb
> > 500 daily_Win.Ransomware.ldb
> > 32 daily_Win.Spyware.ldb
> > 121 daily_Win.Tool.ldb
> > * 12051 daily_Win.Trojan.ldb*
> > 1967 daily_Win.Virus.ldb
> > 966 daily_Win.Worm.ldb
> >
> >
> > Malware and Trojan take the longest, but they also have a majority
> of
> > the signatures.
> >
> > On Tue, Apr 9, 2019 at 11:19 AM Steve Basford
> > <steveb_clamav@sanesecurity.com
> > <mailto:steveb_clamav@sanesecurity.com>>
> > wrote:
> >
> > On 2019-04-09 12:02, Brent Clark via clamav-users wrote:
> > > Cant those be adopted / managed by Sanesecurity?
> > >
> > > For all you know, those are already in Sanesecurity.
> >
> > They are... and have been for quite some time:
> >
> >
> > "The following databases are distributed by Sanesecurity, but
> produced
> > by Porcupine Signatures"
> >
> > phishtank.ndb.
> >
> > Briefly...
> >
> > Number of sigs in phishtank.ndb: 9,309
> >
> > eg:
> >
> > PhishTank.Phishing.6002281, matches:
> >
> > https://www.phishtank.com/phish_detail.php?phish_id=6002281
> >
> > So, there is going to be some possible cross over now that
> > Phish.Phishing.REPHISH_ID_20190404_67-6931549-0
> > type signatures names from PhishTank feed are in daily.ldb and
> > daily.ndb.
> >
> > I'll check back on the thread later.
> >
> > --
> > Cheers,
> >
> > Steve
> > Twitter: @sanesecurity
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net <mailto:
> clamav-users@lists.clamav.net>
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
> >
> >
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
> >
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>