Mailing List Archive

ClamAV Algorithms
Hi,

I am a newbie to ClamAV so require help please. I am doing a research project and would like to evaluate the BM and AC algorithms used by ClamAV. Is there anyway to get ClamAV to use either BM or AC on their own so scanning speed tests can be conducted for each algorithm separately? I have read on this list that you can't get BM to run on its own.

What is the best way to compare the two algorithms and can someone give me more information on how ClamAV uses or chooses between BM and AC please?

Thanks in advance,

Jerry



_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: ClamAV Algorithms [ In reply to ]
On 2011-07-11 01:00, Jerry 270 wrote:
>
> Hi,
>
> I am a newbie to ClamAV so require help please. I am doing a research project and would like to evaluate the BM and AC algorithms used by ClamAV. Is there anyway to get ClamAV to use either BM or AC on their own so scanning speed tests can be conducted for each algorithm separately? I have read on this list that you can't get BM to run on its own.
>
> What is the best way to compare the two algorithms and can someone give me more information on how ClamAV uses or chooses between BM and AC please?

There are some AC/BM discussion in the archive, see for example:
http://lurker.clamav.net/message/20100426.103047.eb6fd9d0.en.html
http://lurker.clamav.net/message/20100427.131931.b705e603.en.html
http://lurker.clamav.net/message/20081204.212941.c9fa45c2.en.html

You can use DevACOnly to use only the AC algorithm for everything (there is no equivalent for BM),

Other than that you can use tools such as 'oprofile', or 'perf record / perf report' to see how much time
is spent in functions from matcher-ac.c and how much time in those from matcher-bm.c.

Also note that there is the prefiltering step too, if you want to measure just the AC/BM performance, you should disable that
(although you'll loose performance by doing so).

What are your research project's goals?

Best regards,
--Edwin

_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: ClamAV Algorithms [ In reply to ]
Hi Edwin,

Thanks for your reply. I am doing a Masters degree for which the research is analyzing & investigating malware. I am interested in evaluating algorithms used in anti-virus software, but just investigating whether this is a possibility at the moment. The research projects goal is to define a problem domain, a scenario in which the problem to be investigated exists. Within this problem domain, a research question is posed. This is the question that the project will seek to answer.

I enabled DevAVOnly and only the AC signatures appear to be loaded when the config file is reread but when I do a scan of some files the debug information appears to suggest that BM signatures are loaded for GENERIC and PE. What should DevACDepth be set to? How is prefiltering disabled?

If AC is used for signatures containing wildcards and BM is used for signatures without wildcards is it possible to scan using just one type of signature and test the performance of each algorithm that way?

Regards,

Jerry

> Date: Mon, 11 Jul 2011 10:28:30 +0300
> From: edwintorok@gmail.com
> To: clamav-devel@lists.clamav.net
> Subject: Re: [Clamav-devel] ClamAV Algorithms
>
> On 2011-07-11 01:00, Jerry 270 wrote:
> >
> > Hi,
> >
> > I am a newbie to ClamAV so require help please. I am doing a research project and would like to evaluate the BM and AC algorithms used by ClamAV. Is there anyway to get ClamAV to use either BM or AC on their own so scanning speed tests can be conducted for each algorithm separately? I have read on this list that you can't get BM to run on its own.
> >
> > What is the best way to compare the two algorithms and can someone give me more information on how ClamAV uses or chooses between BM and AC please?
>
> There are some AC/BM discussion in the archive, see for example:
> http://lurker.clamav.net/message/20100426.103047.eb6fd9d0.en.html
> http://lurker.clamav.net/message/20100427.131931.b705e603.en.html
> http://lurker.clamav.net/message/20081204.212941.c9fa45c2.en.html
>
> You can use DevACOnly to use only the AC algorithm for everything (there is no equivalent for BM),
>
> Other than that you can use tools such as 'oprofile', or 'perf record / perf report' to see how much time
> is spent in functions from matcher-ac.c and how much time in those from matcher-bm.c.
>
> Also note that there is the prefiltering step too, if you want to measure just the AC/BM performance, you should disable that
> (although you'll loose performance by doing so).
>
> What are your research project's goals?
>
> Best regards,
> --Edwin
>
> _______________________________________________
> http://lurker.clamav.net/list/clamav-devel.html
> Please submit your patches to our Bugzilla: http://bugs.clamav.net

_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: ClamAV Algorithms [ In reply to ]
On 07/12/2011 02:11 AM, Jerry 270 wrote:
>
> Hi Edwin,
>
> Thanks for your reply. I am doing a Masters degree for which the research is analyzing & investigating malware. I am interested in evaluating algorithms used in anti-virus software, but just investigating whether this is a possibility at the moment. The research projects goal is to define a problem domain, a scenario in which the problem to be investigated exists. Within this problem domain, a research question is posed. This is the question that the project will seek to answer.
>
> I enabled DevAVOnly and only the AC signatures appear to be loaded when the config file is reread but when I do a scan of some files the debug information appears to suggest that BM signatures are loaded for GENERIC and PE.

If you are using clamscan then use --dev-ac-only. I get 0 BM sigs:

LibClamAV debug: Matcher[0]: GENERIC: AC sigs: 35862 (reloff: 21, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 470 (ac_only mode)
LibClamAV debug: Using filter for trie 1
LibClamAV debug: Matcher[1]: PE: AC sigs: 59482 (reloff: 47699, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 468 (ac_only mode)
LibClamAV debug: Matcher[2]: OLE2: AC sigs: 1726 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 176 (ac_only mode)
LibClamAV debug: Matcher[3]: HTML: AC sigs: 5773 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 799 (ac_only mode)
LibClamAV debug: Using filter for trie 4
LibClamAV debug: Matcher[4]: MAIL: AC sigs: 1146 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 255 (ac_only mode)
LibClamAV debug: Matcher[5]: GRAPHICS: AC sigs: 23 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 227 (ac_only mode)
LibClamAV debug: Matcher[6]: ELF: AC sigs: 47 (reloff: 29, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 400 (ac_only mode)
LibClamAV debug: Using filter for trie 7
LibClamAV debug: Matcher[7]: ASCII: AC sigs: 1568 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 492 (ac_only mode)
LibClamAV debug: Matcher[8]: NOT USED: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[9]: MACH-O: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)


> What should DevACDepth be set to?
>
> If AC is used for signatures containing wildcards and BM is used for signatures without wildcards is it possible to scan using just one type of signature and test the performance of each algorithm that way?

Well you won't be able to load the AC signatures into BM (BM doesn't support the wildcards), so as a first step you would probably be to remove
the signatures that require AC from the DB.
You can use 'sigtool --unpack-current main' and 'sigtool --unpack-current daily' to unpack the databases.

And then load the DB as by default (into BM), and with --dev-ac-only (into AC), and compare them that way.

Also note that the BM algo has an optimization when signatures are tied to a specific offset (PE for example).

> How is prefiltering disabled?

Comment out this 'if' in matcher-ac.c:
if (cli_mtargets[root->type].enable_prefiltering && dconf_prefiltering) {

Best regards,
--Edwin
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Re: ClamAV Algorithms [ In reply to ]
Hi Edwin,

Thanks for your help. I used --dev-ac-only on the command line and that does the trick. I couldn't find anything about using --dev-ac-only in the manual page.

Regards,

Jerry

> Date: Thu, 14 Jul 2011 15:03:12 +0300
> From: edwintorok@gmail.com
> To: clamav-devel@lists.clamav.net
> Subject: Re: [Clamav-devel] ClamAV Algorithms
>
> On 07/12/2011 02:11 AM, Jerry 270 wrote:
> >
> > Hi Edwin,
> >
> > Thanks for your reply. I am doing a Masters degree for which the research is analyzing & investigating malware. I am interested in evaluating algorithms used in anti-virus software, but just investigating whether this is a possibility at the moment. The research projects goal is to define a problem domain, a scenario in which the problem to be investigated exists. Within this problem domain, a research question is posed. This is the question that the project will seek to answer.
> >
> > I enabled DevAVOnly and only the AC signatures appear to be loaded when the config file is reread but when I do a scan of some files the debug information appears to suggest that BM signatures are loaded for GENERIC and PE.
>
> If you are using clamscan then use --dev-ac-only. I get 0 BM sigs:
>
> LibClamAV debug: Matcher[0]: GENERIC: AC sigs: 35862 (reloff: 21, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 470 (ac_only mode)
> LibClamAV debug: Using filter for trie 1
> LibClamAV debug: Matcher[1]: PE: AC sigs: 59482 (reloff: 47699, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 468 (ac_only mode)
> LibClamAV debug: Matcher[2]: OLE2: AC sigs: 1726 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 176 (ac_only mode)
> LibClamAV debug: Matcher[3]: HTML: AC sigs: 5773 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 799 (ac_only mode)
> LibClamAV debug: Using filter for trie 4
> LibClamAV debug: Matcher[4]: MAIL: AC sigs: 1146 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 255 (ac_only mode)
> LibClamAV debug: Matcher[5]: GRAPHICS: AC sigs: 23 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 227 (ac_only mode)
> LibClamAV debug: Matcher[6]: ELF: AC sigs: 47 (reloff: 29, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 400 (ac_only mode)
> LibClamAV debug: Using filter for trie 7
> LibClamAV debug: Matcher[7]: ASCII: AC sigs: 1568 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 492 (ac_only mode)
> LibClamAV debug: Matcher[8]: NOT USED: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
> LibClamAV debug: Matcher[9]: MACH-O: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
>
>
> > What should DevACDepth be set to?
> >
> > If AC is used for signatures containing wildcards and BM is used for signatures without wildcards is it possible to scan using just one type of signature and test the performance of each algorithm that way?
>
> Well you won't be able to load the AC signatures into BM (BM doesn't support the wildcards), so as a first step you would probably be to remove
> the signatures that require AC from the DB.
> You can use 'sigtool --unpack-current main' and 'sigtool --unpack-current daily' to unpack the databases.
>
> And then load the DB as by default (into BM), and with --dev-ac-only (into AC), and compare them that way.
>
> Also note that the BM algo has an optimization when signatures are tied to a specific offset (PE for example).
>
> > How is prefiltering disabled?
>
> Comment out this 'if' in matcher-ac.c:
> if (cli_mtargets[root->type].enable_prefiltering && dconf_prefiltering) {
>
> Best regards,
> --Edwin
> _______________________________________________
> http://lurker.clamav.net/list/clamav-devel.html
> Please submit your patches to our Bugzilla: http://bugs.clamav.net

_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net