Mailing List Archive: [clamav-users] Issue with clamav logical signature generation

[clamav-users] Issue with clamav logical signature generation

Feb 24, 2019, 3:12 PM

Post #1 of 8 (696 views)

Hello,

I am trying to generate clamav signatures for a malware dataset that I
have.

Initially I have recognized some strings which are prominent in a class of
malware, hence, those are considered and a ldb signature is generated using
the below method.

The name of the signature, Engine version, Target as 0. We further have 'x'
number of sub-signatures here x is 100 each with logical or. All the
strings are converted to hex representation. Below is the example which is
generated.

ramnit.Signature;Engine:0-500,Target:0;0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99;636f6e6e6;686b65795;363530393;52656c656;633a5c5c7;436f6e766;313937313;6c6f63616;576169744;363337363;686b65795;353238363;736c65657;633a5c5c7;636f6e6e6;686b65795;633a5c5c7;737663686;363030363;633a5c5c7;313935353;633a5c5c7;636f6e6e6;6765746d6;536574437;313933393;686b65795;633a5c5c7;323232363;353537363;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;353130363;64656c657;633a5c5c7;633a5c5c7;686b65795;53656e644;6b7975666;6c6f63616;494d41474;686b65795;686b65795;686b65795;696573716;737663686;313237303;363033353;363039383;686b65795;686b65795;633a5c5c7;686b65795;333139313;686b65795;437265617;686b65795;476574546;353631323;633a5c5c7;686b65795;496e74657;686b65795;686b65795;686b65795;686b65795;3f7365745;633a5c5c7;476574537;527063426;686b65795;686b65795;566572517;353630353;686b65795;4f70656e5;353138343;4c6f6f6b7;633a5c5c7;476574546;363139393;633a5c5c7;686b65795;353638333;676574707;6f6c65333;5065656b4;343230353;536574576;5c5c3f3f5;5265674f7;633a5c5c7;686b65795;686b65795

Now, the problem is in case there are <=65 sub-signatures then everything
works fine however, if they increase beyond that, it results in the
following error.

LibClamAV Error: cli_loadldb: The number of subsignatures (== 65) doesn't
match the IDs in the logical expression (== 100)
LibClamAV Error: Problem parsing database at line 1
LibClamAV Error: Can't load ramnit.ldb: Malformed database
ERROR: Malformed database

Is it that the ldb signatures are limited to only 65 conditions? If not
what causes this issue and how to solve it?

--
Satwik

Re: [clamav-users] Issue with clamav logical signature generation [ In reply to ]

clamav-users at lists

Feb 24, 2019, 6:35 PM

Post #2 of 8 (696 views)

Permalink

I can’t comment on whether or not there is a 65 field limit or not, though it appears to me to be obviously so based on your experience.

What I will comment on is that your approach is rather unique compared to most the .ldb signatures I’ve observed in the ClamAV database. Most all of the latter use ‘and' rather than ‘or' rules as well as longer ascii strings to ensure against False Positives. I have to believe there is a very significant chance that or’ing that many short strings together will result in many False Positive hits.

Additionally, the strings you provided appear to contain an extra digit. I thought hex strings always contain an even number of digits?

-Al-
ClamXAV User

> On Feb 24, 2019, at 3:12 PM, Satwik B via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Hello,
>
> I am trying to generate clamav signatures for a malware dataset that I have.
>
> Initially I have recognized some strings which are prominent in a class of malware, hence, those are considered and a ldb signature is generated using the below method.
>
> The name of the signature, Engine version, Target as 0. We further have 'x' number of sub-signatures here x is 100 each with logical or. All the strings are converted to hex representation. Below is the example which is generated.
>
> ramnit.Signature;Engine:0-500,Target:0;0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99;636f6e6e6;686b65795;363530393;52656c656;633a5c5c7;436f6e766;313937313;6c6f63616;576169744;363337363;686b65795;353238363;736c65657;633a5c5c7;636f6e6e6;686b65795;633a5c5c7;737663686;363030363;633a5c5c7;313935353;633a5c5c7;636f6e6e6;6765746d6;536574437;313933393;686b65795;633a5c5c7;323232363;353537363;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;353130363;64656c657;633a5c5c7;633a5c5c7;686b65795;53656e644;6b7975666;6c6f63616;494d41474;686b65795;686b65795;686b65795;696573716;737663686;313237303;363033353;363039383;686b65795;686b65795;633a5c5c7;686b65795;333139313;686b65795;437265617;686b65795;476574546;353631323;633a5c5c7;686b65795;496e74657;686b65795;686b65795;686b65795;686b65795;3f7365745;633a5c5c7;476574537;527063426;686b65795;686b65795;566572517;353630353;686b65795;4f70656e5;353138343;4c6f6f6b7;633a5c5c7;476574546;363139393;633a5c5c7;686b65795;353638333;676574707;6f6c65333;5065656b4;343230353;536574576;5c5c3f3f5;5265674f7;633a5c5c7;686b65795;686b65795
>
> Now, the problem is in case there are <=65 sub-signatures then everything works fine however, if they increase beyond that, it results in the following error.
>
> LibClamAV Error: cli_loadldb: The number of subsignatures (== 65) doesn't match the IDs in the logical expression (== 100)
> LibClamAV Error: Problem parsing database at line 1
> LibClamAV Error: Can't load ramnit.ldb: Malformed database
> ERROR: Malformed database
>
> Is it that the ldb signatures are limited to only 65 conditions? If not what causes this issue and how to solve it?
>
> --
> Satwik

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Issue with clamav logical signature generation [ In reply to ]

clamav-users at lists

Feb 25, 2019, 5:38 AM

Post #3 of 8 (696 views)

Permalink

I just took a peek at the code to try to answer your question. I'm not sure why, exactly, but yes there is an explicit MAX_LDB_SUBSIGS set to 64.

Regards,
-Micah

> On Feb 24, 2019, at 9:35 PM, Al Varnell via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> I can’t comment on whether or not there is a 65 field limit or not, though it appears to me to be obviously so based on your experience.
>
> What I will comment on is that your approach is rather unique compared to most the .ldb signatures I’ve observed in the ClamAV database. Most all of the latter use ‘and' rather than ‘or' rules as well as longer ascii strings to ensure against False Positives. I have to believe there is a very significant chance that or’ing that many short strings together will result in many False Positive hits.
>
> Additionally, the strings you provided appear to contain an extra digit. I thought hex strings always contain an even number of digits?
>
> -Al-
> ClamXAV User
>
>> On Feb 24, 2019, at 3:12 PM, Satwik B via clamav-users <clamav-users@lists.clamav.net> wrote:
>>
>> Hello,
>>
>> I am trying to generate clamav signatures for a malware dataset that I have.
>>
>> Initially I have recognized some strings which are prominent in a class of malware, hence, those are considered and a ldb signature is generated using the below method.
>>
>> The name of the signature, Engine version, Target as 0. We further have 'x' number of sub-signatures here x is 100 each with logical or. All the strings are converted to hex representation. Below is the example which is generated.
>>
>> ramnit.Signature;Engine:0-500,Target:0;0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99;636f6e6e6;686b65795;363530393;52656c656;633a5c5c7;436f6e766;313937313;6c6f63616;576169744;363337363;686b65795;353238363;736c65657;633a5c5c7;636f6e6e6;686b65795;633a5c5c7;737663686;363030363;633a5c5c7;313935353;633a5c5c7;636f6e6e6;6765746d6;536574437;313933393;686b65795;633a5c5c7;323232363;353537363;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;353130363;64656c657;633a5c5c7;633a5c5c7;686b65795;53656e644;6b7975666;6c6f63616;494d41474;686b65795;686b65795;686b65795;696573716;737663686;313237303;363033353;363039383;686b65795;686b65795;633a5c5c7;686b65795;333139313;686b65795;437265617;686b65795;476574546;353631323;633a5c5c7;686b65795;496e74657;686b65795;686b65795;686b65795;686b65795;3f7365745;633a5c5c7;476574537;527063426;686b65795;686b65795;566572517;353630353;686b65795;4f70656e5;353138343;4c6f6f6b7;633a5c5c7;476574546;363139393;633a5c5c7;686b65795;353638333;676574707;6f6c65333;5065656b4;343230353;536574576;5c5c3f3f5;5265674f7;633a5c5c7;686b65795;686b65795
>>
>> Now, the problem is in case there are <=65 sub-signatures then everything works fine however, if they increase beyond that, it results in the following error.
>>
>> LibClamAV Error: cli_loadldb: The number of subsignatures (== 65) doesn't match the IDs in the logical expression (== 100)
>> LibClamAV Error: Problem parsing database at line 1
>> LibClamAV Error: Can't load ramnit.ldb: Malformed database
>> ERROR: Malformed database
>>
>> Is it that the ldb signatures are limited to only 65 conditions? If not what causes this issue and how to solve it?
>>
>> --
>> Satwik
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Issue with clamav logical signature generation [ In reply to ]

clamav-users at lists

Feb 25, 2019, 10:44 AM

Post #4 of 8 (696 views)

Permalink

Hi there,

On Mon, 25 Feb 2019, Al Varnell wrote:

> ... the strings you provided appear to contain an extra digit. I
> thought hex strings always contain an even number of digits?

Just as decimal strings are strings composed of decimal digits and can
be any length, hexadecimal strings are strings composed of hexadecimal
digits - and can also be any length. They usually present as an even
number of digits only because they generally represent the even numbers
of four-bit binary numbers found in machine registers: 8-bit bytes and
16-bit words from decades ago, 32-bit, 64-bit and even 128-bit words
(e.g. for IPv6 addresses) in more recent times. I'm sure I did once
use 12-bit word lengths for some reason, but I can't now remember what
the hardware was.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Issue with clamav logical signature generation [ In reply to ]

clamav-users at lists

Feb 25, 2019, 5:13 PM

Post #5 of 8 (691 views)

Permalink

On Feb 25, 2019, at 10:44, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> Hi there,
>
> On Mon, 25 Feb 2019, Al Varnell wrote:
>
>> ... the strings you provided appear to contain an extra digit. I
>> thought hex strings always contain an even number of digits?
>
> Just as decimal strings are strings composed of decimal digits and can
> be any length, hexadecimal strings are strings composed of hexadecimal
> digits - and can also be any length. They usually present as an even
> number of digits only because they generally represent the even numbers
> of four-bit binary numbers found in machine registers: 8-bit bytes and
> 16-bit words from decades ago, 32-bit, 64-bit and even 128-bit words
> (e.g. for IPv6 addresses) in more recent times. I'm sure I did once
> use 12-bit word lengths for some reason, but I can't now remember what
> the hardware was.

Yes, of course that's correct and I said it wrong. When I analyzed those strings, they all appear to translate into four ascii characters, so I jumped to the conclusion that they were meant to all represent ascii and therefore would have an even number of digits. Here's what I found:

> connhkey6509Relec:\\Conv1971locaWait6376hkey5286sleec:\\connhkeyc:\\svch6006c:\\1955c:\\conngetmSetC1939hkeyc:\\22265576hkeyhkeyhkeyhkeyhkeyhkeyhkeyhkey5106delec:\\c:\\hkeySendkyuflocaIMAGhkeyhkeyhkeyiesqsvch127060356098hkeyhkeyc:\\hkey3191hkeyCreahkeyGetT5612c:\\hkeyIntehkeyhkeyhkeyhkey?setc:\\GetSRpcBhkeyhkeyVerQ5605hkeyOpen5184Lookc:\\GetT6199c:\\hkey5683getpole3Peek4205SetW\\??RegOc:\\hkeyhkey

-Al-
--
Al Varnell
Mountain View, CA

Re: [clamav-users] Issue with clamav logical signature generation [ In reply to ]

clamav-users at lists

Feb 27, 2019, 6:47 AM

Post #6 of 8 (691 views)

Permalink

You mother fucker. Stop sending spam.

Thank you
Jayanth D

On Mon, Feb 25, 2019 at 4:43 AM Satwik B via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hello,
>
> I am trying to generate clamav signatures for a malware dataset that I
> have.
>
> Initially I have recognized some strings which are prominent in a class of
> malware, hence, those are considered and a ldb signature is generated using
> the below method.
>
> The name of the signature, Engine version, Target as 0. We further have
> 'x' number of sub-signatures here x is 100 each with logical or. All the
> strings are converted to hex representation. Below is the example which is
> generated.
>
>
> ramnit.Signature;Engine:0-500,Target:0;0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92|93|94|95|96|97|98|99;636f6e6e6;686b65795;363530393;52656c656;633a5c5c7;436f6e766;313937313;6c6f63616;576169744;363337363;686b65795;353238363;736c65657;633a5c5c7;636f6e6e6;686b65795;633a5c5c7;737663686;363030363;633a5c5c7;313935353;633a5c5c7;636f6e6e6;6765746d6;536574437;313933393;686b65795;633a5c5c7;323232363;353537363;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;686b65795;353130363;64656c657;633a5c5c7;633a5c5c7;686b65795;53656e644;6b7975666;6c6f63616;494d41474;686b65795;686b65795;686b65795;696573716;737663686;313237303;363033353;363039383;686b65795;686b65795;633a5c5c7;686b65795;333139313;686b65795;437265617;686b65795;476574546;353631323;633a5c5c7;686b65795;496e74657;686b65795;686b65795;686b65795;686b65795;3f7365745;633a5c5c7;476574537;527063426;686b65795;686b65795;566572517;353630353;686b65795;4f70656e5;353138343;4c6f6f6b7;633a5c5c7;476574546;363139393;633a5c5c7;686b65795;353638333;676574707;6f6c65333;5065656b4;343230353;536574576;5c5c3f3f5;5265674f7;633a5c5c7;686b65795;686b65795
>
> Now, the problem is in case there are <=65 sub-signatures then everything
> works fine however, if they increase beyond that, it results in the
> following error.
>
> LibClamAV Error: cli_loadldb: The number of subsignatures (== 65) doesn't
> match the IDs in the logical expression (== 100)
> LibClamAV Error: Problem parsing database at line 1
> LibClamAV Error: Can't load ramnit.ldb: Malformed database
> ERROR: Malformed database
>
> Is it that the ldb signatures are limited to only 65 conditions? If not
> what causes this issue and how to solve it?
>
> --
> Satwik
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

Re: [clamav-users] Issue with clamav logical signature generation [ In reply to ]

tschmidt at cardtech

Feb 28, 2019, 8:33 AM

Post #7 of 8 (689 views)

Permalink

Am 25.02.19 um 19:44 schrieb G.W. Haywood via clamav-users:
> Just as decimal strings are strings composed of decimal digits and can
> be any length, hexadecimal strings are strings composed of hexadecimal
> digits - and can also be any length. They usually present as an even
> number of digits only because they generally represent the even numbers
> of four-bit binary numbers found in machine registers: 8-bit bytes and
> 16-bit words from decades ago, 32-bit, 64-bit and even 128-bit words
> (e.g. for IPv6 addresses) in more recent times. I'm sure I did once
> use 12-bit word lengths for some reason, but I can't now remember what
> the hardware was.

Can't remember using 12 bit words, but one of the first machines I used,
a CDC 6600, had 60 bit data words and 18 bit addresses.
Text was normally uppercase-only, stored in 6-bit bytes ten characters
per word. But if you wanted to use lower case you could use a different
encoding which would occupy two of the 6-bit bytes per character, so in
a way you had 12-bit bytes.

Of course octal, not hex, was used to represent binary values back then.

Oh, and punchcards had 12 rows so a binary image of a punchcard would
actually consist of 80 12-bit words. (Or bytes.)

Tilman

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Issue with clamav logical signature generation [ In reply to ]

clamav-users at lists

Feb 28, 2019, 8:43 AM

Post #8 of 8 (689 views)

Permalink

> Can't remember using 12 bit words, but one of the first machines I used,
> a CDC 6600, had 60 bit data words and 18 bit addresses.
> Text was normally uppercase-only, stored in 6-bit bytes ten characters
> per word. But if you wanted to use lower case you could use a different
> encoding which would occupy two of the 6-bit bytes per character, so in
> a way you had 12-bit bytes

An instructor called it Consistency Doesn't Count (CDC).
That was the big machine on campus back in the day.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Mailing List Archive

Mailing List Archive

Attached Files: