Mailing List Archive

Hello,

> sigtool --find-sigs Doc.Trojan.Agent-6923124-0 | sigtool --decode-sigs
I don't understand why this signature is so long, and why it is based on
always changing variables.

--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.44.39.76.46
E-mail : aj@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom

Securiteinfo.com
La Sécurité Informatique - La Sécurité des Informations.
266, rue de Villers
60123 Bonneuil en Valois


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

I can reproduce the Malformed pattern problem with a file with just the one signature:

Xls.Downloader.Powload-6923120-0 which is an even longer one .

This is 4 signatures before Doc.Trojan.Agent-6923124-0 in daily.ldb

sigtool reports the wrong line numbering eg with a file with just Xls.Downloader.Powload-6923120-0 it reports
the problem as being on line 2. It seems to be 4 lines out when reporting on the whole daily.ldb

again sigtool --find Xls.Downloader.Powload-6923120-0 | sigtool --decode-sigs

doesn't show a problem.

clamscan --debug -d file_with_just_the_sig_above.ldb somefile
doesn't show a problem.

Xls.Downloader.Powload-6923120-0 turned up in daily 25410 which was when the problem started

Maybe sigtool --list can't handle long signatures in ClamAV 0.100.2

There does seem a pointlessness to signatures based upon exact variable names etc that are obfuscated
and likely will vary with each sample. A regex signature to get any variable name would be better.


David Shrimpton

________________________________________
From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Arnaud Jacques <webmaster@securiteinfo.com>
Sent: Saturday, April 6, 2019 12:27 AM
To: clamav-users@lists.clamav.net
Subject: Re: [clamav-users] Malformed pattern daily.ldb version 25410

Hello,

> sigtool --find-sigs Doc.Trojan.Agent-6923124-0 | sigtool --decode-sigs
I don't understand why this signature is so long, and why it is based on
always changing variables.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

I can recreate that same issue with daily cvd 25410, using ClamAV 0.100.1.
That was the first 0.100.X I had handy to do a quick test.
The problem is something specific to sigtool and only the list-sigs
feature. It does not affect clamscan or clamd, and does not affect the
--find-sigs option of sigtool.
We do ongoing signature load testing with several different versions of
ClamAV, but focus on scan testing.

It does still happen with the latest release so I'll talk with the team
about opening this as a bug.

Thanks for the report.

Dave R.

On Fri, Apr 5, 2019 at 11:12 AM David Shrimpton via clamav-users <
clamav-users@lists.clamav.net> wrote:

> I can reproduce the Malformed pattern problem with a file with just the
> one signature:
>
> Xls.Downloader.Powload-6923120-0 which is an even longer one .
>
> This is 4 signatures before Doc.Trojan.Agent-6923124-0 in daily.ldb
>
> sigtool reports the wrong line numbering eg with a file with just
> Xls.Downloader.Powload-6923120-0 it reports
> the problem as being on line 2. It seems to be 4 lines out when reporting
> on the whole daily.ldb
>
> again sigtool --find Xls.Downloader.Powload-6923120-0 | sigtool
> --decode-sigs
>
> doesn't show a problem.
>
> clamscan --debug -d file_with_just_the_sig_above.ldb somefile
> doesn't show a problem.
>
> Xls.Downloader.Powload-6923120-0 turned up in daily 25410 which was when
> the problem started
>
> Maybe sigtool --list can't handle long signatures in ClamAV 0.100.2
>
> There does seem a pointlessness to signatures based upon exact variable
> names etc that are obfuscated
> and likely will vary with each sample. A regex signature to get any
> variable name would be better.
>
>
> David Shrimpton
>
> ________________________________________
> From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of
> Arnaud Jacques <webmaster@securiteinfo.com>
> Sent: Saturday, April 6, 2019 12:27 AM
> To: clamav-users@lists.clamav.net
> Subject: Re: [clamav-users] Malformed pattern daily.ldb version 25410
>
> Hello,
>
> > sigtool --find-sigs Doc.Trojan.Agent-6923124-0 | sigtool --decode-sigs
> I don't understand why this signature is so long, and why it is based on
> always changing variables.
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>


--
---
Dave Raynor
Talos Security Intelligence and Research Group
draynor@sourcefire.com

On 4/5/2019 12:16 PM, David Raynor wrote:
> I can recreate that same issue with daily cvd 25410, using ClamAV
> 0.100.1. That was the first 0.100.X I had handy to do a quick test.
> The problem is something specific to sigtool and only the list-sigs
> feature. It does not affect clamscan or clamd, and does not affect the
> --find-sigs option of sigtool.
> We do ongoing signature load testing with several different versions
> of ClamAV, but focus on scan testing.
>
> It does still happen with the latest release so I'll talk with the
> team about opening this as a bug.
>
> Thanks for the report.

Hi David,

I wanted to add to this that we started having issues with this today
where freshclam is also saying the signatures are invalid as well as
clamd.  So 25410 with 0.100.3 appears to be quite non-functional.  For
example, if I run /usr/local/clamav/bin/freshclam --datadir=/tmp/3 with
0.100.3, it downloads, says it's invalid, waits 5 seconds and downloads
again eventually giving up after a few minutes:

Fri Apr  5 14:17:59 2019 -> *Current working dir is /tmp/3
Fri Apr  5 14:17:59 2019 -> *Max retries == 3
Fri Apr  5 14:17:59 2019 -> ClamAV update process started at Fri Apr  5
14:17:59 2019
Fri Apr  5 14:17:59 2019 -> *Using IPv6 aware code
Fri Apr  5 14:17:59 2019 -> *Querying current.cvd.clamav.net
Fri Apr  5 14:17:59 2019 -> *TTL: 1298
Fri Apr  5 14:17:59 2019 -> *Software version from DNS: 0.101.2
Fri Apr  5 14:17:59 2019 -> ^Your ClamAV installation is OUTDATED!
Fri Apr  5 14:17:59 2019 -> ^Local version: 0.100.3 Recommended version:
0.101.2
Fri Apr  5 14:17:59 2019 -> DON'T PANIC! Read
https://www.clamav.net/documents/upgrading-clamav
Fri Apr  5 14:17:59 2019 -> *Retrieving http://db.US.clamav.net/main.cvd
Fri Apr  5 14:17:59 2019 -> *Trying to download
http://db.US.clamav.net/main.cvd (IP: 104.16.219.84)
Fri Apr  5 14:18:12 2019 -> Downloading main.cvd [100%]
Fri Apr  5 14:18:12 2019 -> ^[LibClamAV] cli_cvdload: Corrupted CVD header
Fri Apr  5 14:18:12 2019 -> !Verification: Malformed database
Fri Apr  5 14:18:13 2019 -> *Querying main.0.93.0.0.6810DB54.ping.clamav.net
Fri Apr  5 14:18:13 2019 -> Trying again in 5 secs...
Fri Apr  5 14:18:18 2019 -> ClamAV update process started at Fri Apr  5
14:18:18 2019
...

Is there a way to go back to daily-25409, for example, other than using
backups?  I looked at the FAQ,
https://www.clamav.net/documents/clamav-virus-database-faq and didn't
have any thoughts on it.

Regards,

KAM


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

This appears to be a different problem than the sigtool --list problem on daily

I think it may be a problem with integrity of downloaded file and not an incompatibility
of that file with clamav version or something wrong with a sig in the file.
Testing the main.cvd file may be good first step.

It appears to be reported that the main.cvd downloaded is corrupted:

>> Fri Apr 5 14:17:59 2019 -> *Trying to download
>>http://db.US.clamav.net/main.cvd (IP: 104.16.219.84)
>> Fri Apr 5 14:18:12 2019 -> Downloading main.cvd [100%]
>> Fri Apr 5 14:18:12 2019 -> ^[LibClamAV] cli_cvdload: Corrupted CVD header
>>Fri Apr 5 14:18:12 2019 -> !Verification: Malformed database

Some things that may help debug:

# download the main.cvd manually eg if have unix wget or curl

wget http://db.US.clamav.net/main.cvd

# check the size , is it zero length or improbably small ? Did wget report errors.

# Test main.cvd with sigtool look for errors or sensible output as below.

sigtool --info main.cvd

File: main.cvd
Build time: 07 Jun 2017 17:38 -0400
Version: 58
Signatures: 4566249
Functionality level: 60
Builder: sigmgr
MD5: 57462fd73f1cfdb356b9dca66da2b732
Digital signature: KWRdhTG+Own6ohh0wn5+vqg1d8ULKCxxxQeKuSA155B3ijxBKgf+bV3IXPcmZrIBUDn1xi8FmyvB63UieykwN/Avq5mTjHIVO8zFnC7wVF7dhdcEYn9Nt+Pmk/HXXx0voylYkidvgZmrxI8jx4a/Re6n3hHQJoCZrkHM15GER8j
Verification OK.

# examine main.cvd with binary editor eg xxd
main.cvd should have a 512 byte header then a gzipped tar file containing the database files and a main.info
The header has : separated fields . About the 4th field should look like an md5sum like above 57462fd73f1cfdb356b9dca66da2b732.
This is the md5sum of the gz that follows the header. The header seems to end with space padding.
about the 5th field should look like a the value of Digital signature: above. You should see the Builder field eg sigmgr above.

I think sigtool has verified the signature above. If file has been altered then verification failure might be reported. eg
is db.US.clamav.net the real clamav mirror site or an imposter. WARNING if the file isn't verifying it may be malicious
eg a compression bomb , a malicious archive , an exploit against some of the tools below and it might be dangerous to run
some of the tools below against it. Remember only http was used not https to get the file so site might be bogus and file
could be anything.

# Extract gz from main.cvd eg with dd and calling the gz main.gz
ie strip off the 512 byte header at start

dd if=main.cvd of=main.gz skip=1 bs=512

# test the gz
gunzip -t main.gz

# extract gz (it will be large eg 3 times size of the gz on my example)
gunzip main.gz

# this should give a tar file called main for my example
# test the tar file (my tar reports improbable dates)
tar tvf main
---------- 0/0 17992 1970-01-01 10:00 COPYING
---------- 0/0 1060 1970-01-01 10:00 main.info
---------- 0/0 3649543 1970-01-01 10:00 main.hdb
---------- 0/0 24806499 1970-01-01 10:00 main.hsb
etc

# try extracting main.info and some of the database files
tar xf main main.info

# main.info contains sha256sum for each database file.
# test the extracted database files have same 256 sum
eg from main.info

main.sfp:87:ded8b3b340e2da8415f1409959abb54725afad137a66e938080c7c95a9413128

sha256sum main.sfp
ded8b3b340e2da8415f1409959abb54725afad137a66e938080c7c95a9413128 main.sfp

If a sha256 doesn't match that database file is corrupted or altered or main.sfp is wrong

You could look at a database file eg main.ndb with text editor or xxd ans should see lines looking
like clamav signatures. Try 'file main.ndb' first to make sure is a text file . Corrupted file might be binary
and trash your terminal or editor.

If the main.cvd appears to be OK then maybe the problem is it isn't compatible with clamav version.
You'd need to look at things like version and functionality level from the sigtool output and decide
if this is what is expected for a current main.cvd. If it is then I guess that incompatible main.cvd
or some faulty sig in main.cvd might be the issue.


>> Is there a way to go back to daily-25409, for example, other than using
backups? I looked at the FAQ,

If the main.cvd is corrupted I doubt freshclam would replace existing database files and
sigtool --version may show you are already on daily-25409 or earlier.

Note if running

freshclam --datadir

I think any settings other than database location from freshclam.conf would apply. So if you were just trying to
get an example main.cvd you might see side effects you don't want like freshclam writing to a configured log file
or trying to HUP your clamd or writing a mirrors.dat

David Shrimpton


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

On 4/5/2019 9:40 PM, David Shrimpton via clamav-users wrote:
> This appears to be a different problem than the sigtool --list problem on daily

Thanks for the impressive list of debug ideas.  Whatever this is, it's a
bug in clamav or an underlying library.  The machine with the issue is a
hodgepodge system.

Ran wget http://db.US.clamav.net/main.cvd

Here's the file size: 117892267

Here's the sha1: d275ad7d79af6ecf602d8813173a0bb0a0a00a88  main.cvd

All this is correct information but fails with I ran sigtool:

/usr/local/clamav/bin/sigtool --info main.cvd

File: main.cvd
Build time: 07 Jun 2017 17:38 -0400
Version: 58
Signatures: 4566249
Functionality level: 60
Builder: sigmgr
MD5: 57462fd73f1cfdb356b9dca66da2b732
Digital signature:
KWRdhTG+Own6ohh0wn5+vqg1d8ULKCxxxQeKuSA155B3ijxBKgf+bV3IXPcmZrIBUDn1xi8FmyvB63UieykwN/Avq5mTjHIVO8zFnC7wVF7dhdcEYn9Nt+Pmk/HXXx0voylYkidvgZmrxI8jx4a/Re6n3hHQJoCZrkHM15GER8j
LibClamAV Error: cli_cvdload: Corrupted CVD header
ERROR: cvdinfo: Verification: Malformed database


On another machine with the same version of clamav 0.100.3, it passes
sigtool:

/usr/local/clamav/bin/sigtool --info main.cvd
File: main.cvd
Build time: 07 Jun 2017 17:38 -0400
Version: 58
Signatures: 4566249
Functionality level: 60
Builder: sigmgr
MD5: 57462fd73f1cfdb356b9dca66da2b732
Digital signature:
KWRdhTG+Own6ohh0wn5+vqg1d8ULKCxxxQeKuSA155B3ijxBKgf+bV3IXPcmZrIBUDn1xi8FmyvB63UieykwN/Avq5mTjHIVO8zFnC7wVF7dhdcEYn9Nt+Pmk/HXXx0voylYkidvgZmrxI8jx4a/Re6n3hHQJoCZrkHM15GER8j
Verification OK.


Modifying cvd.c, I changed the CL_EMALFDB to be a little more specific
so I can see that the call to dbinfo = engine->dbinfo; is failing. 
After that, though, I need some pointers of what routine/class provides
that.  Maybe I can keep drilling down and find out what's got a bug
that's throwing a fit.

Regards,

KAM


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Is the failing machine running out of memory running engine = cl_engine_new()

David Shrimpton






_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml