Mailing List Archive

Bug with .fp file being ignored
Hi,

I think there's a bug with ClamAV not honouring the contents of a .fp file
within the database directory.

I've tested 0.101.2 as well as previous versions of ClamAV going back to
0.99.4 and the issue seems to have appeared as of 0.100.0 onwards.

To re-create the issue:

Find a zip file which you know reports an infection when scanned.
Use sigtool --md5 to generate an FP sig of the zip file and save it in a
<filename>.fp file in the databse directory.
Use clamscan to scan the file and see that it still reports the file as
being infected.


The output from clamscan --debug shows the .fp file is being loaded, but it
just doesn't seem to be being honoured for some reason.

I see the same thing when I build ClamAV on macOS as well as when using the
apt-get distribution on Ubuntu 18.04

Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg
etc. Simple files are excluded as expected - similarly, if you generate an
FP sig of a simple file and put that file within an archive, it correctly
gets excluded.

I'll clone the source from Git on Monday and have a dig through it myself
to see if I can fix the bug, but thought I'd mention it here in case
someone's already on it, or at least knows where I can start looking!

Cheers
Mark
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Bug with .fp file being ignored [ In reply to ]
OK, so tracking this one down took longer than I like to admit!

The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017.

Attached are patches for devel/HEAD as well as the stable 0.101.2

Tests show that the issue is fixed and doesn't appear to introduce any false negatives.....however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below.

Does anyone know how to fix that duplicate output?

Cheers
Mark

virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND
virus-2009-04-13-id0007662101.zip: OK

----------- SCAN SUMMARY -----------
Known viruses: 6168730
Engine version: 0.101.2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.02 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 33.865 sec (0 m 33 s)
Re: Bug with .fp file being ignored [ In reply to ]
Thanks for tracking this down Mark. Sorry we didn’t respond earlier. It has been a crazy couple weeks over here. Will take a look at the issue and your patches soon.

-Micah

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.



From: clamav-devel <clamav-devel-bounces@lists.clamav.net> on behalf of Mark Allan <markjallan@gmail.com>
Reply-To: ClamAV Development <clamav-devel@lists.clamav.net>
Date: Wednesday, July 17, 2019 at 11:38 AM
To: ClamAV Development <clamav-devel@lists.clamav.net>
Subject: Re: [Clamav-devel] Bug with .fp file being ignored

OK, so tracking this one down took longer than I like to admit!

The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017.

Attached are patches for devel/HEAD as well as the stable 0.101.2

Tests show that the issue is fixed and doesn't appear to introduce any false negatives.....however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below.

Does anyone know how to fix that duplicate output?

Cheers
Mark

virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND
virus-2009-04-13-id0007662101.zip: OK

----------- SCAN SUMMARY -----------
Known viruses: 6168730
Engine version: 0.101.2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.02 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 33.865 sec (0 m 33 s)



> On 12 Jul 2019, at 11:07 pm, Mark Allan <markjallan@gmail.com> wrote:
>
> Hi,
>
> I think there's a bug with ClamAV not honouring the contents of a .fp file within the database directory.
>
> I've tested 0.101.2 as well as previous versions of ClamAV going back to 0.99.4 and the issue seems to have appeared as of 0.100.0 onwards.
>
> To re-create the issue:
>
> Find a zip file which you know reports an infection when scanned.
> Use sigtool --md5 to generate an FP sig of the zip file and save it in a <filename>.fp file in the databse directory.
> Use clamscan to scan the file and see that it still reports the file as being infected.
>
>
> The output from clamscan --debug shows the .fp file is being loaded, but it just doesn't seem to be being honoured for some reason.
>
> I see the same thing when I build ClamAV on macOS as well as when using the apt-get distribution on Ubuntu 18.04
>
> Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg etc. Simple files are excluded as expected - similarly, if you generate an FP sig of a simple file and put that file within an archive, it correctly gets excluded.
>
> I'll clone the source from Git on Monday and have a dig through it myself to see if I can fix the bug, but thought I'd mention it here in case someone's already on it, or at least knows where I can start looking!
>
> Cheers
> Mark
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Bug with .fp file being ignored [ In reply to ]
Hi Mark,

Did you have any luck identifying the source of the bug? I admit I bookmarked your email and failed to find time to look into it myself after that.

-Micah

?On 7/12/19, 6:09 PM, "clamav-devel on behalf of Mark Allan" <clamav-devel-bounces@lists.clamav.net on behalf of markjallan@gmail.com> wrote:

Hi,

I think there's a bug with ClamAV not honouring the contents of a .fp file
within the database directory.

I've tested 0.101.2 as well as previous versions of ClamAV going back to
0.99.4 and the issue seems to have appeared as of 0.100.0 onwards.

To re-create the issue:

Find a zip file which you know reports an infection when scanned.
Use sigtool --md5 to generate an FP sig of the zip file and save it in a
<filename>.fp file in the databse directory.
Use clamscan to scan the file and see that it still reports the file as
being infected.


The output from clamscan --debug shows the .fp file is being loaded, but it
just doesn't seem to be being honoured for some reason.

I see the same thing when I build ClamAV on macOS as well as when using the
apt-get distribution on Ubuntu 18.04

Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg
etc. Simple files are excluded as expected - similarly, if you generate an
FP sig of a simple file and put that file within an archive, it correctly
gets excluded.

I'll clone the source from Git on Monday and have a dig through it myself
to see if I can fix the bug, but thought I'd mention it here in case
someone's already on it, or at least knows where I can start looking!

Cheers
Mark
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Bug with .fp file being ignored [ In reply to ]
Hi Micah,

Yes I did, and I submitted a patch back in July but there was another related issue which I wasn't able to fix. I've copied my email below with the patches.

Best regards
Mark
---

The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017.

Attached are patches for devel/HEAD as well as the stable 0.101.2

Tests show that the issue is fixed and doesn't appear to introduce any false negatives.....however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below.

Does anyone know how to fix that duplicate output?

Cheers
Mark

virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND
virus-2009-04-13-id0007662101.zip: OK

----------- SCAN SUMMARY -----------
Known viruses: 6168730
Engine version: 0.101.2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.02 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 33.865 sec (0 m 33 s)
Re: Bug with .fp file being ignored [ In reply to ]
Mark,

Well that’s embarrassing. Looking at the patch now, it doesn’t seem like the right fix for the issue – given the remaining bug where it alerts and then reports clean afterwards.

It looks like cli_checkfp() is called within cli_append_virus(), and you’d think that would care of it – except that If you grep for “cli_append_virus”, you’ll notice that there are many cases where the return value is ignored. I suspect the real issue is that most, if not all, of those cases should check that the return value is still CL_VIRUS to honor the verdict from cli_checkfp().

I’ll put in a ticket to investigate.

-Micah

From: clamav-devel <clamav-devel-bounces@lists.clamav.net> on behalf of Mark Allan <markjallan@gmail.com>
Reply-To: ClamAV Development <clamav-devel@lists.clamav.net>
Date: Friday, September 20, 2019 at 4:47 PM
To: ClamAV Development <clamav-devel@lists.clamav.net>
Subject: Re: [Clamav-devel] Bug with .fp file being ignored

Hi Micah,

Yes I did, and I submitted a patch back in July but there was another related issue which I wasn't able to fix. I've copied my email below with the patches.

Best regards
Mark
---

The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017.

Attached are patches for devel/HEAD as well as the stable 0.101.2

Tests show that the issue is fixed and doesn't appear to introduce any false negatives.....however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below.

Does anyone know how to fix that duplicate output?

Cheers
Mark

virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND
virus-2009-04-13-id0007662101.zip: OK

----------- SCAN SUMMARY -----------
Known viruses: 6168730
Engine version: 0.101.2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.02 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 33.865 sec (0 m 33 s)



> On 20 Sep 2019, at 10:29 am, Micah Snyder (micasnyd) <micasnyd@cisco.com> wrote:
>
> Hi Mark,
>
> Did you have any luck identifying the source of the bug? I admit I bookmarked your email and failed to find time to look into it myself after that.
>
> -Micah
>
> On 7/12/19, 6:09 PM, "clamav-devel on behalf of Mark Allan" <clamav-devel-bounces@lists.clamav.net on behalf of markjallan@gmail.com> wrote:
>
> Hi,
>
> I think there's a bug with ClamAV not honouring the contents of a .fp file
> within the database directory.
>
> I've tested 0.101.2 as well as previous versions of ClamAV going back to
> 0.99.4 and the issue seems to have appeared as of 0.100.0 onwards.
>
> To re-create the issue:
>
> Find a zip file which you know reports an infection when scanned.
> Use sigtool --md5 to generate an FP sig of the zip file and save it in a
> <filename>.fp file in the databse directory.
> Use clamscan to scan the file and see that it still reports the file as
> being infected.
>
>
> The output from clamscan --debug shows the .fp file is being loaded, but it
> just doesn't seem to be being honoured for some reason.
>
> I see the same thing when I build ClamAV on macOS as well as when using the
> apt-get distribution on Ubuntu 18.04
>
> Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg
> etc. Simple files are excluded as expected - similarly, if you generate an
> FP sig of a simple file and put that file within an archive, it correctly
> gets excluded.
>
> I'll clone the source from Git on Monday and have a dig through it myself
> to see if I can fix the bug, but thought I'd mention it here in case
> someone's already on it, or at least knows where I can start looking!
>
> Cheers
> Mark
> _______________________________________________
>
> clamav-devel mailing list
> clamav-devel@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-devel
>
> Please submit your patches to our Bugzilla: http://bugzilla.clamav.net
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
> _______________________________________________
>
> clamav-devel mailing list
> clamav-devel@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-devel
>
> Please submit your patches to our Bugzilla: http://bugzilla.clamav.net
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Bug with .fp file being ignored [ In reply to ]
Hi Mark,

I wanted to report that I believe we've properly fixed this issue last week. It looked to be the same issue as https://bugzilla.clamav.net/show_bug.cgi?id=12217 where I've linked to the commits where we did the work.

Sorry it's taken so long, but glad we'll finally have FP signatures working properly in 0.103 ????

-Micah

-----Original Message-----
From: clamav-devel <clamav-devel-bounces@lists.clamav.net> On Behalf Of Mark Allan
Sent: Friday, July 12, 2019 3:08 PM
To: ClamAV Development <clamav-devel@lists.clamav.net>
Subject: [Clamav-devel] Bug with .fp file being ignored

Hi,

I think there's a bug with ClamAV not honouring the contents of a .fp file within the database directory.

I've tested 0.101.2 as well as previous versions of ClamAV going back to
0.99.4 and the issue seems to have appeared as of 0.100.0 onwards.

To re-create the issue:

Find a zip file which you know reports an infection when scanned.
Use sigtool --md5 to generate an FP sig of the zip file and save it in a <filename>.fp file in the databse directory.
Use clamscan to scan the file and see that it still reports the file as being infected.


The output from clamscan --debug shows the .fp file is being loaded, but it just doesn't seem to be being honoured for some reason.

I see the same thing when I build ClamAV on macOS as well as when using the apt-get distribution on Ubuntu 18.04

Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg etc. Simple files are excluded as expected - similarly, if you generate an FP sig of a simple file and put that file within an archive, it correctly gets excluded.

I'll clone the source from Git on Monday and have a dig through it myself to see if I can fix the bug, but thought I'd mention it here in case someone's already on it, or at least knows where I can start looking!

Cheers
Mark
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Bugzilla: http://bugzilla.clamav.net

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
_______________________________________________

clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel

Please submit your patches to our Github: https://github.com/Cisco-Talos/clamav-devel/pulls

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml