Mailing List Archive

[clamav-users] Automated submissions to third party databases?
Hi there,

If you've been paying even scant attention to the list mail you'll
know that I've been doing some testing, particularly of clamd, when
it's used for scanning mail.

This is something of side issue, but I'll throw it into the pot to see
if anything comes of it.

The testing that I'm doing is for more than one purpose; there's clamd
itself (that is whether my patched version crashes, or whatever); and
there's the milter which feeds it. The milter isn't the one supplied
with ClamAV, it's one of my own written in pure Perl and it needs much
more thrashing than it's getting at the moment because I need it to be
reliable. And now, there's this side issue - which might blossom into
something which I think may be more interesting - the potential for an
automated submission system for messages which are certainly spam, but
for which the databases don't have a matching signature. It could go
well beyond that, but right now I don't want to get ahead of myself.

There seems to be some kind of a spammer campaign at the moment which
uses IPs from all over the planet to attempt to send much the same
kind of message. Normally I wouldn't see these messages, they'd be
rejected at the CONNECT stage after the connecting IP had been found
in nearly a dozen DNS block lists. But I'm desperate for more traffic
to test clamd and my milter, so I've configured the milter to allow a
message which has already triggered a REJECT response to reach all the
way to End Of Message, so that clamd can scan it. Then, after logging
the message text, even if clamd says "OK", I'll reject it anyway. If
nothing else it might slow them down a little. :)

So I'm flagging up quite a few messages which are guaranteed spam, but
which aren't in any of the third-party databases that I'm using. The
successes are all 'Sanesecurity.Junk.NNNNN', where 'NNNNN' is usually
a five-digit number beginning with '5'. The detection success rate is
in the region of 35% at present, so I'm collecting ~two out of three.

My milter can very easily process these messages, in any way, and then
send them, or the results of this processing, in any format and by any
means, to anyone who'd like to have that information. Once set up, it
could do it all in real time, without manual intervention at my end.

Any takers?

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Have you automated their upload to ClamAV.net using clamsubmit?

Sent from my ? iPhone

> On Sep 2, 2019, at 05:11, G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> wrote:
>
> ?Hi there,
>
> If you've been paying even scant attention to the list mail you'll
> know that I've been doing some testing, particularly of clamd, when
> it's used for scanning mail.
>
> This is something of side issue, but I'll throw it into the pot to see
> if anything comes of it.
>
> The testing that I'm doing is for more than one purpose; there's clamd
> itself (that is whether my patched version crashes, or whatever); and
> there's the milter which feeds it. The milter isn't the one supplied
> with ClamAV, it's one of my own written in pure Perl and it needs much
> more thrashing than it's getting at the moment because I need it to be
> reliable. And now, there's this side issue - which might blossom into
> something which I think may be more interesting - the potential for an
> automated submission system for messages which are certainly spam, but
> for which the databases don't have a matching signature. It could go
> well beyond that, but right now I don't want to get ahead of myself.
>
> There seems to be some kind of a spammer campaign at the moment which
> uses IPs from all over the planet to attempt to send much the same
> kind of message. Normally I wouldn't see these messages, they'd be
> rejected at the CONNECT stage after the connecting IP had been found
> in nearly a dozen DNS block lists. But I'm desperate for more traffic
> to test clamd and my milter, so I've configured the milter to allow a
> message which has already triggered a REJECT response to reach all the
> way to End Of Message, so that clamd can scan it. Then, after logging
> the message text, even if clamd says "OK", I'll reject it anyway. If
> nothing else it might slow them down a little. :)
>
> So I'm flagging up quite a few messages which are guaranteed spam, but
> which aren't in any of the third-party databases that I'm using. The
> successes are all 'Sanesecurity.Junk.NNNNN', where 'NNNNN' is usually
> a five-digit number beginning with '5'. The detection success rate is
> in the region of 35% at present, so I'm collecting ~two out of three.
>
> My milter can very easily process these messages, in any way, and then
> send them, or the results of this processing, in any format and by any
> means, to anyone who'd like to have that information. Once set up, it
> could do it all in real time, without manual intervention at my end.
>
> Any takers?
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Hi Joel,

On Mon, 2 Sep 2019, Joel Esler (jesler) wrote:
>
>> On Sep 2, 2019, at 05:11, G.W. Haywood via clamav-users ... wrote:
>>
>> ... I'm flagging up quite a few messages which are guaranteed spam,
>> but which aren't in any of the third-party databases that I'm using
>> ... My milter can very easily process these messages ... then send
>> ... the results ... to anyone who'd like to have that information.
>
> Have you automated their upload to ClamAV.net using clamsubmit?

Not yet, but as I said it would be easy to do.

This isn't the kind of thing I'd be comfortable to set up without first
discussing it with the recipients. For example, I'd want to check that
I won't be causing unnecessary work for any reason. If you think it's
OK for me to go ahead and submit some samples that way I'll be glad to.

Bear in mind that these are AFAICT purely spam, not viruses, although
I couldn't rule out malicious links and the like. It's depressing to
trawl through this stuff. Makes me feel we really should have stayed
in the trees.

Incidentally I seem to be having issues with the @cisco servers again
so I'm leaving that address out of the reply.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Hello Ged,

> So I'm flagging up quite a few messages which are guaranteed spam, but
> which aren't in any of the third-party databases that I'm using.  The
> successes are all 'Sanesecurity.Junk.NNNNN', where 'NNNNN' is usually
> a five-digit number beginning with '5'.  The detection success rate is
> in the region of 35% at present, so I'm collecting ~two out of three.

Did you try spam_marketing.ndb from securiteinfo.com ? We detect many
spams/phishing.

> My milter can very easily process these messages, in any way, and then
> send them, or the results of this processing, in any format and by any
> means, to anyone who'd like to have that information.  Once set up, it
> could do it all in real time, without manual intervention at my end.
>
> Any takers?

Sure, could you please send spam/phishing/malwares to
malware@surfezsanspub.fr ?
Thank you Ged !


--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.44.39.76.46
E-mail : aj@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom

Securiteinfo.com
La Sécurité Informatique - La Sécurité des Informations.
266, rue de Villers
60123 Bonneuil en Valois

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Hi there,

On Tue, 3 Sep 2019, Arnaud Jacques via clamav-users wrote:
> On Sep 2, 2019, at 05:11, G.W. Haywood via clamav-users wrote:
>
> > ... I'm flagging up quite a few messages which are guaranteed spam,
> > but which aren't in any of the third-party databases that I'm using
> > ... My milter can very easily process these messages ... then send
> > ... the results ... to anyone who'd like to have that information.
> > ...
> Did you try spam_marketing.ndb from securiteinfo.com ? We detect many
> spams/phishing.

Thanks - no, I don't use that one. It's listed at Sanesecurity as
having a high false positive rate.

> ... could you please send spam/phishing/malwares to
> malware@surfezsanspub.fr ?

I will set that up today, and also contact you off-list.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Ged,

>> Did you try spam_marketing.ndb from securiteinfo.com ? We detect many
>> spams/phishing.
>
> Thanks - no, I don't use that one.  It's listed at Sanesecurity as
> having a high false positive rate.

As far as I know, this review has not been updated since years.
We fight false positives as soons as we discover one. This is our priority.
Anyway, the best choice is to give a try, custom the signatures if
necessary, and make your own opinion, not only rely on 3rd party
evaluation from years ago.

About my own tests, on several mail servers, spam_marketing.ndb detects
a lot more spam and phishing than SaneSecurity signatures. No offense to
SaneSecurity, it is just my own opinion. spam_marketing.ndb does not
pretend to replace SaneSecurity, but is a complement.


>> ... could you please send spam/phishing/malwares to
>> malware@surfezsanspub.fr ?
>
> I will set that up today, and also contact you off-list.

Good ! Thank you very much.


--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.44.39.76.46
E-mail : aj@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom

Securiteinfo.com
La Sécurité Informatique - La Sécurité des Informations.
266, rue de Villers
60123 Bonneuil en Valois

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
On Tue, Sep 03, 2019 at 01:17:16PM +0200, Arnaud Jacques wrote:
> Ged,
>
> >>Did you try spam_marketing.ndb from securiteinfo.com ? We detect many
> >>spams/phishing.
> >
> >Thanks - no, I don't use that one.? It's listed at Sanesecurity as
> >having a high false positive rate.
>
> As far as I know, this review has not been updated since years.
> We fight false positives as soons as we discover one. This is our priority.
> Anyway, the best choice is to give a try, custom the signatures if
> necessary, and make your own opinion, not only rely on 3rd party evaluation
> from years ago.
>
> About my own tests, on several mail servers, spam_marketing.ndb detects a
> lot more spam and phishing than SaneSecurity signatures. No offense to
> SaneSecurity, it is just my own opinion. spam_marketing.ndb does not pretend
> to replace SaneSecurity, but is a complement.

General comment:

Using any third party rules with ClamAV is a gamble, but they are very good
for scoring with Amavisd/Spamassassin etc. In my setup I don't even trust
the official signatures, I just score everything along with SA.


_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
?On 9/3/19, 4:15 AM, "clamav-users on behalf of G.W. Haywood via clamav-users" <clamav-users-bounces@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote:

Hi Joel,

On Mon, 2 Sep 2019, Joel Esler (jesler) wrote:
>
>> On Sep 2, 2019, at 05:11, G.W. Haywood via clamav-users ... wrote:
>>
>> ... I'm flagging up quite a few messages which are guaranteed spam,
>> but which aren't in any of the third-party databases that I'm using
>> ... My milter can very easily process these messages ... then send
>> ... the results ... to anyone who'd like to have that information.
>
> Have you automated their upload to ClamAV.net using clamsubmit?

Not yet, but as I said it would be easy to do.

Let me know when you do? We'd like to take a look at what you're submitting.
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Hi there,

On Tue, 3 Sep 2019, Henrik K wrote:

> General comment:
>
> Using any third party rules with ClamAV is a gamble, but

Agreed. In fact I'd go further than that. Relying on something like
ClamAV is a gamble. If there's a new 0-day just out, there may be no
chance of spotting it at all. In my systems ClamAV is the last of the
filters, just a tweak in the already heavily weighted probabilities.
Of course I'm only talking about scanning mail.

> they are very good for scoring with Amavisd/Spamassassin etc. In my
> setup I don't even trust the official signatures, I just score
> everything along with SA.

While I'm very happy to trust official signatures, I do something very
similar with scores, early in the SMTP conversation. Here, under
normal circumstances, ninety-nine point some nines percent of the junk
is filtered out by nearly a dozen DNSBLs and a custom GeoIP database.
ClamAV flags something as 'FOUND' about once a year, because the other
filtering has already taken care of it before clamd even sees it.

I found SpamAssassin too complex for my liking, and it absorbed more
effort than I felt was justified by its efficacy. Using their mailing
list was a most unpleasant experience, although that was some years
ago now and things might well have improved. But I do have the luxury
of being able to write custom milters; without that, things would most
likely be different.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Hi Joel,

On Tue, 3 Sep 2019, Joel Esler (jesler) wrote:
> On Mon, 2 Sep 2019, Joel Esler (jesler) wrote:
> >> On Sep 2, 2019, at 05:11, G.W. Haywood via clamav-users ... wrote:
> >>
> >> ... I'm flagging up quite a few messages which are guaranteed spam,
> >> but which aren't in any of the third-party databases that I'm using
> >> ... My milter can very easily process these messages ... then send
> >> ... the results ... to anyone who'd like to have that information.
> >
> > Have you automated their upload to ClamAV.net using clamsubmit?
>
> Not yet, but as I said it would be easy to do.
>
> Let me know when you do? We'd like to take a look at what you're submitting.

Sure, I'll do that next chance I get. Just battling uninitialized
variables for Securiteinfo at the moment. :/

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: [clamav-users] Automated submissions to third party databases? [ In reply to ]
Hi Joel,

On Wed, 4 Sep 2019, G.W. Haywood wrote:

> ... some junk mails aren't being detected by clamd, even though
> there are valid signatures in the database that are supposed to
> match them.

I guess you have the two files which I attached. You can see below
what happens when I scan them using clamdscan. The one which is not
detected is as it came in on the wire today, and, when my milter sent
it to clamd as it arrived, it wasn't detected then either. The other
file is the same thing, but edited by me. You can see what's in them,
and if you compare them you will see the one change which I made which
allows the detection to succeed. Without knowing more I don't want to
say it's a fault in the scanner, but this looks strange to me.

8<----------------------------------------------------------------------
mail6:~$ >>> clamdscan /tmp/t16289.*
/tmp/t16289.found_1: Sanesecurity.Phishing.Fake.26520.UNOFFICIAL FOUND
/tmp/t16289.not_found_1: OK

----------- SCAN SUMMARY -----------
Infected files: 1
Time: 0.047 sec (0 m 0 s)
8<----------------------------------------------------------------------

I don't understand why one of them triggers a detection and the other
one doesn't. If anyone there can tell me I'd be glad to know. To be
clear, the change that I made is an example. It seems that there may
be many ways of getting the scan to succeed.

--

73,
Ged.

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml