Mailing List Archive

Hostkarma Blacklist Climbing the Charts
For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist
comparison chart. Not a scientific comparison but it's about all there
is to compare blacklists. Now only abuseat.org and spamhaus have me
beat. (apews doesn't count because they blacklist everything)

http://www.sdsc.edu/~jeff/spam/cbc.html
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Hi!

> For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist
> comparison chart. Not a scientific comparison but it's about all there is to
> compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews
> doesn't count because they blacklist everything)
>
> http://www.sdsc.edu/~jeff/spam/cbc.html

Beat you with what, false positives? :-)

Indeed, it doesnt tell much about -quality- of a list. So its only maths.

Bye,
Raymond.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On Thu, 2009-07-09 at 18:57 -0700, Marc Perkel wrote:
> For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist
> comparison chart. Not a scientific comparison but it's about all there
> is to compare blacklists. Now only abuseat.org and spamhaus have me
> beat. (apews doesn't count because they blacklist everything)
>
> http://www.sdsc.edu/~jeff/spam/cbc.html
>
>
Zen still tops it - and rightly so. It's a fantastic list. The question
is how much longer is spamhaus going to exists after they lost that e360
case? Could it spell the end for them?

Barracuda always intended to charge for access to their list. It's been
free for around a year now and I wonder if and when that will happen. If
you take spamhaus and sorbs out of the frame it green lights the digital
shoplifters at Barracuda to start charging. Mind you, you have to laugh
at an organisation that buys in some of it's blacklist data and ends up
listing it's own customer barracuda devices LOL. Better hope that new
lists spring up and Hostkarma keeps climbing.

I don't have the experience of apews blacklisting everything. I've had
two hits from them in six months. They are at the bottom of my lookup
food chain, but I can't cite them as irresponsible in their listing.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 10-Jul-2009, at 01:25, richard@buzzhost.co.uk wrote:
> On Thu, 2009-07-09 at 18:57 -0700, Marc Perkel wrote:
>> For what it's worth I'm now ahead of Barracuda on Jeff Makey's
>> blacklist
>> comparison chart. Not a scientific comparison but it's about all
>> there
>> is to compare blacklists. Now only abuseat.org and spamhaus have me
>> beat. (apews doesn't count because they blacklist everything)
>>
>> http://www.sdsc.edu/~jeff/spam/cbc.html

> Zen still tops it - and rightly so. It's a fantastic list. The
> question
> is how much longer is spamhaus going to exists after they lost that
> e360
> case? Could it spell the end for them?

Spamhaus 'lost' that case a long time ago. It's made no difference,
and e360 no longer exists.

--
Otto: Apes don't read philosophy.
Wanda: Yes, they do Otto, they just don't understand it.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On Fri, 2009-07-10 at 04:57 -0600, LuKreme wrote:
> On 10-Jul-2009, at 01:25, richard@buzzhost.co.uk wrote:
> > On Thu, 2009-07-09 at 18:57 -0700, Marc Perkel wrote:
> >> For what it's worth I'm now ahead of Barracuda on Jeff Makey's
> >> blacklist
> >> comparison chart. Not a scientific comparison but it's about all
> >> there
> >> is to compare blacklists. Now only abuseat.org and spamhaus have me
> >> beat. (apews doesn't count because they blacklist everything)
> >>
> >> http://www.sdsc.edu/~jeff/spam/cbc.html
>
> > Zen still tops it - and rightly so. It's a fantastic list. The
> > question
> > is how much longer is spamhaus going to exists after they lost that
> > e360
> > case? Could it spell the end for them?
>
> Spamhaus 'lost' that case a long time ago. It's made no difference,
> and e360 no longer exists.
>
There is a load of noise in NANAE about the Court coming to a
compensation decision and Spamhaus being 'broke' hence my concern.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 10-Jul-2009, at 05:18, richard@buzzhost.co.uk wrote:
> There is a load of noise in NANAE about the Court coming to a
> compensation decision and Spamhaus being 'broke' hence my concern.

Is NANAE in a time-warp? The court (in the US) has no power to compel
spamhaus (in the UK) to pay a cent.


--
And now, the rest of the story
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On Fri, 2009-07-10 at 05:42 -0600, LuKreme wrote:
> On 10-Jul-2009, at 05:18, richard@buzzhost.co.uk wrote:
> > There is a load of noise in NANAE about the Court coming to a
> > compensation decision and Spamhaus being 'broke' hence my concern.
>
> Is NANAE in a time-warp? The court (in the US) has no power to compel
> spamhaus (in the UK) to pay a cent.

Don't you start! That's what the trolls are fighting about!
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
A more interesting comparison would be to see how much stuff is NOT caught
by spamhaus, but caught by your list or others.... :)

-C

On Thu, 9 Jul 2009, Marc Perkel wrote:
> For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist
> comparison chart. Not a scientific comparison but it's about all there is to
> compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews
> doesn't count because they blacklist everything)
>
> http://www.sdsc.edu/~jeff/spam/cbc.html
>
>
>
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Charles Gregory wrote:

> A more interesting comparison would be to see how much stuff is NOT
> caught by spamhaus, but caught by your list or others.... :)

Right -- that gives you more of a sense of the value of a new list for a
system which already checks other lists.

--
J.D. Falk
Return Path Inc
http://www.returnpath.net/
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Charles Gregory wrote:
>
> A more interesting comparison would be to see how much stuff is NOT
> caught by spamhaus, but caught by your list or others.... :)
>
> -C
>

Some stats from my mail server, zen.spamhaus is deployed at the smtp
level, so these are hits against 323 spam samples (as detected by SA)
that made it through smtp restrictions:

## DNSBL Statistics ##
102 RCVD_IN_UCE_COMBINED
95 RCVD_IN_BRBL
70 RCVD_IN_JMF_BL
62 RCVD_IN_UCEPROTECT1
49 RCVD_IN_UCEPROTECT2
40 RCVD_IN_UBL_UNSUB
34 RCVD_IN_UCEPROTECT3
32 RCVD_IN_SBLXBL
24 RCVD_IN_SORBS_WEB
21 RCVD_IN_BL_SPAMCOP_NET
17 RCVD_IN_PSBL
10 RCVD_IN_JMF_BR
9 RCVD_IN_IADB_SPF
9 RCVD_IN_IADB_LISTED
4 RCVD_IN_DNSWL_LOW
3 RCVD_IN_BSP_TRUSTED
2 RCVD_IN_SORBS_DUL
2 RCVD_IN_NJABL_RELAY
2 RCVD_IN_NJABL_PROXY
1 RCVD_IN_NJABL_SPAM
1 RCVD_IN_DNSWL_MED
323 Total Spam

UCE_COMBINED is a hit against any of UCEPROTECT 1, 2 or 3. In my
experience UCEPROTECT can and does give occasional FPs.

RCVD_IN_SBLXBL checks all IPs, not just last external, hence why we see
still see some hits even though zen.spamhaus is already used.

IMHO, BRBL and JMF_BL both do a good job at adding a little weight to
spam making it past zen.spamhaus.org. All the easy to detect stuff has
long since been blocked, so hits at this stage are against the last ~1%
of spam that has slipped past everything else, so don't judge the
apparent ~20% hit rate too harshly.

I still only trust spamhaus to outright reject mail at the smtp level.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 07/09/2009 09:57 PM, Marc Perkel wrote:
> For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist
> comparison chart. Not a scientific comparison but it's about all there
> is to compare blacklists. Now only abuseat.org and spamhaus have me
> beat. (apews doesn't count because they blacklist everything)
>
> http://www.sdsc.edu/~jeff/spam/cbc.html

Hi Marc,

http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists
Have you considered adding your RCVD_IN_JMF* rules to the sandbox so we
can easily compute some statistics from the weekly net masschecks?

For example...

http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail
http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail
http://psbl.surriel.com/

I've been following the weekly masscheck results on PSBL. The false
positives have helped us to identify problems in PSBL's trap filtering
logic and made it safer to use. The statistics are looking pretty good
now, so hopefully PSBL will become enabled by default in spamassassin-3.3.0.

Separate question for you:
Would you mind if PSBL used your whitelist and yellowlist to help
exclude false positive IP's?

Warren Togami
wtogami@redhat.com
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Warren Togami wrote:
> On 07/09/2009 09:57 PM, Marc Perkel wrote:
>> For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist
>> comparison chart. Not a scientific comparison but it's about all there
>> is to compare blacklists. Now only abuseat.org and spamhaus have me
>> beat. (apews doesn't count because they blacklist everything)
>>
>> http://www.sdsc.edu/~jeff/spam/cbc.html
>
> Hi Marc,
>
> http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists
>
> Have you considered adding your RCVD_IN_JMF* rules to the sandbox so
> we can easily compute some statistics from the weekly net masschecks?
>
> For example...
>
> http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail
> http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail
>
> http://psbl.surriel.com/
>
> I've been following the weekly masscheck results on PSBL. The false
> positives have helped us to identify problems in PSBL's trap filtering
> logic and made it safer to use. The statistics are looking pretty
> good now, so hopefully PSBL will become enabled by default in
> spamassassin-3.3.0.
>
> Separate question for you:
> Would you mind if PSBL used your whitelist and yellowlist to help
> exclude false positive IP's?
>
> Warren Togami
> wtogami@redhat.com
>

I'd be interested in how well it worked. Is there anything I need to do
to help?
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 09/28/2009 01:32 PM, Marc Perkel wrote:
>
>
> Warren Togami wrote:
>> On 07/09/2009 09:57 PM, Marc Perkel wrote:
>>> For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist
>>> comparison chart. Not a scientific comparison but it's about all there
>>> is to compare blacklists. Now only abuseat.org and spamhaus have me
>>> beat. (apews doesn't count because they blacklist everything)
>>>
>>> http://www.sdsc.edu/~jeff/spam/cbc.html
>>
>> Hi Marc,
>>
>> http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists
>>
>> Have you considered adding your RCVD_IN_JMF* rules to the sandbox so
>> we can easily compute some statistics from the weekly net masschecks?
>>
>> For example...
>>
>> http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail
>> http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail
>>
>> http://psbl.surriel.com/
>>
>> I've been following the weekly masscheck results on PSBL. The false
>> positives have helped us to identify problems in PSBL's trap filtering
>> logic and made it safer to use. The statistics are looking pretty good
>> now, so hopefully PSBL will become enabled by default in
>> spamassassin-3.3.0.
>
> I'd be interested in how well it worked. Is there anything I need to do
> to help?

1) I'm waiting to hear back what it will take for me to gain commit
access so I can add this to the sandbox.

2) Do you mind hundreds of thousands of rapid DNS lookups during
masschecks? If not then the two largest servers doing masschecks could
probably use rsync access to your data.

Warren Togami
wtogami@redhat.com
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Warren Togami wrote:
> On 09/28/2009 01:32 PM, Marc Perkel wrote:
>>
>>
>> Warren Togami wrote:
>>> On 07/09/2009 09:57 PM, Marc Perkel wrote:
>>>> For what it's worth I'm now ahead of Barracuda on Jeff Makey's
>>>> blacklist
>>>> comparison chart. Not a scientific comparison but it's about all there
>>>> is to compare blacklists. Now only abuseat.org and spamhaus have me
>>>> beat. (apews doesn't count because they blacklist everything)
>>>>
>>>> http://www.sdsc.edu/~jeff/spam/cbc.html
>>>
>>> Hi Marc,
>>>
>>> http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists
>>>
>>>
>>> Have you considered adding your RCVD_IN_JMF* rules to the sandbox so
>>> we can easily compute some statistics from the weekly net masschecks?
>>>
>>> For example...
>>>
>>> http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail
>>> http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail
>>>
>>>
>>> http://psbl.surriel.com/
>>>
>>> I've been following the weekly masscheck results on PSBL. The false
>>> positives have helped us to identify problems in PSBL's trap filtering
>>> logic and made it safer to use. The statistics are looking pretty good
>>> now, so hopefully PSBL will become enabled by default in
>>> spamassassin-3.3.0.
>>
>> I'd be interested in how well it worked. Is there anything I need to do
>> to help?
>
> 1) I'm waiting to hear back what it will take for me to gain commit
> access so I can add this to the sandbox.
>
> 2) Do you mind hundreds of thousands of rapid DNS lookups during
> masschecks? If not then the two largest servers doing masschecks
> could probably use rsync access to your data.
>
> Warren Togami
> wtogami@redhat.com
>

I think I have a lot of capacity. I suppose we'll see. I should be able
to handle the load. If not then I'll find out.

BTW - if JEF were included in the standard distribution, about how much
bandwidth and server power would I need to handle it?
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 09/28/2009 01:45 PM, Marc Perkel wrote:
>>>
>>> I'd be interested in how well it worked. Is there anything I need to do
>>> to help?
>>
>> 1) I'm waiting to hear back what it will take for me to gain commit
>> access so I can add this to the sandbox.
>>
>> 2) Do you mind hundreds of thousands of rapid DNS lookups during
>> masschecks? If not then the two largest servers doing masschecks could
>> probably use rsync access to your data.
>>
>> Warren Togami
>> wtogami@redhat.com
>>
>
> I think I have a lot of capacity. I suppose we'll see. I should be able
> to handle the load. If not then I'll find out.
>
> BTW - if JEF were included in the standard distribution, about how much
> bandwidth and server power would I need to handle it?
>

We don't really know how much traffic being default in spamassassin will
cause. You would have to ask the other list maintainers like DNSWL if
they have any statistics.

If PSBL becomes enabled by default in spamassassin-3.3.0 then we may be
able to estimate the jump in traffic from that.

Warren Togami
wtogami@redhat.com
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 09/28/2009 01:32 PM, Marc Perkel wrote:
>
> I'd be interested in how well it worked. Is there anything I need to do
> to help?

http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists
Could you provide a URL redirector to this page? This URL is very long. Perhaps shorter URL in the describe of each rule like: http://hostkarma.junkemailfilter.com ?

This URL will be in spam reports so folks can click-thru and see why their message triggered on this rule.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212
I filed the request to add it to sandbox for testing here. The line wrapping got screwed up in Bugzilla.

You might also want to consider standardizing the name of the blacklist. You called it JEF earlier in this thread. Your Wiki page calls the rules JMF. And it also seems to be called Hostkarma. It will be confusing to people if they see different names referring to the same thing. Perhaps we should call it JMF to avoid confusion?

Warren Togami
wtogami@redhat.com
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Warren Togami wrote:
> On 09/28/2009 01:32 PM, Marc Perkel wrote:
>>
>> I'd be interested in how well it worked. Is there anything I need to do
>> to help?
>
> http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists
> Could you provide a URL redirector to this page? This URL is very
> long. Perhaps shorter URL in the describe of each rule like:
> http://hostkarma.junkemailfilter.com ?
I'm working on that. Trying to figure out how to give it an A record.
>
> This URL will be in spam reports so folks can click-thru and see why
> their message triggered on this rule.
>
> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212
> I filed the request to add it to sandbox for testing here. The line
> wrapping got screwed up in Bugzilla.
>
> You might also want to consider standardizing the name of the
> blacklist. You called it JEF earlier in this thread. Your Wiki page
> calls the rules JMF. And it also seems to be called Hostkarma. It
> will be confusing to people if they see different names referring to
> the same thing. Perhaps we should call it JMF to avoid confusion?

I'd like to keep the name HOSTKARMA as standard.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 09/28/2009 06:53 PM, Marc Perkel wrote:
>
>
> Warren Togami wrote:
>> On 09/28/2009 01:32 PM, Marc Perkel wrote:
>>>
>>> I'd be interested in how well it worked. Is there anything I need to do
>>> to help?
>>
>> http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists
>> Could you provide a URL redirector to this page? This URL is very
>> long. Perhaps shorter URL in the describe of each rule like:
>> http://hostkarma.junkemailfilter.com ?
> I'm working on that. Trying to figure out how to give it an A record.
>>
>> This URL will be in spam reports so folks can click-thru and see why
>> their message triggered on this rule.
>>
>> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212
>> I filed the request to add it to sandbox for testing here. The line
>> wrapping got screwed up in Bugzilla.
>>
>> You might also want to consider standardizing the name of the
>> blacklist. You called it JEF earlier in this thread. Your Wiki page
>> calls the rules JMF. And it also seems to be called Hostkarma. It will
>> be confusing to people if they see different names referring to the
>> same thing. Perhaps we should call it JMF to avoid confusion?
>
> I'd like to keep the name HOSTKARMA as standard.

If that's so, then we probably want that in the spamassassin rule name.
Your wiki page suggests JMF is the name. A number of people probably
already configured their spamassassin using your suggested JMF rule
names and they would need to be educated to remove it.

How about these for rule names, so the rule names are not too long?

RCVD_HOSTKARMA_BL Black
RCVD_HOSTKARMA_WL White
RCVD_HOSTKARMA_YL Yellow
RCVD_HOSTKARMA_BR Brown

Warren Togami
wtogami@redhat.com
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Warren Togami wrote:
> On 09/28/2009 06:53 PM, Marc Perkel wrote:
>>
>>
>> Warren Togami wrote:
>>> On 09/28/2009 01:32 PM, Marc Perkel wrote:
>>>>
>>>> I'd be interested in how well it worked. Is there anything I need
>>>> to do
>>>> to help?
>>>
>>> http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists
>>> Could you provide a URL redirector to this page? This URL is very
>>> long. Perhaps shorter URL in the describe of each rule like:
>>> http://hostkarma.junkemailfilter.com ?
>> I'm working on that. Trying to figure out how to give it an A record.
>>>
>>> This URL will be in spam reports so folks can click-thru and see why
>>> their message triggered on this rule.
>>>
>>> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212
>>> I filed the request to add it to sandbox for testing here. The line
>>> wrapping got screwed up in Bugzilla.
>>>
>>> You might also want to consider standardizing the name of the
>>> blacklist. You called it JEF earlier in this thread. Your Wiki page
>>> calls the rules JMF. And it also seems to be called Hostkarma. It will
>>> be confusing to people if they see different names referring to the
>>> same thing. Perhaps we should call it JMF to avoid confusion?
>>
>> I'd like to keep the name HOSTKARMA as standard.
>
> If that's so, then we probably want that in the spamassassin rule
> name. Your wiki page suggests JMF is the name. A number of people
> probably already configured their spamassassin using your suggested
> JMF rule names and they would need to be educated to remove it.
>
> How about these for rule names, so the rule names are not too long?
>
> RCVD_HOSTKARMA_BL Black
> RCVD_HOSTKARMA_WL White
> RCVD_HOSTKARMA_YL Yellow
> RCVD_HOSTKARMA_BR Brown
>
> Warren Togami
> wtogami@redhat.com
>

Hi Warren,

No one has actually implemented the rules for my blacklists correctly.
My lists support both IP and hostname lookups. The hostname assumes that
you have forward confirmed the RDNS so that you eliminate those who
might spoof.

Yellow means that the IP or hostname contains no useful information as
to spam or no spam. On my system once I determine a host is yellow I
skip all blacklists and whitelists tests. Yellow is for Yahoo, Hotmail,
Gmail, etc where the IP has no information and all host tests are
meaningless.

My NoBL list is similar to yellow except that you can skip black list
lookup but maybe might be whitelisted somewhere.

If you just want to score points then Black, White, and Brown can be
assigned points. Yellow should be zero points regardless of how it tests.

I think the real power of my lists is in the host name lookups. It would
be worthwhile to implement that.

I think my white listing is very accurate at this point. The thing about
white servers is that they aren't evasive like spammers. There should be
some short circuiting options to reduce system load on SA for white
lookups.

And - I'm hoping others will catch on to some of the things I'm doing
because when other people adopt my tricks they usually improve them.

Let me know what I need to do to help make this happen.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On 09/28/2009 10:07 PM, Marc Perkel wrote:
>>> I'd like to keep the name HOSTKARMA as standard.
>>
>> If that's so, then we probably want that in the spamassassin rule
>> name. Your wiki page suggests JMF is the name. A number of people
>> probably already configured their spamassassin using your suggested
>> JMF rule names and they would need to be educated to remove it.
>>
>> How about these for rule names, so the rule names are not too long?
>>
>> RCVD_HOSTKARMA_BL Black
>> RCVD_HOSTKARMA_WL White
>> RCVD_HOSTKARMA_YL Yellow
>> RCVD_HOSTKARMA_BR Brown

Hi Marc,

I appreciate your desire for everyone to wholly benefit from your work,
but please let us implement this for spamassassin in stages starting
from the lowest hanging fruit.

First please confirm that you approve of the above new rule names, if
you don't want it to be known as JMF.

> Hi Warren,
>
> No one has actually implemented the rules for my blacklists correctly.
> My lists support both IP and hostname lookups. The hostname assumes that
> you have forward confirmed the RDNS so that you eliminate those who
> might spoof.

Please explain in greater detail? Can this be determined wholly from
the Headers and message body after the MTA had passed the mail to the MDA?

>
> Yellow means that the IP or hostname contains no useful information as
> to spam or no spam. On my system once I determine a host is yellow I
> skip all blacklists and whitelists tests. Yellow is for Yahoo, Hotmail,
> Gmail, etc where the IP has no information and all host tests are
> meaningless.
>
> My NoBL list is similar to yellow except that you can skip black list
> lookup but maybe might be whitelisted somewhere.

Please help me better understand, what are examples of a sequence of
events that would land an IP address on the NoBL?

>
> If you just want to score points then Black, White, and Brown can be
> assigned points. Yellow should be zero points regardless of how it tests.

I am aware that Yellow isn't useful for scores. It is however useful
for statistical analysis in masschecks, and it doesn't cost spamassassin
any more to print if it hits. In particular I'm looking to see if there
are any reliable trends of overlap between Yellow and other spamassassin
rules.

>
> I think the real power of my lists is in the host name lookups. It would
> be worthwhile to implement that.

Please describe how this is more effective than IP lookups?

>
> I think my white listing is very accurate at this point. The thing about
> white servers is that they aren't evasive like spammers. There should be
> some short circuiting options to reduce system load on SA for white
> lookups.

Generally spamassassin does not short-circuit by default for any reason.
There is an option to do so, but I think it is only to stop testing
rules if the score goes beyond a certain point. Please file a separate
bug for this if it is important to you.

Warren Togami
wtogami@redhat.com
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
From: "Marc Perkel" <marc@perkel.com>
Sent: Monday, 2009/September/28 19:07


>
>
> Warren Togami wrote:
>> On 09/28/2009 06:53 PM, Marc Perkel wrote:
...
>>> I'd like to keep the name HOSTKARMA as standard.
>>
>> If that's so, then we probably want that in the spamassassin rule name.
>> Your wiki page suggests JMF is the name. A number of people probably
>> already configured their spamassassin using your suggested JMF rule names
>> and they would need to be educated to remove it.
>>
>> How about these for rule names, so the rule names are not too long?
>>
>> RCVD_HOSTKARMA_BL Black
>> RCVD_HOSTKARMA_WL White
>> RCVD_HOSTKARMA_YL Yellow
>> RCVD_HOSTKARMA_BR Brown
>>
>> Warren Togami
>> wtogami@redhat.com
>>
>
> Hi Warren,
>
> No one has actually implemented the rules for my blacklists correctly. My
> lists support both IP and hostname lookups. The hostname assumes that you
> have forward confirmed the RDNS so that you eliminate those who might
> spoof.
>
> Yellow means that the IP or hostname contains no useful information as to
> spam or no spam. On my system once I determine a host is yellow I skip all
> blacklists and whitelists tests. Yellow is for Yahoo, Hotmail, Gmail, etc
> where the IP has no information and all host tests are meaningless.
>
> My NoBL list is similar to yellow except that you can skip black list
> lookup but maybe might be whitelisted somewhere.
>
> If you just want to score points then Black, White, and Brown can be
> assigned points. Yellow should be zero points regardless of how it tests.
>
> I think the real power of my lists is in the host name lookups. It would
> be worthwhile to implement that.
>
> I think my white listing is very accurate at this point. The thing about
> white servers is that they aren't evasive like spammers. There should be
> some short circuiting options to reduce system load on SA for white
> lookups.
>
> And - I'm hoping others will catch on to some of the things I'm doing
> because when other people adopt my tricks they usually improve them.
>
> Let me know what I need to do to help make this happen.

So what SHOULD this, which I clipped off your site, really look like
for SpamAssassin rules?
===8<---
header __RCVD_IN_JMF
eval:check_rbl('JMF-lastexternal','hostkarma.junkemailfilter.com.')
describe __RCVD_IN_JMF Sender listed in JunkEmailFilter
tflags __RCVD_IN_JMF net

header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
describe RCVD_IN_JMF_W Sender listed in JMF-WHITE
tflags RCVD_IN_JMF_W net nice
score RCVD_IN_JMF_W -5

header RCVD_IN_JMF_BL eval:check_rbl_sub('JMF-lastexternal', '127.0.0.2')
describe RCVD_IN_JMF_BL Sender listed in JMF-BLACK
tflags RCVD_IN_JMF_BL net
score RCVD_IN_JMF_BL 3.0

header RCVD_IN_JMF_BR eval:check_rbl_sub('JMF-lastexternal', '127.0.0.4')
describe RCVD_IN_JMF_BR Sender listed in JMF-BROWN
tflags RCVD_IN_JMF_BR net
score RCVD_IN_JMF_BR 1.0
===8<---

You pick the names and then the world can use them. The JMF names are out
there today.

{^_^} Joanne
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Hi,

> header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
> describe RCVD_IN_JMF_W Sender listed in JMF-WHITE
> tflags RCVD_IN_JMF_W net nice
> score RCVD_IN_JMF_W -5

Hopefully my comment isn't out of place with the current discussion of
JMF/Hostkarma. I think this is not only a really bad default score,
but it should be reduced to -0.5 or perhaps not used at all.

I have a money/fraud email that hit RCVD_IN_JMF_W that passed through
these servers:

Received: from 41.220.75.3
Received: from webmail.stu.qmul.ac.uk (138.37.100.37) by mercury.stu.qmul.ac.uk
Received: from qmwmail2.stu.qmul.ac.uk ([138.37.100.210]
Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6])

It also hit these other rules:

X-Spam-Status: No, hits=1.3 tagged_above=-300.0 required=5.0 use_bayes=1
tests=AE_GBP, BAYES_50, LOTS_OF_MONEY, LOTTERY_PH_004470,
LOTTO_RELATED, MONEY_TO_NO_R, RCVD_IN_DNSWL_MED, RCVD_IN_JMF_W,
RELAYCOUNTRY_UK, SPF_FAIL, SPF_HELO_FAIL

Unless I'm really missing something, which server has JMF/Hostkarma
whitelisted that shouldn't be?

This happens time after time.

Thanks,
Alex













>
> header RCVD_IN_JMF_BL eval:check_rbl_sub('JMF-lastexternal', '127.0.0.2')
> describe RCVD_IN_JMF_BL Sender listed in JMF-BLACK
> tflags RCVD_IN_JMF_BL net
> score RCVD_IN_JMF_BL 3.0
>
> header RCVD_IN_JMF_BR eval:check_rbl_sub('JMF-lastexternal', '127.0.0.4')
> describe RCVD_IN_JMF_BR Sender listed in JMF-BROWN
> tflags RCVD_IN_JMF_BR net
> score RCVD_IN_JMF_BR 1.0
> ===8<---
>
> You pick the names and then the world can use them. The JMF names are out
> there today.
>
> {^_^}    Joanne
>
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Hi!

> No one has actually implemented the rules for my blacklists correctly. My
> lists support both IP and hostname lookups. The hostname assumes that you
> have forward confirmed the RDNS so that you eliminate those who might spoof.

Most people copy/paste from your wiki, so if this is true ... i am not
sure where the real problem lies ;)

> Yellow means that the IP or hostname contains no useful information as to
> spam or no spam. On my system once I determine a host is yellow I skip all
> blacklists and whitelists tests. Yellow is for Yahoo, Hotmail, Gmail, etc
> where the IP has no information and all host tests are meaningless.
>
> My NoBL list is similar to yellow except that you can skip black list lookup
> but maybe might be whitelisted somewhere.

Please dont combine black and whitelists together in one BL. This will
trouble you. Many tools cannot look at the retern values. I think its a
bad idea. You can say hey not my problem but it will give a BL a bad karma
;)

> If you just want to score points then Black, White, and Brown can be assigned
> points. Yellow should be zero points regardless of how it tests.

Why would it be added to SA if the score is zero?

> I think the real power of my lists is in the host name lookups. It would be
> worthwhile to implement that.
>
> I think my white listing is very accurate at this point. The thing about
> white servers is that they aren't evasive like spammers. There should be some
> short circuiting options to reduce system load on SA for white lookups.

Ouch, from your point of view it might be fine, but we see strange stuff
with DNSWL allready i certainly would not use this to shortcircuit things.

A question from the operational side, how many people are working on the
BL? Just you i assume? Not telling this is bad, but its a risk when adding
this into SA i feel personally. Same for the infra the BL is running on.

I might sounds harsh, but i am rather carefull, then again, we have SA
update. So it might not hurt that much. But during outages or DDoS it will
hurt for hours till its gone again.

Bye,
Raymond.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
Hi!

> If that's so, then we probably want that in the spamassassin rule name. Your
> wiki page suggests JMF is the name. A number of people probably already
> configured their spamassassin using your suggested JMF rule names and they
> would need to be educated to remove it.
>
> How about these for rule names, so the rule names are not too long?
>
> RCVD_HOSTKARMA_BL Black
> RCVD_HOSTKARMA_WL White
> RCVD_HOSTKARMA_YL Yellow
> RCVD_HOSTKARMA_BR Brown

I would use the names that are advertised for months on the WIKI now, so
you can override them and not duplicate lookups on installs that have it
in their local.cf (or any place else).

Why did you invent (Marc) completely new names out of the blue?
The JMF_ stuff is there for months, please stick to it. We didnt invent
those, you did....

Bye,
Raymond.
Re: Hostkarma Blacklist Climbing the Charts [ In reply to ]
On Tue, Sep 29, 2009 at 09:29:16AM +0200, Raymond Dijkxhoorn wrote:
>
> Ouch, from your point of view it might be fine, but we see strange stuff
> with DNSWL allready i certainly would not use this to shortcircuit
> things.

What exactly is the strange stuff you see with DNSWL?

Granted, I'm not processing millions of messages, only tens of thousands,
but I'm not seeing anything fuzzy. I basically shortcircuit on DNSWL_MED and
DNSWL_HI, when there aren't any suspicious rules hit (ClamAV/Sanesecurity,
relay from africa, bayes over 60 etc). The FP rate is abysmally low.

1 2  View All