Mailing List Archive

i tried to set up bayes training before but i feel that i was
unsuccessful.  is there a definitive guide on setting this up on postfix
with amavis?  if my user were the one that was training it for the other
users that would be ideal

On 10/07/2020 10:11, Reindl Harald wrote:
>
> Am 10.07.20 um 09:58 schrieb Matthew Broadhead:
>> On Thunderbird if I train the spam filter for a short time I get really
>> great results with it catching most of what I would consider spam.
> the same with spamassassin when you train bayes
>
>> Also I wondered if there is a way to ask spamassassin to put the spam in
>> the user's spam folder rather than deleting it.
> spamassasin don't delete messages at all, it only flags them and what
> happens with spam flag in the headers is decided by the glue (amavis)
>
> if you not reject them and write sieve filters you can place messages
> somewhere, spamassassin has no concept of folders because it's a
> completly different layer

On 10.07.20 11:02, Matthew Broadhead wrote:
>i tried to set up bayes training before but i feel that i was
>unsuccessful.?

bayes need some time and some mails to learn until it starts hitting.
You can help it by training manually, using bayes database in amavis' home
directory. You also should do that, because automatic training is not ideal
ane may miss many spams.

>>Am 10.07.20 um 09:58 schrieb Matthew Broadhead:
>>>On Thunderbird if I train the spam filter for a short time I get really
>>>great results with it catching most of what I would consider spam.
>>the same with spamassassin when you train bayes
>>
>>>Also I wondered if there is a way to ask spamassassin to put the spam in
>>>the user's spam folder rather than deleting it.
>>spamassasin don't delete messages at all, it only flags them and what
>>happens with spam flag in the headers is decided by the glue (amavis)
>>
>>if you not reject them and write sieve filters you can place messages
>>somewhere, spamassassin has no concept of folders because it's a
>>completly different layer

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I just got lost in thought. It was unfamiliar territory.

On Friday 10 July 2020 at 11:02:23, Matthew Broadhead wrote:

> i tried to set up bayes training before but i feel that i was unsuccessful.

It would be helpful to know what gives you that feeling.

How are you training and how are you testing?

> is there a definitive guide on setting this up on postfix with amavis? if my
> user were the one that was training it for the other users that would be
> ideal

In general, Bayes works best when each user trains for the ham and the spam
that *they* receive - you can share Bayes databases between users, but it's
less effective.

Do remember also that Bayes *needs* to be fed ham as well as spam for it to be
able to learn the difference - it's not just a question of giving it all the
spam you receive and assuming it can work out from that what isn't spam.


Regards,


Antony.

--
90% of networking problems are routing problems.
9 of the remaining 10% are routing problems in the other direction.
The remaining 1% might be something else, but check the routing anyway.

Please reply to the list;
please *don't* CC me.

You can add rspamd they have all inclusive soluation and they do not
even like if you want to customize the 3000 line counting configuration
file to do just one check.




-----Original Message-----
From: Matthew Broadhead [mailto:matthew.broadhead@nbmlaw.co.uk]
Sent: vrijdag 10 juli 2020 9:58
To: users
Subject: Really simple setup guide

Hi,

I am running postfix, dovecot, amavis, spamassassin and clamav on centos
7.

Everything is working great, and if I send an email with the spam
signature on it it gets blocked by spamassassin.

But still a lot of spam comes spamassassin and I wondered if there was a
really great guide out there which might tell me how to tune the spam
filter.

On Thunderbird if I train the spam filter for a short time I get really
great results with it catching most of what I would consider spam.

Also I wondered if there is a way to ask spamassassin to put the spam in
the user's spam folder rather than deleting it.

Thanks in advance,
Matthew

i set it up a long time ago and i have never had any success with it.  i
can post a copy of my configurations.

local.cf
required_hits 5.0
report_safe 0
required_score 5
rewrite_header Subject [SPAM]

use_bayes          1
bayes_auto_learn   1
bayes_auto_expire  1

# Store bayesian data in MySQL
bayes_store_module Mail::SpamAssassin::BayesStore::MySQL
bayes_sql_dsn       DBI:mysql:sa_bayes:localhost:3306

i remember setting up the sa-learn command somewhere and there were
definitely records appearing in the database

On 10/07/2020 11:46, Antony Stone wrote:
> On Friday 10 July 2020 at 11:02:23, Matthew Broadhead wrote:
>
>> i tried to set up bayes training before but i feel that i was unsuccessful.
> It would be helpful to know what gives you that feeling.
>
> How are you training and how are you testing?
>
>> is there a definitive guide on setting this up on postfix with amavis? if my
>> user were the one that was training it for the other users that would be
>> ideal
> In general, Bayes works best when each user trains for the ham and the spam
> that *they* receive - you can share Bayes databases between users, but it's
> less effective.
>
> Do remember also that Bayes *needs* to be fed ham as well as spam for it to be
> able to learn the difference - it's not just a question of giving it all the
> spam you receive and assuming it can work out from that what isn't spam.
>
>
> Regards,
>
>
> Antony.
>

i have 788787 messages in my bayes_seen directory.? 33494 are flagged
's'.? i am not sure why it seems to have no effect on the spam coming
through

On 10/07/2020 11:40, Matus UHLAR - fantomas wrote:
> On 10.07.20 11:02, Matthew Broadhead wrote:
>> i tried to set up bayes training before but i feel that i was
>> unsuccessful.
>
> bayes need some time and some mails to learn until it starts hitting.
> You can help it by training manually, using bayes database in amavis'
> home
> directory. You also should do that, because automatic training is not
> ideal
> ane may miss many spams.
>
>>> Am 10.07.20 um 09:58 schrieb Matthew Broadhead:
>>>> On Thunderbird if I train the spam filter for a short time I get
>>>> really
>>>> great results with it catching most of what I would consider spam.
>>> the same with spamassassin when you train bayes
>>>
>>>> Also I wondered if there is a way to ask spamassassin to put the
>>>> spam in
>>>> the user's spam folder rather than deleting it.
>>> spamassasin don't delete messages at all, it only flags them and what
>>> happens with spam flag in the headers is decided by the glue (amavis)
>>>
>>> if you not reject them and write sieve filters you can place messages
>>> somewhere, spamassassin has no concept of folders because it's a
>>> completly different layer
>

On 10.07.20 13:00, Matthew Broadhead wrote:
>i have 788787 messages in my

your?

> bayes_seen directory

what is the bayes_seen directory? I have bayes_seen file, which is part of
the bayes database.

>.? 33494 are flagged
>'s'.? i am not sure why it seems to have no effect on the spam coming
>through


>On 10/07/2020 11:40, Matus UHLAR - fantomas wrote:
>>On 10.07.20 11:02, Matthew Broadhead wrote:
>>>i tried to set up bayes training before but i feel that i was
>>>unsuccessful.
>>
>>bayes need some time and some mails to learn until it starts hitting.
>>You can help it by training manually, using bayes database in
>>amavis' home
>>directory. You also should do that, because automatic training is
>>not ideal
>>ane may miss many spams.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I don't have lysdexia. The Dog wouldn't allow that.

i mean bayes_seen table in my MySQL database

On 10/07/2020 13:12, Matus UHLAR - fantomas wrote:
> On 10.07.20 13:00, Matthew Broadhead wrote:
>> i have 788787 messages in my
>
> your?
>
>> bayes_seen directory
>
> what is the bayes_seen directory? I have bayes_seen file, which is
> part of
> the bayes database.
>
>> .? 33494 are flagged 's'.? i am not sure why it seems to have no
>> effect on the spam coming through
>
>
>> On 10/07/2020 11:40, Matus UHLAR - fantomas wrote:
>>> On 10.07.20 11:02, Matthew Broadhead wrote:
>>>> i tried to set up bayes training before but i feel that i was
>>>> unsuccessful.
>>>
>>> bayes need some time and some mails to learn until it starts hitting.
>>> You can help it by training manually, using bayes database in
>>> amavis' home
>>> directory. You also should do that, because automatic training is
>>> not ideal
>>> ane may miss many spams.

On 7/10/20 11:02 AM, Matthew Broadhead wrote:
> i tried to set up bayes training before but i feel that i was
> unsuccessful.  is there a definitive guide on setting this up on postfix
> with amavis?  if my user were the one that was training it for the other
> users that would be ideal

unless you tell us exactly what you're doing to train bayes, it's
anybody's guess and possibly just confusing.
iirc, the Amavis documentation should include the info you're looking for.

> But still a lot of spam comes spamassassin and I wondered if there was a
> really great guide out there which might tell me how to tune the spam
> filter.

Hi Matthew,

I would recommend you add KAM.cf Ruleset and see if that helps.

https://mcgrail.com/downloads/KAM.cf

https://mcgrail.com/downloads/KAM_deadweight.cf   

https://mcgrail.com/downloads/KAM_heavyweight.cf

https://mcgrail.com/downloads/nonKAMrules.cf

Regards,
KAM

--
Kevin A. McGrail
KMcGrail@Apache.org

Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

If you don't know much about Bayes and aren't training HAM and SPAM,
it's likely got improper bias.? I'd recommend adding KAM.cf (see my
other email) and disable Bayesian and see how it looks.

On 7/10/2020 7:00 AM, Matthew Broadhead wrote:
> i have 788787 messages in my bayes_seen directory.? 33494 are flagged
> 's'.? i am not sure why it seems to have no effect on the spam coming
> through

--
Kevin A. McGrail
KMcGrail@Apache.org

Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

On 10.07.20 13:16, Matthew Broadhead wrote:
>i mean bayes_seen table in my MySQL database

where did you configure bayes to use mysql?
are you sure amavis uses it?

and... did you train manually or do you completely rely on automatic
training?

>On 10/07/2020 13:12, Matus UHLAR - fantomas wrote:
>>On 10.07.20 13:00, Matthew Broadhead wrote:
>>>i have 788787 messages in my
>>
>>your?
>>
>>>bayes_seen directory
>>
>>what is the bayes_seen directory? I have bayes_seen file, which is
>>part of
>>the bayes database.
>>
>>>.? 33494 are flagged 's'.? i am not sure why it seems to have no
>>>effect on the spam coming through
>>
>>
>>>On 10/07/2020 11:40, Matus UHLAR - fantomas wrote:
>>>>On 10.07.20 11:02, Matthew Broadhead wrote:
>>>>>i tried to set up bayes training before but i feel that i was
>>>>>unsuccessful.
>>>>
>>>>bayes need some time and some mails to learn until it starts hitting.
>>>>You can help it by training manually, using bayes database in
>>>>amavis' home
>>>>directory. You also should do that, because automatic training
>>>>is not ideal
>>>>ane may miss many spams.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Support bacteria - they're the only culture some people have.

it is manual training.? i have been looking through the configuration
but i cannot figure out how it is set up.? i think it works when email
is moved in and out of spam directory .? i will keep looking and report
back if i remember how it was set up.
in the meantime i sent a message to the amavis mailing list

On 10/07/2020 13:59, Matus UHLAR - fantomas wrote:
> On 10.07.20 13:16, Matthew Broadhead wrote:
>> i mean bayes_seen table in my MySQL database
>
> where did you configure bayes to use mysql?
> are you sure amavis uses it?
>
> and... did you train manually or do you completely rely on automatic
> training?
>
>> On 10/07/2020 13:12, Matus UHLAR - fantomas wrote:
>>> On 10.07.20 13:00, Matthew Broadhead wrote:
>>>> i have 788787 messages in my
>>>
>>> your?
>>>
>>>> bayes_seen directory
>>>
>>> what is the bayes_seen directory? I have bayes_seen file, which is
>>> part of
>>> the bayes database.
>>>
>>>> .? 33494 are flagged 's'.? i am not sure why it seems to have no
>>>> effect on the spam coming through
>>>
>>>
>>>> On 10/07/2020 11:40, Matus UHLAR - fantomas wrote:
>>>>> On 10.07.20 11:02, Matthew Broadhead wrote:
>>>>>> i tried to set up bayes training before but i feel that i was
>>>>>> unsuccessful.
>>>>>
>>>>> bayes need some time and some mails to learn until it starts hitting.
>>>>> You can help it by training manually, using bayes database in
>>>>> amavis' home
>>>>> directory. You also should do that, because automatic training is
>>>>> not ideal
>>>>> ane may miss many spams.
>

i am starting to remember that sa-learn is being called from dovecot

in dovecot/conf.d/90-plugin.conf i have
plugin {
? # man antispam
? #setting_name = value
? antispam_debug_target = syslog
? antispam_verbose_debug = 1

? antispam_backend = pipe
? antispam_signature = X-DSPAM-Signature
? # action to take on mails without signature
? # (used with any backend requiring a signature)
? # (we recommend only setting this to 'move' after verifying that the
? # whole setup is working)
? # antispam_signature_missing = move # move silently without training
? antispam_signature_missing = error

? antispam_trash = Trash
? antispam_spam = Junk
? antispam_pipe_program_spam_arg = --spam
? antispam_pipe_program_notspam_arg = --ham
? antispam_pipe_program = /usr/bin/sa-learn
? antispam_pipe_program_args = --username=%Lu
}


On 10/07/2020 13:59, Matus UHLAR - fantomas wrote:
> On 10.07.20 13:16, Matthew Broadhead wrote:
>> i mean bayes_seen table in my MySQL database
>
> where did you configure bayes to use mysql?
> are you sure amavis uses it?
>
> and... did you train manually or do you completely rely on automatic
> training?
>
>> On 10/07/2020 13:12, Matus UHLAR - fantomas wrote:
>>> On 10.07.20 13:00, Matthew Broadhead wrote:
>>>> i have 788787 messages in my
>>>
>>> your?
>>>
>>>> bayes_seen directory
>>>
>>> what is the bayes_seen directory? I have bayes_seen file, which is
>>> part of
>>> the bayes database.
>>>
>>>> .? 33494 are flagged 's'.? i am not sure why it seems to have no
>>>> effect on the spam coming through
>>>
>>>
>>>> On 10/07/2020 11:40, Matus UHLAR - fantomas wrote:
>>>>> On 10.07.20 11:02, Matthew Broadhead wrote:
>>>>>> i tried to set up bayes training before but i feel that i was
>>>>>> unsuccessful.
>>>>>
>>>>> bayes need some time and some mails to learn until it starts hitting.
>>>>> You can help it by training manually, using bayes database in
>>>>> amavis' home
>>>>> directory. You also should do that, because automatic training is
>>>>> not ideal
>>>>> ane may miss many spams.
>

should i be going to the dovecot team with this problem?

On 10/07/2020 14:26, Matthew Broadhead wrote:
> i am starting to remember that sa-learn is being called from dovecot
>
> in dovecot/conf.d/90-plugin.conf i have
> plugin {
> ? # man antispam
> ? #setting_name = value
> ? antispam_debug_target = syslog
> ? antispam_verbose_debug = 1
>
> ? antispam_backend = pipe
> ? antispam_signature = X-DSPAM-Signature
> ? # action to take on mails without signature
> ? # (used with any backend requiring a signature)
> ? # (we recommend only setting this to 'move' after verifying that the
> ? # whole setup is working)
> ? # antispam_signature_missing = move # move silently without training
> ? antispam_signature_missing = error
>
> ? antispam_trash = Trash
> ? antispam_spam = Junk
> ? antispam_pipe_program_spam_arg = --spam
> ? antispam_pipe_program_notspam_arg = --ham
> ? antispam_pipe_program = /usr/bin/sa-learn
> ? antispam_pipe_program_args = --username=%Lu
> }
>
>
> On 10/07/2020 13:59, Matus UHLAR - fantomas wrote:
>> On 10.07.20 13:16, Matthew Broadhead wrote:
>>> i mean bayes_seen table in my MySQL database
>>
>> where did you configure bayes to use mysql?
>> are you sure amavis uses it?
>>
>> and... did you train manually or do you completely rely on automatic
>> training?
>>
>>> On 10/07/2020 13:12, Matus UHLAR - fantomas wrote:
>>>> On 10.07.20 13:00, Matthew Broadhead wrote:
>>>>> i have 788787 messages in my
>>>>
>>>> your?
>>>>
>>>>> bayes_seen directory
>>>>
>>>> what is the bayes_seen directory? I have bayes_seen file, which is
>>>> part of
>>>> the bayes database.
>>>>
>>>>> .? 33494 are flagged 's'.? i am not sure why it seems to have no
>>>>> effect on the spam coming through
>>>>
>>>>
>>>>> On 10/07/2020 11:40, Matus UHLAR - fantomas wrote:
>>>>>> On 10.07.20 11:02, Matthew Broadhead wrote:
>>>>>>> i tried to set up bayes training before but i feel that i was
>>>>>>> unsuccessful.
>>>>>>
>>>>>> bayes need some time and some mails to learn until it starts
>>>>>> hitting.
>>>>>> You can help it by training manually, using bayes database in
>>>>>> amavis' home
>>>>>> directory. You also should do that, because automatic training is
>>>>>> not ideal
>>>>>> ane may miss many spams.
>>
>

On 10.07.20 17:35, Matthew Broadhead wrote:
>should i be going to the dovecot team with this problem?
>
>On 10/07/2020 14:26, Matthew Broadhead wrote:
>>i am starting to remember that sa-learn is being called from dovecot

yeah, very good idea.

>>in dovecot/conf.d/90-plugin.conf i have
>>plugin {
>>? # man antispam
>>? #setting_name = value
>>? antispam_debug_target = syslog
>>? antispam_verbose_debug = 1
>>
>>? antispam_backend = pipe
>>? antispam_signature = X-DSPAM-Signature
>>? # action to take on mails without signature
>>? # (used with any backend requiring a signature)
>>? # (we recommend only setting this to 'move' after verifying that the
>>? # whole setup is working)
>>? # antispam_signature_missing = move # move silently without training
>>? antispam_signature_missing = error
>>
>>? antispam_trash = Trash
>>? antispam_spam = Junk
>>? antispam_pipe_program_spam_arg = --spam
>>? antispam_pipe_program_notspam_arg = --ham
>>? antispam_pipe_program = /usr/bin/sa-learn
>>? antispam_pipe_program_args = --username=%Lu
>>}

amavisd-new runs under amavis user, so the username is either not used, or
breaks things up.

where did you set up SQL BAYES?
if it's in spamassassin's local.cf, passed username may not matter
(I haven't tried...)


>>On 10/07/2020 13:59, Matus UHLAR - fantomas wrote:
>>>On 10.07.20 13:16, Matthew Broadhead wrote:
>>>>i mean bayes_seen table in my MySQL database
>>>
>>>where did you configure bayes to use mysql?
>>>are you sure amavis uses it?
>>>
>>>and... did you train manually or do you completely rely on automatic
>>>training?

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
One OS to rule them all, One OS to find them,
One OS to bring them all and into darkness bind them

Marc Roos skrev den 2020-07-10 12:10:
> You can add rspamd they have all inclusive soluation and they do not
> even like if you want to customize the 3000 line counting configuration
> file to do just one check.

i find that its hard to configure rspamd, thats one of the reasons i
dropped trying with it, i was aswell unvelcommed as a first gentoo
ebuild maintainer

current rspamd in gentoo is masked unstable, dont know if i need to know
more

Matthew Broadhead skrev den 2020-07-10 11:02:
> i tried to set up bayes training before but i feel that i was
> unsuccessful.  is there a definitive guide on setting this up on
> postfix with amavis?  if my user were the one that was training it for
> the other users that would be ideal

are you sure you want to use amavis ?, have you considered fuglu ?

or even other not so hard to make it work, like spampd ?

Matthew Broadhead skrev den 2020-07-10 13:00:
> i have 788787 messages in my bayes_seen directory.  33494 are flagged
> 's'.  i am not sure why it seems to have no effect on the spam coming
> through

diffrent bayes user in user_prefs

incorrect amavisd bayes user mapping pr recipient

hardcoded global user that is not used anyway

is the above data from mysql, postgresql, sqlite, or even redis ?, or is
it from a berkdb path ?

On Fri, 10 Jul 2020 14:26:17 +0200
Matthew Broadhead wrote:

> i am starting to remember that sa-learn is being called from dovecot

Previously you indicated that you had 755293 ham trained (from
788787-33494). Unless you have an extreme number of FPs your training
is mostly from auto-learning.

> in dovecot/conf.d/90-plugin.conf i have
> plugin {
...
>   antispam_signature = X-DSPAM-Signature
...
> training antispam_signature_missing = error

This looks wrong, you are checking for a DSPAM signature, and then
training SpamAssassin. Possibly this is preventing the plug-in from
working, if it's treating the absence of the header as an error.

I'm not keen on using that plugin with SA as it trains only based on
full misclassifications which can (or should) lead to a very low rate
of ham training.

fuglu looks nice.  it even handles the vacation messages from database,
whereas i have been struggling with sieve.
can it train the bayes as well?

On 10/07/2020 23:00, Benny Pedersen wrote:
> Matthew Broadhead skrev den 2020-07-10 11:02:
>> i tried to set up bayes training before but i feel that i was
>> unsuccessful.  is there a definitive guide on setting this up on
>> postfix with amavis?  if my user were the one that was training it for
>> the other users that would be ideal
>
> are you sure you want to use amavis ?, have you considered fuglu ?
>
> or even other not so hard to make it work, like spampd ?

On 7/11/20 10:11 AM, Matthew Broadhead wrote:
> fuglu looks nice.  it even handles the vacation messages from database,
> whereas i have been struggling with sieve.
> can it train the bayes as well?

nope - bayes learning is handled by SA's sa-learn (see docs)

fuglu is just glue between your MTA and SA, your AVs, etc, etc.

>
> On 10/07/2020 23:00, Benny Pedersen wrote:
>> Matthew Broadhead skrev den 2020-07-10 11:02:
>>> i tried to set up bayes training before but i feel that i was
>>> unsuccessful.  is there a definitive guide on setting this up on
>>> postfix with amavis?  if my user were the one that was training it for
>>> the other users that would be ideal
>>
>> are you sure you want to use amavis ?, have you considered fuglu ?
>>
>> or even other not so hard to make it work, like spampd ?
>

because my box is centos 7 i think i am going to grab an old box and
stick centos 8 on there and start from scratch with a small domain. like
postfix, dovecot (full text search), fuglu (spamassassim, clamav, dkim,
spf, vacation) .  would be nice to build everything with docker so that
the configuration is easily reproducible and it is obvious what has
changed from base config.

On 11/07/2020 11:21, Axb wrote:
> On 7/11/20 10:11 AM, Matthew Broadhead wrote:
>> fuglu looks nice.  it even handles the vacation messages from
>> database, whereas i have been struggling with sieve.
>> can it train the bayes as well?
>
> nope - bayes learning is handled by SA's sa-learn (see docs)
>
> fuglu is just glue between your MTA and SA, your AVs, etc, etc.
>
>>
>> On 10/07/2020 23:00, Benny Pedersen wrote:
>>> Matthew Broadhead skrev den 2020-07-10 11:02:
>>>> i tried to set up bayes training before but i feel that i was
>>>> unsuccessful.  is there a definitive guide on setting this up on
>>>> postfix with amavis?  if my user were the one that was training it for
>>>> the other users that would be ideal
>>>
>>> are you sure you want to use amavis ?, have you considered fuglu ?
>>>
>>> or even other not so hard to make it work, like spampd ?
>>
>
>

Matthew Broadhead skrev den 2020-07-11 10:11:
> fuglu looks nice.  it even handles the vacation messages from
> database, whereas i have been struggling with sieve.
> can it train the Bayes as well?

i see no problem in sieve Bayes learning, its documented in wiki from
dovecot how to make this work

for more deeply fuglu help join the maillist