Mailing List Archive

New Release Candidate 4.0.0-rc4 Testers Needed
Hello Everyone,

I know a number of you have been looking at the release candidates for
the 4.0.0 release and have been helpful in finding issues with them.

We have just announced a new release candidate 4 that looks very close
to ready for the full 4.0.0 release.

We could use as many people as possible who are in a position to try it out.

Here is a link to the archived announcement we made on the developers
mailing list, which has all the details on downloading, what's new, and
upgrading from version 3.4.x.

https://lists.apache.org/thread/j10xp2b9166ctqsydhjqo5y9h8dw7zdp

Thanks,

Sidney Markowitz
Chair, Apache SpamAssassin PMC
sidney@apache.org
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
Sidney Markowitz <sidney@apache.org> writes:

> I know a number of you have been looking at the release candidates for
> the 4.0.0 release and have been helpful in finding issues with them.
>
> We have just announced a new release candidate 4 that looks very close
> to ready for the full 4.0.0 release.
>
> We could use as many people as possible who are in a position to try it out.
>
> Here is a link to the archived announcement we made on the developers
> mailing list, which has all the details on downloading, what's new,
> and upgrading from version 3.4.x.
>
> https://lists.apache.org/thread/j10xp2b9166ctqsydhjqo5y9h8dw7zdp

I am testing on NetBSD 9 amd64, likely a platform you don't have a
report for yet.

I have locally updated the mail/spamassasin entry in pkgsrc, which only
required minor defuzzing of our patches. (They are almost all about
accomodating to our non-default prefix and handling of config files.
I'll try to sort through and make sure anything that isn't that gets
sent upstream for discussion.)

I've run it on a machine which doesn't really do mail, starting spamd,
and running spamassassin -t on a message. That looks good; it gets
sensible scores include ARC_SIGNED and ARC_VALID.

I don't have a staging server, and could try it on my production server.
I therefore wonder:

Do people think it's prudent to try this in production (for a small
server for just personal mail, by someone who knows what they are
doing)?

If I run it, and I decide to flip back (because there are issues), is
there anything to worry about? I am using TxRep but I didn't see in
the announcement that there was a database schema change.

Have other put this in production and was it ok? I am guessing yes
and yes.
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
I have it in production.

On Sun, Dec 11, 2022, 09:00 Greg Troxel <gdt@lexort.com> wrote:

>
> Sidney Markowitz <sidney@apache.org> writes:
>
> > I know a number of you have been looking at the release candidates for
> > the 4.0.0 release and have been helpful in finding issues with them.
> >
> > We have just announced a new release candidate 4 that looks very close
> > to ready for the full 4.0.0 release.
> >
> > We could use as many people as possible who are in a position to try it
> out.
> >
> > Here is a link to the archived announcement we made on the developers
> > mailing list, which has all the details on downloading, what's new,
> > and upgrading from version 3.4.x.
> >
> > https://lists.apache.org/thread/j10xp2b9166ctqsydhjqo5y9h8dw7zdp
>
> I am testing on NetBSD 9 amd64, likely a platform you don't have a
> report for yet.
>
> I have locally updated the mail/spamassasin entry in pkgsrc, which only
> required minor defuzzing of our patches. (They are almost all about
> accomodating to our non-default prefix and handling of config files.
> I'll try to sort through and make sure anything that isn't that gets
> sent upstream for discussion.)
>
> I've run it on a machine which doesn't really do mail, starting spamd,
> and running spamassassin -t on a message. That looks good; it gets
> sensible scores include ARC_SIGNED and ARC_VALID.
>
> I don't have a staging server, and could try it on my production server.
> I therefore wonder:
>
> Do people think it's prudent to try this in production (for a small
> server for just personal mail, by someone who knows what they are
> doing)?
>
> If I run it, and I decide to flip back (because there are issues), is
> there anything to worry about? I am using TxRep but I didn't see in
> the announcement that there was a database schema change.
>
> Have other put this in production and was it ok? I am guessing yes
> and yes.
>
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
"Kevin A. McGrail" <kmcgrail@apache.org> writes:

> I have it in production.

Thanks - I just reinstalled, re-ran sa-update for base and KAM rules,
and so far it's looking good modulo a few nits:

UPGRADE says:

- All rules, functions, command line options and modules that contain
"whitelist" or "blacklist" have been renamed to contain more
racially neutral "welcomelist" and "blocklist" terms. This allows
acronyms like WL and BL to remain the same. Previous options will
continue work at least until version 4.1.0 is released. If you have
local settings including scores or meta rules referring to old rule
names, these should be changed and "enable_compat
welcomelist_blocklist" added in init.pre. See:
https://wiki.apache.org/spamassassin/WelcomelistBlocklist (Bug 7826)

1) I find that URL gets me "page not found.

2) It feels like a bug not to have compat by default. My memory is that
while 3.4.6 has compat-for-new, the output was such as to suggest that
the WELCOME syntax was non-standard, so I hadn't done query-replace yet.
For a From: that is "whitelist_from" (yes I know I know I need to tell
them to set up DKIM, and use whitelist_from_rcvd in the meantime), I
get:

-0.0 USER_IN_WELCOMELIST User is listed in 'welcomelist_from'
-100 USER_IN_WHITELIST DEPRECATED: See USER_IN_WELCOMELIST

Changing all entries to welcomelist_from gets me the same:

-0.0 USER_IN_WELCOMELIST User is listed in 'welcomelist_from'
-100 USER_IN_WHITELIST DEPRECATED: See USER_IN_WELCOMELIST

(I have run sa-update less than an hour ago; the mod time on my rules is
0959 EST.)

3) Maybe I'm misreading that there is no change in default and if you
want to use the new terms, you should turn on enable_compat
welcomelist_blocklist, but I never would have guessed that you need to
opt in to the new standard approach. I would expect that the new way
would work with no noise and the old way, for now, would work with
something that feels like a warning. And that some future release would
maybe fail to start if there are such warnings.


So it seems like 1) there is compat by default, which is good and 2)
somehow it is sort-of-old-compat, instead of omitting the whitelist line
since I hadn't configured it. Maybe this is something on my end, but I
did merge to the new local.cf. And as always maybe I am confused.



The good news is that I can find nothing else wrong. I am not seeing
short-circuiting happening but that is no different than 3.4.6 so I
think that's my fault, not a 4.0.0 issues.
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
On 2022-12-11 at 09:00:00 UTC-0500 (Sun, 11 Dec 2022 09:00:00 -0500)
Greg Troxel <gdt@lexort.com>
is rumored to have said:

> Sidney Markowitz <sidney@apache.org> writes:
>
>> I know a number of you have been looking at the release candidates for
>> the 4.0.0 release and have been helpful in finding issues with them.
>>
>> We have just announced a new release candidate 4 that looks very close
>> to ready for the full 4.0.0 release.
>>
>> We could use as many people as possible who are in a position to try it out.
>>
>> Here is a link to the archived announcement we made on the developers
>> mailing list, which has all the details on downloading, what's new,
>> and upgrading from version 3.4.x.
>>
>> https://lists.apache.org/thread/j10xp2b9166ctqsydhjqo5y9h8dw7zdp
>
> I am testing on NetBSD 9 amd64, likely a platform you don't have a
> report for yet.
>
> I have locally updated the mail/spamassasin entry in pkgsrc, which only
> required minor defuzzing of our patches. (They are almost all about
> accomodating to our non-default prefix and handling of config files.
> I'll try to sort through and make sure anything that isn't that gets
> sent upstream for discussion.)
>
> I've run it on a machine which doesn't really do mail, starting spamd,
> and running spamassassin -t on a message. That looks good; it gets
> sensible scores include ARC_SIGNED and ARC_VALID.
>
> I don't have a staging server, and could try it on my production server.
> I therefore wonder:
>
> Do people think it's prudent to try this in production (for a small
> server for just personal mail, by someone who knows what they are
> doing)?

I've been running the SVN trunk (currently identical to RC4) for a few months without significant problems on my personal server.

> If I run it, and I decide to flip back (because there are issues), is
> there anything to worry about? I am using TxRep but I didn't see in
> the announcement that there was a database schema change.

I don't see it stated explicitly, but I would not expect Bayes & TxRep databases to be revertible to an older version due to hash changes. This likely would not be fatal.

> Have other put this in production and was it ok? I am guessing yes
> and yes.

I believe that many PMC members have, like myself, been running trunk or each RC as it was released since we went into pre-release mode. We have bug reports specifically reported against 4.0 issues, all of which are resolved, so we do have external testers as well whose issues have all been dealt with.



--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
Greg Troxel wrote on 12/12/22 4:47 am:
> UPGRADE says:
> https://wiki.apache.org/spamassassin/WelcomelistBlocklist (Bug 7826)
>
> 1) I find that URL gets me "page not found.

We missed creating that wiki page when bug 7826 was handled. I have just
created a minimal page so the URL no longer 404s. If anyone has anything
more to say on the topic, feel free to edit the wiki page, but I think
what is there will do for now.
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
I am finding that short-circuiting seems not to be working, but this is
not new and I am not 100% clueful about it. However in trying to figure
things out I am running into things I do not understand and think that
at least a bit more doc clarity would help.

I have a fairly normal installation, milter with postfix, base rules,
KAM, some custom rules, and (now with 4.0.0rcN, renamed) a bunch of
welcomelist, which I try to do with dkim, then rcvd and for some give up
and just welcomelist. Plus a bunch of blocklist. These rules work and
I get sensible scores with occasional minor issues.

I just got a mail which was a little spammy and reasonably got 1.4, but
I decided to call it ham and added a "welcomelist_from_dkim
news@example.com mailchimpapp.net". It then scored -98.6, so that's
good.

I had in local.cf;

shortcircuit USER_IN_WELCOMELIST on

and realized that doesn't cover USER_IN_DKIM_WELCOMELIST, so I added

shortcircuit USER_IN_DKIM_WELCOMELIST on

but still scoring looks like:

-0.0 USER_IN_DKIM_WELCOMELIST From: address is in the user's DKIM
-100 USER_IN_DKIM_WHITELIST DEPRECATED: See USER_IN_DKIM_WELCOMELIST

plus a bunch of other stuff including network tests (all done
correctly). It took 36 seconds and there was no sign of short
circuiting.

The wiki page in the release notes says:

In SpamAssassin version 4.0.0 all rules, functions, command line
options and modules that contain "whitelist" or "blacklist" have
been renamed to contain "welcomelist" and "blocklist" terms. This
allows acronyms like WL and BL to remain the same. Previous options
will continue work at least until version 4.1.0 is released. If you
have local settings including scores or meta rules referring to old
rule names, these should be changed and "enable_compat
welcomelist_blocklist" added in init.pre.

I haven't enabled compat, but I did rename. I would expect that with
the transition to new keywords in 4.0.0, the normal approach is to edit
one's config and be all set. Or, one could leave the old words and have
them treated as compatible, maybe with a warning. Or possibly have to
enable compatibility for the old ones.

Am I really supposed to change the keywords to welcome/block *and* set
"enable_compat"? The man page Mail::SpamAssassin::Conf.3 doesn't say
that, that I was able to find.

I wonder if I am not getting short circuit because the -100 is awarded
to USER_IN_DKIM_WHITELIST, not USER_IN_DKIM_WELCOMELIST? Or is DKIM a
network test, and thus doesn't really work for short circuiting?
Something else?

(I realize that 36s is a clue that an RBL is being queried that times
out and I should find and fix that, but I think that's orthogonal to my
questions.)
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
We have had issues with shortcircuit that cropped up.in the rc process from
optimization that was performed. Can you open up bugzilla ticket please?

On Tue, Dec 13, 2022, 08:37 Greg Troxel <gdt@lexort.com> wrote:

>
> I am finding that short-circuiting seems not to be working, but this is
> not new and I am not 100% clueful about it. However in trying to figure
> things out I am running into things I do not understand and think that
> at least a bit more doc clarity would help.
>
> I have a fairly normal installation, milter with postfix, base rules,
> KAM, some custom rules, and (now with 4.0.0rcN, renamed) a bunch of
> welcomelist, which I try to do with dkim, then rcvd and for some give up
> and just welcomelist. Plus a bunch of blocklist. These rules work and
> I get sensible scores with occasional minor issues.
>
> I just got a mail which was a little spammy and reasonably got 1.4, but
> I decided to call it ham and added a "welcomelist_from_dkim
> news@example.com mailchimpapp.net". It then scored -98.6, so that's
> good.
>
> I had in local.cf;
>
> shortcircuit USER_IN_WELCOMELIST on
>
> and realized that doesn't cover USER_IN_DKIM_WELCOMELIST, so I added
>
> shortcircuit USER_IN_DKIM_WELCOMELIST on
>
> but still scoring looks like:
>
> -0.0 USER_IN_DKIM_WELCOMELIST From: address is in the user's DKIM
> -100 USER_IN_DKIM_WHITELIST DEPRECATED: See USER_IN_DKIM_WELCOMELIST
>
> plus a bunch of other stuff including network tests (all done
> correctly). It took 36 seconds and there was no sign of short
> circuiting.
>
> The wiki page in the release notes says:
>
> In SpamAssassin version 4.0.0 all rules, functions, command line
> options and modules that contain "whitelist" or "blacklist" have
> been renamed to contain "welcomelist" and "blocklist" terms. This
> allows acronyms like WL and BL to remain the same. Previous options
> will continue work at least until version 4.1.0 is released. If you
> have local settings including scores or meta rules referring to old
> rule names, these should be changed and "enable_compat
> welcomelist_blocklist" added in init.pre.
>
> I haven't enabled compat, but I did rename. I would expect that with
> the transition to new keywords in 4.0.0, the normal approach is to edit
> one's config and be all set. Or, one could leave the old words and have
> them treated as compatible, maybe with a warning. Or possibly have to
> enable compatibility for the old ones.
>
> Am I really supposed to change the keywords to welcome/block *and* set
> "enable_compat"? The man page Mail::SpamAssassin::Conf.3 doesn't say
> that, that I was able to find.
>
> I wonder if I am not getting short circuit because the -100 is awarded
> to USER_IN_DKIM_WHITELIST, not USER_IN_DKIM_WELCOMELIST? Or is DKIM a
> network test, and thus doesn't really work for short circuiting?
> Something else?
>
> (I realize that 36s is a clue that an RBL is being queried that times
> out and I should find and fix that, but I think that's orthogonal to my
> questions.)
>
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
Greg Troxel <gdt@lexort.com> writes:

> The wiki page in the release notes says:
>
> In SpamAssassin version 4.0.0 all rules, functions, command line
> options and modules that contain "whitelist" or "blacklist" have
> been renamed to contain "welcomelist" and "blocklist" terms. This
> allows acronyms like WL and BL to remain the same. Previous options
> will continue work at least until version 4.1.0 is released. If you
> have local settings including scores or meta rules referring to old
> rule names, these should be changed and "enable_compat
> welcomelist_blocklist" added in init.pre.
>
> I haven't enabled compat, but I did rename. I would expect that with
> the transition to new keywords in 4.0.0, the normal approach is to edit
> one's config and be all set. Or, one could leave the old words and have
> them treated as compatible, maybe with a warning. Or possibly have to
> enable compatibility for the old ones.
>
> Am I really supposed to change the keywords to welcome/block *and* set
> "enable_compat"? The man page Mail::SpamAssassin::Conf.3 doesn't say
> that, that I was able to find.

There's nothing like reading the code to answer questions. So

- The basic rule is USER_IN_WELCOMELIST, and the basic score for it is
-100. This is in 60_welcomelist.cf.

- If "enable_compat welcomelist_blocklist" has been given, it stays
that way.

- Without that compat statement, a meta rule USER_IN_WHITELIST is
created and given a score of -100, and the score for
USER_IN_WELCOMELIST is set to -0.01.

I don't understand this approach at all:

- I think it's important that if a user has "whitelist_from" in their
config, that's still followed. As far as I can tell that's in the
config parser not the scoring, so that is separate.

- The words are changed in this release, so if the user doesn't ask
for anything special, their scoring output should have -100 for the
WELCOMELIST, and shouldn't show WHITELIST.

- If someone has meta rules that include USER_IN_WHITELIST, then there
needs to be a compat rule with that name for that to work. But that
seems like a very unusual thing to do, as usually
WELCOMELIST/WHITELIST rules have a score such that no further rules
are needed.

- Someone might reasonably want to turn on shortcircuiting for
USER_IN_WELCOMELIST, and it seems awkward at best for that to
fire on a -0.01 score and expect the -100 meta rule to kick in.
This is the default behavior.


With considerable trepidation from not really understanding, I would
instead

- have a

enable_compat whitelist_blacklist

that adds the meta rules (USER_IN_WHITELIST) with scores -0.01, and
leaves the scores for USER_IN_WELCOMELIST alone, to accomodate
people with meta rules that refer to this.

- adjust the wiki documentation to say that both welcomlist_from
(standard approach) and whitelist_from (deprecated, compatibility)
are recognized in config files, and explain about the compat for
meta rules in the first bullet point.
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
"Kevin A. McGrail" <kmcgrail@apache.org> writes:

>> I am finding that short-circuiting seems not to be working, but this is
>> not new and I am not 100% clueful about it. However in trying to figure
>> things out I am running into things I do not understand and think that
>> at least a bit more doc clarity would help.

> We have had issues with shortcircuit that cropped up.in the rc process from
> optimization that was performed. Can you open up bugzilla ticket
> please?

False alarm. The issue was that I wasn't loading the shortcircuit
plugin, since it never occured to me from reading local.cf that it was
not default. Regular welcomelist short circuits very well, and DKIM
welcomelist is also working well, but dns queries are launched (I get
why of course).

There is also the possibility of confusion between shortcircuit config for
USER_IN_WELCOMELIST when the enable_compat is not on, but I have no
evidence for that. I'll try to check that later.


So, my issues with the RC are now down to understanding the welcomelist
compat strategy (which is good, because that's not really a big deal and
easy to address).


(My "taking 30s" issue was that the NFS locking method fails on NetBSD,
and I switched to flock. My "core dump" issue is a bug someplace in the
BDB hash table support (or in the bdb code on my system) showing up in
TxRep use, which feels like off-by-1 errot but I'll figure it out, and I
have a really gross workaround.)
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
Excellent news! Please let us know more about the WL/BL changes and open a
bugzilla bug.
--
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171


On Wed, Dec 14, 2022 at 9:54 AM Greg Troxel <gdt@lexort.com> wrote:

>
> "Kevin A. McGrail" <kmcgrail@apache.org> writes:
>
> >> I am finding that short-circuiting seems not to be working, but this is
> >> not new and I am not 100% clueful about it. However in trying to figure
> >> things out I am running into things I do not understand and think that
> >> at least a bit more doc clarity would help.
>
> > We have had issues with shortcircuit that cropped up.in the rc process
> from
> > optimization that was performed. Can you open up bugzilla ticket
> > please?
>
> False alarm. The issue was that I wasn't loading the shortcircuit
> plugin, since it never occured to me from reading local.cf that it was
> not default. Regular welcomelist short circuits very well, and DKIM
> welcomelist is also working well, but dns queries are launched (I get
> why of course).
>
> There is also the possibility of confusion between shortcircuit config for
> USER_IN_WELCOMELIST when the enable_compat is not on, but I have no
> evidence for that. I'll try to check that later.
>
>
> So, my issues with the RC are now down to understanding the welcomelist
> compat strategy (which is good, because that's not really a big deal and
> easy to address).
>
>
> (My "taking 30s" issue was that the NFS locking method fails on NetBSD,
> and I switched to flock. My "core dump" issue is a bug someplace in the
> BDB hash table support (or in the bdb code on my system) showing up in
> TxRep use, which feels like off-by-1 errot but I'll figure it out, and I
> have a really gross workaround.)
>
Re: New Release Candidate 4.0.0-rc4 Testers Needed [ In reply to ]
On 12/14/22 10:51?AM, Kevin A. McGrail wrote:
> Excellent news!  Please let us know more about the WL/BL changes and
> open a bugzilla bug.

My other post about this has the info, but I just wrote a bug entry that
is probably more succinct and coherent now that I understand better.

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8092