Mailing List Archive

Bayes files LOCK
Hi..
i am getting  errors from Byes because it is not able to lock Bayes files... 
Error log is: 
  bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_* R/W: lock failed: Interrupted system call


 SA tries to lock bayes files always in "EXclusive mode", hence when a sa takes too long, all other processes have to wait for the lock...

This is my config:
use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn 0bayes_auto_expire 0lock_method flock

SA uses EXclusive lock because it has "to write"...  my question is...    Is there any way to avoid SA writing Bayes files? in that case a non-exclusive lock would be enough...
Thanks!

Pedro.
Re: Bayes files LOCK [ In reply to ]
On 12.02.20 18:03, Pedro David Marco wrote:
>i am getting? errors from Byes because it is not able to lock Bayes files...?
>Error log is:?
>? bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_* R/W: lock failed: Interrupted system call
>
>?SA tries to lock bayes files always in "EXclusive mode", hence when a sa takes too long, all other processes have to wait for the lock...
>
>This is my config:
>use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn 0bayes_auto_expire 0lock_method flock

this is your problem. bayes is designed for one user. not for all of them.
If you want, you can move bayes database to SQL or to redis, but I don't
recommend that for multiuser machine.
For a single-user, bayes database can be simply in your $HOME/.spamassassin/

>SA uses EXclusive lock because it has "to write"...? my question is...? ?
> Is there any way to avoid SA writing Bayes files? in that case a
> non-exclusive lock would be enough...

no.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam is for losers who can't get business any other way.
Re: Bayes files LOCK [ In reply to ]
Fully aligned with my suspicions...
I love Redis...  i think someone posted sometime ago a performance table showing local / sql / redis  ... i will look for it...
but it would not surprise me if remote Redis is even faster than local DB...
Thanks! ?akujem mnohokrát Fantomas!
Pedro.

On Wednesday, February 12, 2020, 7:32:42 PM GMT+1, Matus UHLAR - fantomas <uhlar@fantomas.sk> wrote:

On 12.02.20 18:03, Pedro David Marco wrote:
>i am getting  errors from Byes because it is not able to lock Bayes files... 
>Error log is: 
>  bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_* R/W: lock failed: Interrupted system call
>
> SA tries to lock bayes files always in "EXclusive mode", hence when a sa takes too long, all other processes have to wait for the lock...
>
>This is my config:
>use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn 0bayes_auto_expire 0lock_method flock

this is your problem. bayes is designed for one user. not for all of them.
If you want, you can move bayes database to SQL or to redis, but I don't
recommend that for multiuser machine.
For a single-user, bayes database can be simply in your $HOME/.spamassassin/

>SA uses EXclusive lock because it has "to write"...  my question is...   
> Is there any way to avoid SA writing Bayes files?  in that case a
> non-exclusive lock would be enough...

no.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam is for losers who can't get business any other way.
Re: Bayes files LOCK [ In reply to ]
I don't know for sure but Alex Broens aka axb is a master at Bayes with
Redis. Definitely look for his postings.
--
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171


On Wed, Feb 12, 2020 at 2:02 PM Pedro David Marco <pedrod_marco@yahoo.com>
wrote:

> Fully aligned with my suspicions...
>
> I love Redis... i think someone posted sometime ago a performance table
> showing local / sql / redis ... i will look for it...
>
> but it would not surprise me if remote Redis is even faster than local
> DB...
>
> Thanks! ?akujem mnohokrát Fantomas!
>
> Pedro.
>
>
> On Wednesday, February 12, 2020, 7:32:42 PM GMT+1, Matus UHLAR - fantomas <
> uhlar@fantomas.sk> wrote:
>
>
> On 12.02.20 18:03, Pedro David Marco wrote:
> >i am getting errors from Byes because it is not able to lock Bayes
> files...
> >Error log is:
> > bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_* R/W:
> lock failed: Interrupted system call
> >
> > SA tries to lock bayes files always in "EXclusive mode", hence when a sa
> takes too long, all other processes have to wait for the lock...
> >
> >This is my config:
> >use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn
> 0bayes_auto_expire 0lock_method flock
>
> this is your problem. bayes is designed for one user. not for all of them.
> If you want, you can move bayes database to SQL or to redis, but I don't
> recommend that for multiuser machine.
> For a single-user, bayes database can be simply in your
> $HOME/.spamassassin/
>
>
> >SA uses EXclusive lock because it has "to write"... my question is...
> > Is there any way to avoid SA writing Bayes files? in that case a
> > non-exclusive lock would be enough...
>
> no.
>
> --
> Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> Spam is for losers who can't get business any other way.
>
>
Re: Bayes files LOCK [ In reply to ]
Kevin A. McGrail skrev den 2020-02-12 23:14:
> I don't know for sure but Alex Broens aka axb is a master at Bayes
> with Redis. Definitely look for his postings.

lets see sqlite with DBI: supported in spamassassin

i am happy with postgresql
Re: Bayes files LOCK [ In reply to ]
On Wed, 12 Feb 2020, Pedro David Marco wrote:

> Hi..
> i am getting? errors from Byes because it is not able to lock Bayes files...?
> Error log is:?
> ? bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_* R/W: lock failed: Interrupted system call
>
> ?SA tries to lock bayes files always in "EXclusive mode", hence when a
> sa takes too long, all other processes have to wait for the lock...
>
> This is my config:
> use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn 0bayes_auto_expire 0lock_method flock

Turn off Bayes auto-expire and do that in a cron job.

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
People seem to have this obsession with objects and tools as being
dangerous in and of themselves, as though a weapon will act of its
own accord to cause harm. A weapon is just a force multiplier. It's
*humans* that are (or are not) dangerous.
-----------------------------------------------------------------------
Today: Abraham Lincoln's and Charles Darwin's 211th Birthdays
Re: Bayes files LOCK [ In reply to ]
On 2020-02-13 04:11, John Hardin wrote:
> On Wed, 12 Feb 2020, Pedro David Marco wrote:
>
>> Hi..
>> i am getting  errors from Byes because it is not able to lock Bayes
>> files...
>> Error log is:
>>   bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_*
>> R/W: lock failed: Interrupted system call
>>
>>  SA tries to lock bayes files always in "EXclusive mode", hence when a
>> sa takes too long, all other processes have to wait for the lock...
>>
>> This is my config:
>> use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn
>> 0bayes_auto_expire 0lock_method flock
>
> Turn off Bayes auto-expire and do that in a cron job.
>

and use SDBM which is a bit faster.

in local.cf
bayes_store_module Mail::SpamAssassin::BayesStore::SDBM
Re: Bayes files LOCK [ In reply to ]
On 12.02.20 19:02, Pedro David Marco wrote:
> Fully aligned with my suspicions...
>I love Redis...? i think someone posted sometime ago a performance table showing local / sql / redis? ... i will look for it...
>but it would not surprise me if remote Redis is even faster than local DB...

last time I checked (half year maybe?) local redis was slower than local file.
However, it should be faster for multiple accesses and do expiration on its
own.

However, what others have advised:
- do expiration from cron job, not automatically
- use SDBM
should help too.

>Thanks! ?akujem mnohokr?t Fantomas!

you are welcome.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"One World. One Web. One Program." - Microsoft promotional advertisement
"Ein Volk, ein Reich, ein Fuhrer!" - Adolf Hitler
Re: Bayes files LOCK [ In reply to ]
>On 12.02.20 19:02, Pedro David Marco wrote:
>>Fully aligned with my suspicions...
>>I love Redis...? i think someone posted sometime ago a performance table showing local / sql / redis? ... i will look for it...
>>but it would not surprise me if remote Redis is even faster than local DB...

On 13.02.20 09:39, Matus UHLAR - fantomas wrote:
>last time I checked (half year maybe?) local redis was slower than local file.
>However, it should be faster for multiple accesses and do expiration on its
>own.
>
>However, what others have advised:
>- do expiration from cron job, not automatically
>- use SDBM
>should help too.

and I forgot to add:

storing in redis makes only sense for shared BAYES database.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Windows found: (R)emove, (E)rase, (D)elete
Re: Bayes files LOCK [ In reply to ]
On Wed, 12 Feb 2020 18:03:01 +0000 (UTC)
Pedro David Marco wrote:

> Hi..
> i am getting  errors from Byes because it is not able to lock Bayes
> files... Error log is: 
>   bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_*
> R/W: lock failed: Interrupted system call
>
>
>  SA tries to lock bayes files always in "EXclusive mode", hence when
> a sa takes too long, all other processes have to wait for the lock...
>
> This is my config:
> use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn
> 0bayes_auto_expire 0lock_method flock
>
> SA uses EXclusive lock because it has "to write"...  my question
> is...    Is there any way to avoid SA writing Bayes files? in that
> case a non-exclusive lock would be enough... Thanks!

Bayes doesn't write on scans (unless it does an opportunistic sync or
expiry):

$ printf '\nhello world' | spamassassin -D bayes 1>/dev/null
...
dbg: bayes: tie-ing to DB file R/O .../bayes_toks
dbg: bayes: tie-ing to DB file R/O .../bayes_seen

What you are seeing is characteristic of a failed auto-expire, but you
have bayes_auto_expire 0.

Are you allowing very large emails to be scanned or learned?
Re: Bayes files LOCK [ In reply to ]
>On Wed, 12 Feb 2020 18:03:01 +0000 (UTC)
>Pedro David Marco wrote:
>> i am getting? errors from Byes because it is not able to lock Bayes
>> files... Error log is:?
>> ? bayes: cannot open bayes databases /etc/spamassassin/bayes/bayes_*
>> R/W: lock failed: Interrupted system call
>>
>>
>> ?SA tries to lock bayes files always in "EXclusive mode", hence when
>> a sa takes too long, all other processes have to wait for the lock...
>>
>> This is my config:
>> use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn
>> 0bayes_auto_expire 0lock_method flock
>>
>> SA uses EXclusive lock because it has "to write"...? my question
>> is...? ? Is there any way to avoid SA writing Bayes files? in that
>> case a non-exclusive lock would be enough... Thanks!

On 13.02.20 12:30, RW wrote:
>Bayes doesn't write on scans (unless it does an opportunistic sync or
>expiry):

doesn't it record token access times to journal?

>$ printf '\nhello world' | spamassassin -D bayes 1>/dev/null
>...
>dbg: bayes: tie-ing to DB file R/O .../bayes_toks
>dbg: bayes: tie-ing to DB file R/O .../bayes_seen
>
>What you are seeing is characteristic of a failed auto-expire, but you
>have bayes_auto_expire 0.
>
>Are you allowing very large emails to be scanned or learned?

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
LSD will make your ECS screen display 16.7 million colors
Re: Bayes files LOCK [ In reply to ]
>On 13.02.20 12:30, RW wrote:
>>Bayes doesn't write on scans (unless it does an opportunistic sync or
>expiry):
>
>doesn't it record token access times to journal?

i think SA always does an EXclusive lock despite the parameters, probably because of that, Fantomas....
I will try with SDBM...
Thanks...
Pedro.
Re: Bayes files LOCK [ In reply to ]
On Thu, 13 Feb 2020 15:05:57 +0100
Matus UHLAR - fantomas wrote:

> >On Wed, 12 Feb 2020 18:03:01 +0000 (UTC)
> >Pedro David Marco wrote:
> >> i am getting  errors from Byes because it is not able to lock Bayes
> >> files... Error log is: 
> >>   bayes: cannot open bayes databases
> >> /etc/spamassassin/bayes/bayes_* R/W: lock failed: Interrupted
> >> system call
> >>
> >>
> >>  SA tries to lock bayes files always in "EXclusive mode", hence
> >> when a sa takes too long, all other processes have to wait for the
> >> lock...
> >>
> >> This is my config:
> >> use_bayes 1bayes_path /etc/spamassassin/bayes/bayesbayes_auto_learn
> >> 0bayes_auto_expire 0lock_method flock
> >>
> >> SA uses EXclusive lock because it has "to write"...  my question
> >> is...    Is there any way to avoid SA writing Bayes files? in that
> >> case a non-exclusive lock would be enough... Thanks!
>
> On 13.02.20 12:30, RW wrote:
> >Bayes doesn't write on scans (unless it does an opportunistic sync or
> >expiry):
>
> doesn't it record token access times to journal?


I meant it doesn't write to a database file. The main point of the
journal is that it bypasses reader-writer locking.
Re: Bayes files LOCK [ In reply to ]
On Thu, 13 Feb 2020 15:29:46 +0000 (UTC)
Pedro David Marco wrote:

>
> >On 13.02.20 12:30, RW wrote:
> >>Bayes doesn't write on scans (unless it does an opportunistic sync
> >>or
> >expiry):
> >
> >doesn't it record token access times to journal?
>
> i think SA always does an EXclusive lock despite the parameters,
> probably because of that,

That would defeat the object of having a journal file.

Even if you are right, it doesn't really explain anything because it
applies to everyone using BDB/DBM/SDBM.

IIWY I'd be looking at what's different for you.

> I will try with SDBM...
Re: Bayes files LOCK [ In reply to ]
>On Friday, February 14, 2020, 1:17:29 PM GMT+1, RW <rwmaillists@googlemail.com> wrote:
>That would defeat the object of having a journal file.>>Even if you are right, it doesn't really explain anything because it>applies to everyone using  BDB/DBM/SDBM. >>IIWY I'd be looking at what's different for you.
I basically agree whit you RW... 
but my hopes are that using any other DB than files may allow concurrent write to the DB...
Re: Bayes files LOCK [ In reply to ]
On Fri, 14 Feb 2020 14:38:05 +0000 (UTC)
Pedro David Marco wrote:

>
>
> >On Friday, February 14, 2020, 1:17:29 PM GMT+1, RW
> ><rwmaillists@googlemail.com> wrote: That would defeat the object of
> >having a journal file.>>Even if you are right, it doesn't really
> >explain anything because it>applies to everyone using  BDB/DBM/SDBM.
> >>>IIWY I'd be looking at what's different for you.
> I basically agree whit you RW... 
> but my hopes are that using any other DB than files may allow
> concurrent write to the DB...

I was looking at it in a bit more detail and it looks like there isn't
a reader-writer lock, just write locks for the toks and seen
files. As scans defer their writes through the journal they are
lockless.
Re: Bayes files LOCK [ In reply to ]
On Fri, 14 Feb 2020, RW wrote:

> On Fri, 14 Feb 2020 14:38:05 +0000 (UTC)
> Pedro David Marco wrote:
>
>>> On Friday, February 14, 2020, 1:17:29 PM GMT+1, RW
>>> <rwmaillists@googlemail.com> wrote: That would defeat the object of
>>> having a journal file.>>Even if you are right, it doesn't really
>>> explain anything because it>applies to everyone using  BDB/DBM/SDBM.
>>>>> IIWY I'd be looking at what's different for you.
>> I basically agree whit you RW... 
>> but my hopes are that using any other DB than files may allow
>> concurrent write to the DB...
>
> I was looking at it in a bit more detail and it looks like there isn't
> a reader-writer lock, just write locks for the toks and seen
> files. As scans defer their writes through the journal they are
> lockless.

So, auto-training may be problematic w/r/t locking as well. I presume that
window is coded to be as small as possible.


--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Once more, please; I missed it the last time: what's the difference
between "Quantitative Easing" and "Counterfeiting"?
-----------------------------------------------------------------------
8 days until George Washington's 288th Birthday
Re: Bayes files LOCK [ In reply to ]
On Friday, February 14, 2020, 7:46:18 PM GMT+1, John Hardin <jhardin@impsec.org> wrote:
>> I was looking at it in a bit more detail and it looks like there isn't
>> a reader-writer lock,  just write locks for the toks and seen
>> files. As scans defer their writes through the journal they are
>> lockless.
>So, auto-training may be problematic w/r/t locking as well. I presume that
>window is coded to be as small as possible.


Sure RW and John... but in theory the use of a DB should avoid that need, that makes sense when using just files...
Pedro.
Re: Bayes files LOCK [ In reply to ]
On Fri, 14 Feb 2020 10:46:00 -0800 (PST)
John Hardin wrote:

> On Fri, 14 Feb 2020, RW wrote:
>

> > I was looking at it in a bit more detail and it looks like there
> > isn't a reader-writer lock, just write locks for the toks and seen
> > files. As scans defer their writes through the journal they are
> > lockless.
>
> So, auto-training may be problematic w/r/t locking as well. I presume
> that window is coded to be as small as possible.

I wasn't referring auto-training, that's done at the end under locking,
and picking-up count updates made by other processes since the Bayes
classification. The OP has 'bayes_auto_learn 0', so it's not really
relevant.