Mailing List Archive

Hello,

I'm running amavisd-new (milter) with spamassassin 4.x on FreeBSD, both from pkg, and I see the same problem. It appears that my /var/amavis/tmp holds every single sspamassassin tmp file from the day I've upgraded to 4.x.

cheers.

> On 20 Jun 2023, at 15:07, natan <natan@epf.pl> wrote:
>
> Hi
> I have extrenal serwers with amavis+spamassassin (via cpan)
> 4 of them debian10 and spamassin-3.4.4
> 2 of them debian11 and spamassassin-4.x
>
> I have a problem (only debianb10 and spamassassin-4.x) with many many files like ".spamassassin99712ZzMoDatmp" in /var/lib/amavis/tmp/
>
> example:
> debian10 + spamassin-3.4.4 ~ 100MB
> debian11 + spamassassin-4-x ~8GB
>
> like
> 20K /var/lib/amavis/tmp/.spamassassin271206ncRha2tmp
> 60K /var/lib/amavis/tmp/.spamassassin271211dRnXCJtmp
> 248K /var/lib/amavis/tmp/.spamassassin271211ODEgWgtmp
> 60K /var/lib/amavis/tmp/.spamassassin271211SDyjfWtmp
> 32K /var/lib/amavis/tmp/.spamassassin271223eMCFtWtmp
> 116K /var/lib/amavis/tmp/.spamassassin271223OP9AQXtmp
> 100K /var/lib/amavis/tmp/.spamassassin271223Vhq7iHtmp
> 16K /var/lib/amavis/tmp/.spamassassin271223zRUFiwtmp
> 36K /var/lib/amavis/tmp/.spamassassin27122488JRTwtmp
> 12K /var/lib/amavis/tmp/.spamassassin271224DwXjUvtmp
> 48K /var/lib/amavis/tmp/.spamassassin271224ndlcZZtmp
> 4,0K /var/lib/amavis/tmp/.spamassassin271224psFL1itmp
> 132K /var/lib/amavis/tmp/.spamassassin27122814VcUntmp
> 228K /var/lib/amavis/tmp/.spamassassin271228ITPqiKtmp
> 8,0K /var/lib/amavis/tmp/.spamassassin271228MHHOoPtmp
>
>
> For test I downgrade one spamassassin from 4.x to 3.4.6 and problem not exists
>
> any idea ?
>
> --
>

> For test I downgrade one spamassassin from 4.x to 3.4.6 and problem
> not exists
Hm, I don't see any significant code-difference regarding tempfiles.
> any idea ?
> root@a48c6a1d5fb0:~# sudo -u amavis perl -Mv5.14 -MFile::Spec -e 'say
> File::Spec->tmpdir'
> /tmp
What output do you get?

W dniu 20.06.2023 o 16:50, Damian pisze:
>> For test I downgrade one spamassassin from 4.x to 3.4.6 and problem
>> not exists
> Hm, I don't see any significant code-difference regarding tempfiles.
>> any idea ?
>> root@a48c6a1d5fb0:~# sudo -u amavis perl -Mv5.14 -MFile::Spec -e 'say
>> File::Spec->tmpdir'
>> /tmp
> What output do you get?

In debian11 with spamasassin-4.x
root@amavis-4:~# su - amavis
$ perl -Mv5.14 -MFile::Spec -e 'say File::Spec->tmpdir'
/tmp


In debian11 with spamassassin-3.4.x
root@amavis5:~# su amavis
$ perl -Mv5.14 -MFile::Spec -e 'say File::Spec->tmpdir'
/tmp

--

On 2023-06-20 at 09:39:04 UTC-0400 (Tue, 20 Jun 2023 15:39:04 +0200)
Patrick Proniewski <patpro@patpro.net>
is rumored to have said:

> Hello,
>
> I'm running amavisd-new (milter) with spamassassin 4.x on FreeBSD,
> both from pkg, and I see the same problem. It appears that my
> /var/amavis/tmp holds every single sspamassassin tmp file from the day
> I've upgraded to 4.x.
>

Did you upgrade or reconfigure Amavis at the same time?

> cheers.
>
>> On 20 Jun 2023, at 15:07, natan <natan@epf.pl> wrote:
>>
>> Hi
>> I have extrenal serwers with amavis+spamassassin (via cpan)
>> 4 of them debian10 and spamassin-3.4.4
>> 2 of them debian11 and spamassassin-4.x
>>
>> I have a problem (only debianb10 and spamassassin-4.x) with many many
>> files like ".spamassassin99712ZzMoDatmp" in /var/lib/amavis/tmp/
>>
>> example:
>> debian10 + spamassin-3.4.4 ~ 100MB
>> debian11 + spamassassin-4-x ~8GB
>>
>> like
>> 20K /var/lib/amavis/tmp/.spamassassin271206ncRha2tmp
>> 60K /var/lib/amavis/tmp/.spamassassin271211dRnXCJtmp
>> 248K /var/lib/amavis/tmp/.spamassassin271211ODEgWgtmp
>> 60K /var/lib/amavis/tmp/.spamassassin271211SDyjfWtmp
>> 32K /var/lib/amavis/tmp/.spamassassin271223eMCFtWtmp
>> 116K /var/lib/amavis/tmp/.spamassassin271223OP9AQXtmp
>> 100K /var/lib/amavis/tmp/.spamassassin271223Vhq7iHtmp
>> 16K /var/lib/amavis/tmp/.spamassassin271223zRUFiwtmp
>> 36K /var/lib/amavis/tmp/.spamassassin27122488JRTwtmp
>> 12K /var/lib/amavis/tmp/.spamassassin271224DwXjUvtmp
>> 48K /var/lib/amavis/tmp/.spamassassin271224ndlcZZtmp
>> 4,0K /var/lib/amavis/tmp/.spamassassin271224psFL1itmp
>> 132K /var/lib/amavis/tmp/.spamassassin27122814VcUntmp
>> 228K /var/lib/amavis/tmp/.spamassassin271228ITPqiKtmp
>> 8,0K /var/lib/amavis/tmp/.spamassassin271228MHHOoPtmp
>>
>>
>> For test I downgrade one spamassassin from 4.x to 3.4.6 and problem
>> not exists
>>
>> any idea ?
>>
>> --
>>


--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

> In debian11 with spamasassin-4.x
> root@amavis-4:~# su - amavis
> $ perl -Mv5.14 -MFile::Spec -e 'say File::Spec->tmpdir'
> /tmp
>
>
> In debian11 with spamassassin-3.4.x
> root@amavis5:~# su amavis
> $ perl -Mv5.14 -MFile::Spec -e 'say File::Spec->tmpdir'
> /tmp

Nevermind, Debian's amavis actually defines TMPDIR as /var/lib/amavis/tmp.

On 20 Jun 2023, at 17:49, Bill Cole <sausers-20150205@billmail.scconsult.com> wrote:
>
> On 2023-06-20 at 09:39:04 UTC-0400 (Tue, 20 Jun 2023 15:39:04 +0200)
> Patrick Proniewski <patpro@patpro.net>
> is rumored to have said:
>
>> Hello,
>>
>> I'm running amavisd-new (milter) with spamassassin 4.x on FreeBSD, both from pkg, and I see the same problem. It appears that my /var/amavis/tmp holds every single sspamassassin tmp file from the day I've upgraded to 4.x.
>>
>
> Did you upgrade or reconfigure Amavis at the same time?


I did not upgrade. I did not reconfigure either.
Do you suggest I should deinstall / reinstall Amavisd-new?

thanks
patpro

Patrick Proniewski skrev den 2023-06-20 18:33:

> I did not upgrade. I did not reconfigure either.
> Do you suggest I should deinstall / reinstall Amavisd-new?

if i just did install amavisd-new in gentoo nothing will work unmodified
in amavisd.conf

i don't belive debian is more helpfull

reinstall gives same problem as you have now

vim /etc/amavisd.conf

note for spamassassin 4.x there is many plugins with bugs unless debian
maintainers do a diff -urp oldtarball newfilesdir >patch.patch

i do this patch from spamassassin 4 to sa-trunk, no problems

wait for 4.0.1 else

> i don't belive debian is more helpfull
>
> reinstall gives same problem as you have now

FWIW, I cannot reproduce the issue either on (Debian bookworm with
standard packages) or (Debian bullseye with SA4 from CPAN), so it is not
categorically broken.

> /var/lib/amavis/tmp/ CREATE .spamassassin1730317ACUmn1tmp
> /var/lib/amavis/tmp/ OPEN .spamassassin1730317ACUmn1tmp
> /var/lib/amavis/tmp/ MODIFY .spamassassin1730317ACUmn1tmp
> /var/lib/amavis/tmp/ CLOSE_WRITE,CLOSE .spamassassin1730317ACUmn1tmp
> /var/lib/amavis/tmp/ OPEN .spamassassin1730317ACUmn1tmp
> /var/lib/amavis/tmp/ ACCESS .spamassassin1730317ACUmn1tmp
> /var/lib/amavis/tmp/ CLOSE_NOWRITE,CLOSE .spamassassin1730317ACUmn1tmp
> /var/lib/amavis/tmp/ DELETE .spamassassin1730317ACUmn1tmp

On 2023-06-20 at 12:33:05 UTC-0400 (Tue, 20 Jun 2023 18:33:05 +0200)
Patrick Proniewski <patpro@patpro.net>
is rumored to have said:

> On 20 Jun 2023, at 17:49, Bill Cole
> <sausers-20150205@billmail.scconsult.com> wrote:
>>
>> On 2023-06-20 at 09:39:04 UTC-0400 (Tue, 20 Jun 2023 15:39:04 +0200)
>> Patrick Proniewski <patpro@patpro.net>
>> is rumored to have said:
>>
>>> Hello,
>>>
>>> I'm running amavisd-new (milter) with spamassassin 4.x on FreeBSD,
>>> both from pkg, and I see the same problem. It appears that my
>>> /var/amavis/tmp holds every single sspamassassin tmp file from the
>>> day I've upgraded to 4.x.
>>>
>>
>> Did you upgrade or reconfigure Amavis at the same time?
>
>
> I did not upgrade. I did not reconfigure either.
> Do you suggest I should deinstall / reinstall Amavisd-new?

Only if you'd prefer rebuilding the config from defaults instead of
trying to validate what you have. Or seek help from whatever channels
are available for Amavis.

I DO NOT KNOW that there is any particular way to make this happen, only
that if files are being left in the Amavis temp directory, something
closely tied to Amavis is causing them to land there and failing to
clean up. Despite the filenames, I'm skeptical that it is SA itself
creating these files, but rather suspect that something using SA and
seeing that as its temp directory (i.e. Amavis.) is causing the problem.
Easy for me to say, as I don't use Amavis. :)


I can say that I've had no similar problems on with such detritus
running SA4 via spamd/spamc on FreeBSD 12 or 13, nor when running it via
MIMEDefang on Darwin 15.

--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire

Bill Cole wrote:
> On 2023-06-20 at 12:33:05 UTC-0400 (Tue, 20 Jun 2023 18:33:05 +0200)
> Patrick Proniewski <patpro@patpro.net>
> is rumored to have said:
>
>> On 20 Jun 2023, at 17:49, Bill Cole
>> <sausers-20150205@billmail.scconsult.com> wrote:
>>>
>>> On 2023-06-20 at 09:39:04 UTC-0400 (Tue, 20 Jun 2023 15:39:04 +0200)
>>> Patrick Proniewski <patpro@patpro.net>
>>> is rumored to have said:
>>>
>>>> Hello,
>>>>
>>>> I'm running amavisd-new (milter) with spamassassin 4.x on FreeBSD,
>>>> both from pkg, and I see the same problem. It appears that my
>>>> /var/amavis/tmp holds every single sspamassassin tmp file from the
>>>> day I've upgraded to 4.x.
>>>>
>>>
>>> Did you upgrade or reconfigure Amavis at the same time?
>>
>>
>> I did not upgrade. I did not reconfigure either.
>> Do you suggest I should deinstall / reinstall Amavisd-new?
>
> Only if you'd prefer rebuilding the config from defaults instead of
> trying to validate what you have. Or seek help from whatever channels
> are available for Amavis.
>
> I DO NOT KNOW that there is any particular way to make this happen, only
> that if files are being left in the Amavis temp directory, something
> closely tied to Amavis is causing them to land there and failing to
> clean up. Despite the filenames, I'm skeptical that it is SA itself
> creating these files, but rather suspect that something using SA and
> seeing that as its temp directory (i.e. Amavis.) is causing the problem.
> Easy for me to say, as I don't use Amavis. :)
>
>
> I can say that I've had no similar problems on with such detritus
> running SA4 via spamd/spamc on FreeBSD 12 or 13, nor when running it via
> MIMEDefang on Darwin 15.
>

Just to put another data point in, I've had intermittent runs of
leftover temp files across several SA releases. Running spamd here so
it's nothing inherent with Amavis. Might be side effects of some blip
in processing with one or the other of Vipul's Razor, Pyzor, or a custom
clamd instance using only third-party and local spam-targeted sigs. To
date it's been rare enough I just clean them up by hand once in a while,
but the same cleanup could be dropped in cron.

-kgd

Hi
Files such as .spamassassin are deleted but not all.
Files that are not deleted do not appear to be invalid

I build spamassassin-4.x from cpan - not from repo


Workaound solutions is not good but works
find /var/lib/amavis/tmp/.spam* -mmin +200 -delete;

I'd rather not use it though

I thinking to downgrade to spamassassin-3.4.6
--

natan skrev den 2023-06-21 10:05:
> Hi
> Files such as .spamassassin are deleted but not all.
> Files that are not deleted do not appear to be invalid
>
> I build spamassassin-4.x from cpan - not from repo
>
>
> Workaound solutions is not good but works
> find /var/lib/amavis/tmp/.spam* -mmin +200 -delete;
>
> I'd rather not use it though
>
> I thinking to downgrade to spamassassin-3.4.6

check tmp path in all software used, this includes how perl versions
handle tmp dirs and files, in your case its amavisd that create dirs,
and its amavisd that tells spamassassin in with dir to scan email with
embed spamassassin engine in amavisd, check if other spamscanners works
in amavisd, eg change from spamassassin to spamdclient, and then start
spamd

does this leave temp files still ?

is cpan latest trunk version ?, i fear its not

> eg change from spamassassin to spamdclient, and then start spamd
>
> does this leave temp files still ?

Note: Look in /tmp then, not in /var/lib/amavis/tmp

> check tmp path in all software used, this includes how perl versions
> handle tmp dirs and files, in your case its amavisd that create dirs,
> and its amavisd that tells spamassassin in with dir to scan email with
> embed spamassassin engine in amavisd, check if other spamscanners
> works in amavisd, eg change from spamassassin to spamdclient, and then
> start spamd
>
> does this leave temp files still ?
>
> is cpan latest trunk version ?, i fear its not
>
>
>
This is problem not in amavis or system but in spamassassin-4.

Problem not exists in this sam machine in older spamassasin like
spamassasin-3.x
I tested via configurations

1)dovecot10 + spamassasin-3.x - problem not exists
2)dovecot11 + spamassasin-3.x - problem not exists
3)dovecot10 + spamassasin-4.x - problem exists
4)dovecot11 + spamassasin-4.x - problem exists

all dovecot have this same amavisd-new-2.11.1


If there was a problem in the path /tmp and/or amavis configuration, the
problem would be everywhere, not just with only in spamassassin-4.x

Again it says that spamassassin-4.x deletes teporary files but not all
and it creates more than it deletes
It seems as if for some reason specific files (e-mails) could not be deleted

ergo
Problem is in binary spamassassin-4.x

--

On Wed, Jun 21, 2023 at 12:00:41PM +0200, natan wrote:
> I tested via configurations
>
> 1)dovecot10 + spamassasin-3.x - problem not exists
> 2)dovecot11 + spamassasin-3.x - problem not exists
> 3)dovecot10 + spamassasin-4.x - problem exists
> 4)dovecot11 + spamassasin-4.x - problem exists
>
> all dovecot have this same amavisd-new-2.11.1
>
> If there was a problem in the path /tmp and/or amavis configuration, the
> problem would be everywhere, not just with only in spamassassin-4.x

That does not logically necessarily follow. It would only follow if
*only* possibilities were "some bug outside of spamassasssin ALWAYS
triggers" and "some bug outside of spamassassin NEVER triggers".

There however also exists the possibility that "some bug outside of
spamassassin ONLY SOMETIMES triggers - depending on some yet undefined
condition (timing, race condition, specific thing produced by SA4 but
not SA3, etc.)"

In which case, the problem might be something other than SA4, but
which triggers (only or just much more regularly) in conjunction with
SA4. But of course, it very well could be SA4.

To test, I'd use "strace -ff -o /tmp/log.txt -e file -p $PID" to
determine which process exactly is creating which files, and which
process is removing them, both on SA3 and SA4 machines, and see which
process creates something which it then doesn't remove.

> Again it says that spamassassin-4.x deletes teporary files but not all and
> it creates more than it deletes

That could be one of the possible reasons, yes.

> It seems as if for some reason specific files (e-mails) could not be deleted

If that is indeed the case, first step would be finding out exactly
which files are not being deleted (and by whom).

Another way to prove that the problem is exclusively in spamassassin
is to reproduce the problem ONLY with spamassassin (i.e. without
dovecot, amavisd, etc. being involved at all) - e.g. by manually
running "spamassassin -t /tmp/blah.mbox" and showing that it creates
and leaves behind some temporary files.

--
Opinions above are GNU-copylefted.

June 21, 2023 12:59 PM, "Matija Nalis" <mnalis-sa-list@voyager.hr> wrote:

> Another way to prove that the problem is exclusively in spamassassin
> is to reproduce the problem ONLY with spamassassin (i.e. without
> dovecot, amavisd, etc. being involved at all) - e.g. by manually
> running "spamassassin -t /tmp/blah.mbox" and showing that it creates
> and leaves behind some temporary files.


Fully agree with that.

More importantly, when one uses Amabosd-new with Spamassassin, there is no spamassassin running, Amavisd-new loads the SA lib and uses the code. So may be there is a mismatch between Amavisd-new and the new code of SA 4.x.

patpro

patpro@patpro.net skrev den 2023-06-21 13:33:
> June 21, 2023 12:59 PM, "Matija Nalis" <mnalis-sa-list@voyager.hr>
> wrote:
>
>> Another way to prove that the problem is exclusively in spamassassin
>> is to reproduce the problem ONLY with spamassassin (i.e. without
>> dovecot, amavisd, etc. being involved at all) - e.g. by manually
>> running "spamassassin -t /tmp/blah.mbox" and showing that it creates
>> and leaves behind some temporary files.
>
>
> Fully agree with that.
>
> More importantly, when one uses Amabosd-new with Spamassassin, there
> is no spamassassin running, Amavisd-new loads the SA lib and uses the
> code. So may be there is a mismatch between Amavisd-new and the new
> code of SA 4.x.

spamassassin spamd/spamc might use another $HOME then amavisd-new

i just ask people to check it

in @spam_scanners amavisd.conf it defaults to spamassassin, but change
this to spamdclient and start spamd in spamassassin, if this still leave
tmp files, create a bug for spamassassin

running spamassassin as root changes all problems

more info from OP please

Do you have any unmaintained or self-maintained SA Plugins (eg CRM114)?