Mailing List Archive

On 08/09/2023 11:16, Mike Clarke via Exim-users wrote:
> I've just upgraded from 4.95 to 4.96 and have started to have problems with exim_tidydb
>
> curlew:/tmp# exim_tidydb /var/spool/exim wait-remote_auth_smtp
> Tidying Exim hints database /var/spool/exim/db/wait-remote_auth_smtp
> ** Failed to open DBM file /var/spool/exim/db/wait-remote_auth_smtp for writing:
> No such file or directorycurlew:/tmp# exim_tidydb /var/spool/exim wait-remote_smtp
> Tidying Exim hints database /var/spool/exim/db/wait-remote_smtp
> ** Failed to open DBM file /var/spool/exim/db/wait-remote_smtp for writing:
> No such file or directory
> But both these files exist

Looks fun. Try running it under strace? It presumably
does an open() syscall which returns an error. Check the
args to that open.

--
Cheers,
Jeremy


--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Fri, 8 Sep 2023, Mike Clarke via Exim-users wrote:

> I've just upgraded from 4.95 to 4.96 and have started to have problems with exim_tidydb
>
> curlew:/tmp# exim_tidydb /var/spool/exim wait-remote_auth_smtp
> Tidying Exim hints database /var/spool/exim/db/wait-remote_auth_smtp
> ** Failed to open DBM file /var/spool/exim/db/wait-remote_auth_smtp for writing:
> No such file or directorycurlew:/tmp# exim_tidydb /var/spool/exim wait-remote_smtp
> Tidying Exim hints database /var/spool/exim/db/wait-remote_smtp
> ** Failed to open DBM file /var/spool/exim/db/wait-remote_smtp for writing:
> No such file or directory
> But both these files exist
>
> curlew:/tmp# ls -l /var/spool/exim/db
> total 5
> -rw-r----- 1 mailnull mail 0 14 Oct 2022 retry.lockfile
> -rw-r----- 1 mailnull mail 131072 4 Sep 11:21 wait-remote_auth_smtp
> -rw-r----- 1 mailnull mail 0 22 Aug 2020 wait-remote_auth_smtp.lockfile
> -rw-r----- 1 mailnull mail 131072 4 Sep 08:41 wait-remote_smtp
> -rw-r----- 1 mailnull mail 0 19 Jun 2021 wait-remote_smtp.lockfile
>
> curlew:/tmp# exim --version
> Exim version 4.96 #1 (FreeBSD 13.2) built 29-Aug-2023 15:38:40
> Copyright (c) University of Cambridge, 1995 - 2018
> (c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 - 2022
> Probably Berkeley DB version 1.8x (compatibility mode)
> Support for: crypteq iconv() IPv6 use_setclassresources PAM Perl Expand_dlfunc OpenSSL
> TLS_resume Content_Scan
> ning DKIM DNSSEC Event I18N OCSP PIPECONNECT PRDR Queue_Ramp SPF
> TCP_Fast_Open
> Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb
> dsearch passwd
> Authenticators: cram_md5 dovecot plaintext spa
> Routers: accept dnslookup ipliteral manualroute queryprogram redirect
> Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
> Malware: f-protd f-prot6d drweb fsecure sophie clamd avast sock cmdline
> Fixed never_users: 0
> Configure owner: 0:0
> Size of off_t: 8
> Configuration file is /usr/local/etc/exim/configure

Configure owner is 0:0 - is /var/spool/exim/db mounted rootsquash ?
(It should not be.)

Does /usr/local/etc/exim/configure set exim_user to mailnull ?

Check permissions on parents of /var/spool/exim/db too.

... but why any of these would have changed between 4.95 and 4.96,
I cannot guess.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Friday, 8 September 2023 12:09:12 BST Andrew C Aitchison via Exim-users wrote:

> Configure owner is 0:0 - is /var/spool/exim/db mounted rootsquash ?
> (It should not be.)

I'm not familiar with rootsquash but I understand it only relates to NFS mounts. My /var/
spool/exim/db is part of the local file system so I assume this won't be relevant.

> Does /usr/local/etc/exim/configure set exim_user to mailnull ?

curlew:/tmp# grep exim_user /usr/local/etc/exim/configure
exim_user = mailnull

> Check permissions on parents of /var/spool/exim/db too.

curlew:/tmp# ls -ld /var/spool/exim/db /var/spool/exim
drwxr-x--- 8 mailnull mail 10 25 Feb 2021 /var/spool/exim/
drwxr-x--- 2 mailnull mail 9 8 Sep 13:49 /var/spool/exim/db/
--
Mike Clarke

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Friday, 8 September 2023 11:36:14 BST Jeremy Harris via Exim-users wrote:

> Looks fun. Try running it under strace? It presumably
> does an open() syscall which returns an error. Check the
> args to that open.

Here's the last few lines of running a trace with the command
exim_tidydb /var/spool/exim wait-remote_smtp

=======================================================================
open("/var/spool/exim/db/wait-remote_smtp.db",O_RDWR|O_CLOEXEC,00) = 4 (0x4)
fstat(4,{ mode=-rw-r----- ,inode=1180752,size=131072,blksize=131072 }) = 0 (0x0)
read(4,"\0\^F\^Ua\0\0\0\^B\0\0\^D\M-R\0"...,260) = 260 (0x104)
pread(4,"\0\0\M-z\^?\0\M^@\M^[\^?\M-%\^?"...,32768,0x8000) = 32768 (0x8000)
pread(4,"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,32768,0x10000) = 32768 (0x8000)
pwrite(4,"\0\^F\^Ua\0\0\0\^B\0\0\^D\M-R\0"...,260,0x0) = 260 (0x104)
fsync(4) = 0 (0x0)
close(4) = 0 (0x0)
close(3) = 0 (0x0)
write(1,"Tidying complete\n",17) = 17 (0x11)
exit(0x0)
process exit, rval = 0
=======================================================================

So it's looking for wait-remote_smtp.db instead of wait-remote_smtp.

As a temporary workaround I've added symbolic links:

curlew:/tmp# ls -l /var/spool/exim/db
total 6
-rw-r----- 1 mailnull mail 0 14 Oct 2022 retry.lockfile
-rw-r----- 1 mailnull mail 131072 8 Sep 19:40 wait-remote_auth_smtp
lrwxr-xr-x 1 root mail 21 8 Sep 13:49 wait-remote_auth_smtp.db@ -> wait-
remote_auth_smtp
-rw-r----- 1 mailnull mail 0 22 Aug 2020 wait-remote_auth_smtp.lockfile
-rw-r----- 1 mailnull mail 131072 8 Sep 19:41 wait-remote_smtp
lrwxr-xr-x 1 root mail 16 8 Sep 13:49 wait-remote_smtp.db@ -> wait-remote_smtp
-rw-r----- 1 mailnull mail 0 19 Jun 2021 wait-remote_smtp.lockfile

With these symlinks exim_tidydb now works without errors. I was wondering if this is a
bug with 4.96 or if the new version now require the database files to have a .db suffix so I
fired up a new virtual system with exim 4.96 which created database files with the ,db
suffix. So for my current system should I just rename the database files, or even just
delete them and let exim create new ones?

--
Mike Clarke

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On 08/09/2023 20:27, Mike Clarke via Exim-users wrote:
> So it's looking for wait-remote_smtp.db instead of wait-remote_smtp.

> With these symlinks exim_tidydb now works without errors. I was wondering if this is a
> bug with 4.96 or if the new version now require the database files to have a .db suffix so I
> fired up a new virtual system with exim 4.96 which created database files with the ,db
> suffix. So for my current system should I just rename the database files, or even just
> delete them and let exim create new ones?

Delete them.

Either the library used for these hints DBs has changed
the names of files it uses, or a different library is being used.
Starting over is always safe so long as all "old" exim processes
have terminated. You lose any long-term history (but for this
specific use, we don't even care about that).
--
Cheers,
Jeremy


--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Friday, 8 September 2023 20:49:26 BST Jeremy Harris via Exim-users wrote:

> On 08/09/2023 20:27, Mike Clarke via Exim-users wrote:
> > So it's looking for wait-remote_smtp.db instead of wait-remote_smtp.
> >
> > With these symlinks exim_tidydb now works without errors. I was wondering
> > if this is a bug with 4.96 or if the new version now require the
> > database files to have a .db suffix so I fired up a new virtual system
> > with exim 4.96 which created database files with the ,db suffix. So for
> > my current system should I just rename the database files, or even just
> > delete them and let exim create new ones?
>
> Delete them.
>
> Either the library used for these hints DBs has changed
> the names of files it uses, or a different library is being used.
> Starting over is always safe so long as all "old" exim processes
> have terminated. You lose any long-term history (but for this
> specific use, we don't even care about that).

Thanks. Before resorting to deleting the old files I tried with just renaming them with a
'.db' suffix (after stopping the exim service) and things have been running fine for the last
few days.

However it does appear that there is a minor bug in need of fixing. Tracing exim_tidydb /
var/spool/exim wait-remote_smtp when the correctly named database file doesn't exist
shows that it tries to open remote_auth_smtp.db but refers to remote_auth_smtp (without
the .db suffix) in the error message

open("/var/spool/exim/db/wait-remote_smtp.db",O_RDWR|O_CLOEXEC,00) ERR#2 'No
such file or directory'
fstatat(AT_FDCWD,"/usr/share/nls/C/libc.cat",0x820551010,0x0) ERR#2 'No such file or
directory'
fstatat(AT_FDCWD,"/usr/share/nls/libc/C",0x820551010,0x0) ERR#2 'No such file or
directory'
fstatat(AT_FDCWD,"/usr/local/share/nls/C/libc.cat",0x820551010,0x0) ERR#2 'No such file
or directory'
fstatat(AT_FDCWD,"/usr/local/share/nls/libc/C",0x820551010,0x0) ERR#2 'No such file or
directory'
** Failed to open DBM file /var/spool/exim/db/wait-remote_smtp for writing:
write(1,"** Failed to open DBM file /var/"...,76) = 76 (0x4c)
No such file or directory
write(1," No such file or directory\n",29) = 29 (0x1d)

--
Mike Clarke

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On 8.09.2023 12:16, Mike Clarke via Exim-users wrote:
> I've just upgraded from 4.95 to 4.96 and have started to have problems with exim_tidydb
> Exim version 4.96 #1 (FreeBSD 13.2) built 29-Aug-2023 15:38:40

I had similar problem on FreeBSD but on 12.4 (and visible after upgrade
from 4.95 to 4.96).

I think that on FreeBSD dbm doesn't use file pair mode (.pag + .dir) but
only one-file-mode with .db

I was getting in main exim log:

failed to open DB file /var/spool/exim/db/retry: File exists
failed to open DB file /var/spool/exim/db/wait-remote_smtp: File exists

and also for once=

R=mysql_vacation T=vacation_autoreply
defer (17): File exists DT=0s: Failed to open ndbm file
/mail/db/once-e31077438ea612e370820729a6503c11.db when sending message

owners are not changed and privileges are proper but hints are not saved.

in config I had
once = /mail/db/once-${md5:${local_part}@${domain}}.db

There's changelog entry for probably related change in 4.96:

JH/20 When built with NDBM for hints DB's check for nonexistence of a
name supplied as the db file-pair basename. Previously, if a directory
path was given, for example via the autoreply "once" option, the DB
file.pag and file.dir files would be created in that directory's parent.

https://github.com/Exim/exim/commit/7d238090f5fbd23d7077eba95c1ba8f2ee016a00


I think this is what happened (not that I understand why)
1. in 4.95 exim used retry file as database
2. in 4.96 exim used retry.db file as database
3. but the check noticed existence of old "retry" (which was not
directory) and failed

Nevertheless, I stopped exim, deleted all files from /var/spool/exim/db
and from /mail/db (my place for once- files), restarted and now it seems
to work well.

If I'm right with the guess above then maybe this could be a bit more
robust - because it breaks upgrade process in a bit non-obvious way.

best regards
--
Marcin Gryszkalis, PGP 0xA5DBEEC7 http://fork.pl/gpg.txt

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/