Mailing List Archive

On Mon, Feb 05, 2024 at 03:01:03PM +0000, graeme vetterlein via Exim-users wrote:
> BTW this came to my attention while debugging a maildrop setup I noted it
> using flock(2) I also note it can use dot-lock files. Now fetchmail,
> maildrop, exmim4, procmail etc all work together in many combinations , so
> I'm sure this is a "solved problem" but I can't see any explicit references
> to it anywhere.

Fetchmail, exim and procmail run independently, within their own workspaces,
they have no concurency. The only point of concurency is access to mailbox,
controlled by maildrop, and it is properly protected by maildrop's locks.

However, if you want to serialize deliveries behind of maildrop (assuming
that only one maildrop process allowed to run, globally or per-mailbox),
you can use procmail as a wrapper to maildrop with procmail's builtin
dot-locking. Procmail package has also "lockfile" utility for external
locking, it can be called from Exim (via ${run..}).
--
Eugene Berdnikov

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On 2/5/24 15:01, graeme vetterlein via Exim-users wrote:
> But it occurs to me I'm rather "living on the edge" here I have multiple cronjobs an I guess with a single edit I could end up with multiple writers. So I guess I need to ensure maildrop and exim4 share a single view of locking .

Exim's locking for mailboxes depends on its config for the "appendfile"
transport being used. See:

https://exim.org/exim-html-current/doc/html/spec_html/ch-the_appendfile_transport.html
--
Cheers,
Jeremy


--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Mon, 5 Feb 2024, Jeremy Harris via Exim-users wrote:

> On 2/5/24 15:01, graeme vetterlein via Exim-users wrote:
>> But it occurs to me I'm rather "living on the edge" here I have multiple
>> cronjobs an I guess with a single edit I could end up with multiple
>> writers. So I guess I need to ensure maildrop and exim4 share a single view
>> of locking .
>
> Exim's locking for mailboxes depends on its config for the "appendfile"
> transport being used. See:
>
> https://exim.org/exim-html-current/doc/html/spec_html/ch-the_appendfile_transport.html

*If* maildrop has problems working exim's way, there is exim_lock
https://www.exim.org/exim-html-current/doc/html/spec_html/ch-exim_utilities.html#SECTmailboxmaint
which will lock a given mailbox and then call whatever shell or script
you wish to work on the mailbox ...

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Mon, Feb 05, 2024 at 04:20:19PM +0000, Andrew C Aitchison via Exim-users wrote:

> *If* maildrop has problems working exim's way, there is exim_lock
> which will lock a given mailbox and then call whatever shell or
> script you wish to work on the mailbox ...

> https://www.exim.org/exim-html-current/doc/html/spec_html/ch-exim_utilities.html#SECTmailboxmaint

There are also multiple external utilities for this purpose, such as
dotlockfile (in the liblockfile-bin package on Debian derived
systems).

--
Ian

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

So, taking jgh@wizmail.org URL:
https://exim.org/exim-html-current/doc/html/spec_html/ch-the_appendfile_transport.html

I see the text:  (I've added **highlights**)



...Before appending to a file, a number of security checks are made, and
the file
is locked. A detailed description is given below, after the list of private
options.

...  The file option specifies a single file, to which the message is
 appended. One or more of use_fcntl_lock, use_flock_lock, or
use_lockfile must
 be set with file

...

use_fcntl_lock     Use: appendfile     Type: boolean Default: see below

This option controls the use of the fcntl() function to lock a file for
exclusive use when a message is being appended. **It is set by default
unless
use_flock_lock is set.**

So this is set by default UNLESS ...one of the other locking options is
chosen:

If I do:

root@ybox:/var/lib/exim4# grep -i use_ /var/lib/exim4/config.autogenerated
root@ybox:/var/lib/exim4# grep -i lock /var/lib/exim4/config.autogenerated
<no output>
And I note, I have in that same file:

mail_spool:
  debug_print = "T: appendfile for $local_part@$domain"
  driver = appendfile
  file = /var/mail/$local_part_data
  delivery_date_add
  envelope_to_add
  return_path_add
  group = mail
  mode = 0660
  mode_fail_narrower = false

So I take these together to mean:

   exim will use "apendfile" to append mail to /var/mail/<myuserid>
   and it will use fcntl(2) (I'm assuming advisory locking?)

So I (may need to) just ensure maildrop(1) also uses fcntl(2) locking

FYI, my concern is the end of one of my maildrop configs, which ends:

if ("$ADDRLIST" ne "")
   {
   log "@ in maildrop.manual $NOW, will send (using
/etc/maildroprc.manual) copies like this: $ADDRLIST [SENDMAIL=$SENDMAIL]"

# Spaces at the start really mess it up
    ADDRLIST=substr($ADDRLIST,1)
   to '!'"$ADDRLIST"
   }
else
   {
    log "@ in maildrop.manual $NOW, no routing found, will drop through
to DEFAULT=$DEFAULT"
   }

So if it uses [ to '!'"$ADDRLIST" ] then it will pass the message to
SMTP (aka
exim4) and exim4 will write to e.g. /var/mail/myuserid ... or if it drops
through, then maildrop will write to (the file) $DEFAULT, which could
well be
/var/mail/myuserid. Now I can't actually work out a case where 2
instances of
this could be running simultaneously, but I'm just nervous of two
different bits
of code writing to the same file. I'm reasonably confident exim4 would
protect
itself from another exim4 instance and likewise maildrop.

I think I'll go back to maildrop and see why I saw flock(2) getting used
an if
it can use fcntl(2) instead.

From maildrop:


--enable-use-flock=flag - if this option is set to 1, maildrop will use
either
  the flock(), the lockf(), or the fcntl() system call to lock a
mailbox file
  when delivering a message. **On most systems, all three use
compatible locking
  mechanisms.** In some very isolated cases, flock(), lockf(), and
fcntl(), are
  different, incompatible, locking mechanisms. maildrop must use the same
  locking mechanism as any mail reading programs. The configuration
script will
  run some tests to determine what locking function calls are
available, and
  will choose one by itself. The --with-locking-method can be used to
manually
  choose the locking function call to use.

--with-locking-method=name - manually select a locking function call.
name is
  either "fcntl", "flock", or "lockf". Otherwise the configuration
script will
  pick one by itself.

The comment there is somewhat at odds, to the flock(2) man page:

NOTES
       Since kernel 2.0, flock() is implemented as a system call  in 
its  own
       right  rather than being emulated in the GNU C library as a call
to fc?
       ntl(2).  With this implementation, **there is no interaction
between  the
       types  of lock placed by flock() and fcntl(2),** and flock()
does not de?
       tect deadlock.


... I'm probably worrying about nothing, this is a standard Debian
install and this same issue exists for Thunderbird, pine, mutt etc

so I suspect it was "sorted out" during the packaging.




--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On 2024-02-10 graeme vetterlein via Exim-users <exim-users@lists.exim.org> wrote:
> So, taking jgh@wizmail.org URL:
> https://exim.org/exim-html-current/doc/html/spec_html/ch-the_appendfile_transport.html

> I see the text:? (I've added **highlights**)

[...]

> use_fcntl_lock ??? Use: appendfile ??? Type: boolean Default: see below

> This option controls the use of the fcntl() function to lock a file for
> exclusive use when a message is being appended. **It is set by default
> unless
> use_flock_lock is set.**

> So this is set by default UNLESS ...one of the other locking options is
> chosen:

> If I do:

> root@ybox:/var/lib/exim4# grep -i use_ /var/lib/exim4/config.autogenerated
[...]

> ?? exim will use "apendfile" to append mail to /var/mail/<myuserid>
> ?? and it will use fcntl(2) (I'm assuming advisory locking?)

ametzler@argenau:~$ /usr/sbin/exim4 -bP transport mail_spool | grep use_fcntl
use_fcntl_lock

> So I (may need to) just ensure maildrop(1) also uses fcntl(2) locking

[...]
> From maildrop:
[...]
> --with-locking-method=name - manually select a locking function call. name
> is either "fcntl", "flock", or "lockf". Otherwise the configuration
> script will pick one by itself.
[...]

Looking at
https://salsa.debian.org/debian/maildrop/-/blob/master/debian/rules?ref_type=heads
we find:
--with-locking-method=fcntl
cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On 10/02/2024 16:24, graeme vetterlein via Exim-users wrote:
> mail_spool:
>   debug_print = "T: appendfile for $local_part@$domain"
>   driver = appendfile
>   file = /var/mail/$local_part_data
>   delivery_date_add
>   envelope_to_add
>   return_path_add
>   group = mail
>   mode = 0660
>   mode_fail_narrower = false
>
> So I take these together to mean:
>
>    exim will use "apendfile" to append mail to /var/mail/<myuserid>

ONLY IF this transport is the one being used for your delivery.
The delivery lines ("=>") in the log should give a hint.

>    and it will use fcntl(2) (I'm assuming advisory locking?)

I'm assuming you've read it right. Yes, "fcntl" does an F_SETLOCKW.

If you're really nervous, I spot a debug output in the transport code which
says what sort of locking method is being used.
--
Cheers,
Jeremy


PS: "Exim". Not "Exim 4" (unless you're a committed Debian-ite),
Version 3 of Exim went obsolete twenty years ago.
There is no Exim 3, nor Exim 5.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Sat, 10 Feb 2024, graeme vetterlein via Exim-users wrote:

> ... I'm probably worrying about nothing, this is a standard Debian install
> and this same issue exists for Thunderbird, pine, mutt etc

If you are being paranoid, I'll mention that if you are mounting the
file system over the network (traditionally NFS, but samba/SMB/CIFS or
fuse would be significant too) you need to check how/whether locking
works on that file system.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/