Mailing List Archive

>Here are some suggestions for improvements of the perl "flock" function:

>1. It would be nice if "flock" did a flush.=20

>2. On systems with F_SETLK/F_SETLKW it would be nice if "flock" used =
>them instead of "lockf" for emulating "flock". This would allow =
>differentiation of LOCK_SH and LOCK_EX (F_RDLCK and F_WRLCK).

I'll give you a nickel if you just stop sending things in
this bizarro format. :-(
Make the =signs go=
away, please=.

OK=?=

I don't think that you want to be extending flock() beyond
the way it is currently defined. It does, however, make
sense to support flock on systems that require emulation.
This is a better strategy because it presents a uniform
user-interface to people no matter what platform they are on.

Also, I don't like to see people suggest lockf as the way
to go. The idea that "it works with the network" is misleading.
You aren't ever going to get the pid back over the network,
and if you do, it will be wrong. As for locking not working
right with flock vs lockf, that depends on the system. I'm
98% certain that flock does network locking on the current BSD
releases, and I would be sadly disappointed if the Linux versions
didn't do this at all.o

In short, all we really need is a very very SIMPLE file locking
interface. Let's not overengineer this into some crazy distributed
object-oriented redundant mime-encoded proxy PGP-signed piece of
crud.

The flock is a fine high-level interface that assumes maximal
portability.

Sorry, jik.

--tom

Well, I guess I have to say something about this myself before
somebody else notices that I have been hacking for an extension called
File::Lock 1) for (all too) long time now...

I agree with Tom: flock() should and must not be overloaded (not as in
OO, as in "featuritis") with any shared/exclusive whole/segment
local/network bells and whistles. To paraphrase someone:

a flock() is a flock() is a flock()

flock() is a UNIX 2) system call, full stop, period. It would be very
bad idea to imagine more functionality inherent in it. It would be
very wrong to make the perl flock() to be a general file locking
mechanism, "just like C flock() but on steroids". Perl does mirror in
its functions the underlying C (ANSI C, POSIX, SPEC1170, whatever)
libraries almost perfectly. Let us not make a wrong move here.

I will _try_ to get File::Lock out "RSN" in some form, maybe
I'll stop thinking about the general/OO bit by myself and just
release the system call parts so that we (perl5-porters) can
start arg^H^H^Hdiscussing about the Correct Interface.

++jhi;

1) flock, lockf, fcntl locking, possibly some OO stuff thrown in
2) and "just" a BSD UNIX system call to be more exact

P.S. If _I_ could decide, I would integrate file locking into open().
That is where it logically resides.

P.P.S. One quite portable way to do exclusive locking is to attempt
a mkdir() before the open. Forget a lock file: NFS will
ruin your best intentions. But a directory creation, mkdir(),
is synchronous and atomic (says I, I have no proof...)

# something like this for lock()ing things
$lockdirname = ...;
$lockfilename = ...($lockdirname, ...);
if (-d $lockdirname) {
# somebody else beat us to it, print out the lockfile contents
} else {
if (mkdir($lockdirname, 0755)) {
if (-f $lockfilename) {
# yeaaaargghh! who on earth could be here except us?
# suspect a race condition and wimp out
} else {
if (open(LOCKFILE, ">$lockfilename")) {
print LOCKFILE <<__EOL__;
pid $$
uid $<
gid $(
user ...
host ...
GMT YYYYMMDD-HHMMSS
__EOL__
close(LOCKFILE)
...
} else {
# lock failed? why? permissions?
}
}
}

Jarkko Hietaniemi wrote :
|| P.P.S. One quite portable way to do exclusive locking is to attempt
|| a mkdir() before the open. Forget a lock file: NFS will
|| ruin your best intentions. But a directory creation, mkdir(),
|| is synchronous and atomic (says I, I have no proof...)

I'm pretty sure that NFS can mess up mkdir too - it can seem to
fail, but actually succeed - simply by losing the acknowledge,
and there is no way for you to tell whether it was your attempt
that succeeded or someone else's. After that, unless there is
some sort of deltion of the directory after a specified length
of time, *nobody* will ever get the lock, becuase the process
that created the directory doesn't know to remove it. The
safest approach is:

(1) make a file with a unique name, and with unique contents

(2) attempt to link that file to "LOCK" (or the common
agreed upon name)

(3) if the file exists and contains _your_ unique name,
you've got the lock else you don't

(4) if you don't get the lock, then sleep for a while - a
short while if the lock file doesn't seem to have been there
in step 3 or a longer while if it was - and then try again

If you're on an OS that doesn't support link, then you have to
go with some other non-portable approach.

The key aspect here is that the (supposedly) atomic action not
only creates the lock, but also creates a lock that can be
distinguished as being owned by its creator - even if NFS drops
a packet in the middle.

--
Maybe we can fix reality one of these days. | John Macdonald
<- Larry Wall Carl Dichter -> | jmm@Elegant.COM
I'd be happy if we could just index it. |

> (4) if you don't get the lock, then sleep for a while - a
> short while if the lock file doesn't seem to have been there
> in step 3 or a longer while if it was

Sorry to jump in so late, but did somebody mention this step?

(5) if the LOCK file is more than XXX minutes old, suspect that
the process (or host) which created it has died, and yell for help.


Regards,

Hallvard

On Mon, 20 Nov 1995, John Macdonald wrote:

> [...] The
> safest approach is:
>
> (1) make a file with a unique name, and with unique contents

Specifically, something like "`hostname`.$$.$randomnumber.$time". Since
you are talking NFS, this name must be unique across the entire NFS zone.

> (2) attempt to link that file to "LOCK" (or the common
> agreed upon name)

Usually "$target.lock", if you are locking a file.

> (3) if the file exists and contains _your_ unique name,
> you've got the lock else you don't

Alternatively, check the link count on your unique name. If it's 2, then
it must have succeeded. (Note, NFS may be able to screw this up too -- I
don't know for certain.)

> (4) if you don't get the lock, then sleep for a while - a
> short while if the lock file doesn't seem to have been there
> in step 3 or a longer while if it was - and then try again

You probably need programmable settings for this, for both the delay and
the retry.

> If you're on an OS that doesn't support link, then you have to
> go with some other non-portable approach.

rename-with-overwrite, if NFS supports _that_.

> The key aspect here is that the (supposedly) atomic action not
> only creates the lock, but also creates a lock that can be
> distinguished as being owned by its creator - even if NFS drops
> a packet in the middle.

One more bit is missing: detecting the case where a lock failed, but
because the program that created the lock is dead. Obviously, if the file
contains it's PID then you can see if it still exists, but this doesn't
work too well over NFS.

Hmm. Just thought of a clever scheme: create a $unique filename, link it
to $lockfile. Now both have a link count of two. Open $unqiue, but delete
it's file. $lockfile should still have a link count of two, and will until
the program exits. (Of course NFS will screw all of this up too.)

> --
> Maybe we can fix reality one of these days. | John Macdonald
> <- Larry Wall Carl Dichter -> | jmm@Elegant.COM
> I'd be happy if we could just index it. |
>

--
Kenneth Albanowski (kjahds@kjahds.com, CIS: 70705,126)

Tom> I'll give you a nickel if you just stop sending things in
Tom> this bizarro format. :-(
Tom> Make the =signs go=
Tom> away, please=.

Oh how boring. I have to hit the return key. Well, just for you. (I'd give
in if I were you though - The invasion of the MIME bodysnatchers is
unstoppable).

Tom> I don't think that you want to be extending flock() beyond
Tom> the way it is currently defined. It does, however, make
Tom> sense to support flock on systems that require emulation.

Ok, so we don't extend flock() to allow locks on segments of files rather than
whole files. This makes me a little bit sad but that's life. However we should
use fcntl F_SETLK/F_SETLKW instead of "lockf" because it can provide a better
emulation of flock that lockf can. (fcntl F_SETLK knows the difference between
a shared and an exclusive lock, lockf doesn't).

Tom> Also, I don't like to see people suggest lockf as the way
Tom> to go.

Well perl already uses "lockf" if it's there and "flock" isn't. I'm suggesting
that "lockf" should not be used on systems where it's just a less flexible
layer on top of fcntl.

Tom> The idea that "it works with the network" is misleading.
Tom> You aren't ever going to get the pid back over the network,
Tom> and if you do, it will be wrong.

SVR4 returns a "system id" with the pid. God knows what it is but I don't
care. I'm not suggesting use of the "F_GETLCK" function.

Here's a function to emulate flock if fcntl(F_SETLK[W]) is available:

int
fcntl_emulate_flock (fd, operation)
int fd;
int operation;
{
struct flock flock;
int i;

switch (operation & ~LOCK_NB) {

/* LOCK_SH - get a shared lock */
case LOCK_SH:
flock.l_type = F_RDLCK;
break;

/* LOCK_EX - get an exclusive lock */
case LOCK_EX:
flock.l_type = F_WRLCK;
break;

case LOCK_UN:
flock.l_type = F_UNLCK;
break;

default:
errno = EINVAL;
return -1;
}

flock.l_whence = SEEK_SET;
flock.l_start = 0;
flock.l_len = 0;

return = fnctl (fd, operation & LOCK_NB ? F_SETLK : F_SETLKW, &flock);
}


--
John Hughes <john@AtlanTech.COM> or <JH10@CalvaCom.FR>,
Atlantic Technologies Inc. Tel: +33-1-43204546
24 rue Montbrun, Fax: +33-1-43204579
75014 PARIS.

>> (4) if you don't get the lock, then sleep for a while - a
>> short while if the lock file doesn't seem to have been there
>> in step 3 or a longer while if it was

HBF> Sorry to jump in so late, but did somebody mention this step?

HBF> (5) if the LOCK file is more than XXX minutes old, suspect that
HBF> the process (or host) which created it has died, and yell for help.

I suggest you look at how lockfile (which is part of the procmail
package) does it. I believe it claims to work over NFS, and is widely
portable.

P.

this locking stuff really is depressingly complex, eh?

--tom

>Tom> I'll give you a nickel if you just stop sending things in
>Tom> this bizarro format. :-(
>Tom> Make the =signs go=
>Tom> away, please=.
>
>Oh how boring. I have to hit the return key. Well, just for you. (I'd give
>in if I were you though - The invasion of the MIME bodysnatchers is
>unstoppable).

What do you think I wrote my autofaq for? :-)

I'm reading things with /usr/ucb/Mail. This is my normal
mailer when I'm at a customer site, and frequently it's
going through a PC vt-100 emulator that doesn't wrap
lines and doesn't let me reconfigure it. That means that
using anything except that standard mail format is bound
to be a loss for me, and often for others as well. The invasion
of the PCs with their broken notion of infinite line lengths
is a real problem. The PC/Mac world coming to the internet
and imposing their standards is like Americans expecting
French waiters to speak English. It doesn't work well.

Tom> I don't think that you want to be extending flock() beyond
Tom> the way it is currently defined. It does, however, make
Tom> sense to support flock on systems that require emulation.

Ok, so we don't extend flock() to allow locks on segments of files rather than
whole files. This makes me a little bit sad but that's life.

As others have said, a flock if a flock. I would like flock to have
a very simple, standard, and portable interface. Can anyone report its
feasibility on MS-DOS, Mac, OS/2, NT, VMS, CMS et cetera ad nauseum?

--tom

teachers writes:

> Ok, so we don't extend flock() to allow locks on segments of files rather than
> whole files. This makes me a little bit sad but that's life.
>
> As others have said, a flock if a flock. I would like flock to have
> a very simple, standard, and portable interface. Can anyone report its
> feasibility on MS-DOS, Mac, OS/2, NT, VMS, CMS et cetera ad nauseum?
>

In the current EMX (POSIXish subsystem for DOS/OS/2) lockf is missing,
flock is a stub. I may try to find out what functions are available as
OS/2 API if this is important.

Ilya